Page History

DSpace 1.6 and newer versions uses the Apache SOLR application underlying the statistics. SOLR enables performant searching and adding to vast amounts of (usage) data.
Unlike previous versions, enabling statistics in DSpace does not require additional installation or customization. All the necessary software is included.

Table of Contents

minLevel	2
outline	true
style	none

What is exactly being logged ?

Each time a page or file gets requested, this request is being logged. The logging happens at the server side, and doesn't require a javascript like Google Analytics does, to provide usage data.

Definition of which fields are to be stored happens in the file dspace/solr/statistics/conf/schema.xml.
The fields, stored in a usage event by default are:

DSpace 1.6 and newer

After the introduction of the SOLR Statistics logging in DSpace 1.6, every pageview and file download is logged in a dedicated SOLR statistics core.

DSpace 3.0 and newer

In addition to the already existing logging of pageviews and downloads, DSpace 3.0 now also logs search queries users enter in the DSpace search dialog and workflow events.

Warning

title	JSP UI Search Query logging

Due to the very recent addition of Discovery for search & faceted browsing in JSPUI, these search queries are not yet logged. Regular (non-discovery) search queries are being logged in JSP UI.

Warning

title	Workflow Events logging

Only workflow events, initiated and executed by a physical user are being logged. Automated workflow steps or ingest procedures are currently not being logged by the workflow events logger.

The logging happens at the server side, and doesn't require a javascript like Google Analytics does, to provide usage data. Definition of which fields are to be stored happens in the file dspace/solr/statistics/conf/schema.xml.

Although they are stored in the same index, the stored fields for views, search queries and workflow events are different. A new field, statistics_type determines which kind of a usage event you are dealing with. The three possible values for this field are view, search and workflow.

Code Block

language	html/xml

<field name="statistics_type" type="string

Code Block


<field name="type" type="integer" indexed="true" stored="true" required="true" />
<field name="id" type="integer" indexed="true" stored="true" required="true" />
<field name="ip" type="string" indexed="true" stored="true" required="false" />
<field name="time" type="date" indexed="true" stored="true" required="true" />
<field name="epersonid" type="integer" indexed="true" stored="true" required="falsetrue" />

Common stored fields for all usage events

Code Block

language	html/xml


<field name="continenttype" type="stringinteger" indexed="true" stored="true" required="falsetrue" />
<field name="countryid" type="stringinteger" indexed="true" stored="true" required="falsetrue" />
<field name="countryCodeip" type="string" indexed="true" stored="true" required="false" />
<field name="citytime" type="stringdate" indexed="true" stored="true" required="falsetrue" />
<field name="longitudeepersonid" type="floatinteger" indexed="true" stored="true" required="false" />
<field name="latitudecontinent" type="floatstring" indexed="true" stored="true" required="false"/>
<field name="owningCommcountry" type="integerstring" indexed="true" stored="true" required="false" multiValued="true"/>
<field name="owningCollcountryCode" type="integerstring" indexed="true" stored="true" required="false" multiValued="true"/>
<field name="owningItemcity" type="integerstring" indexed="true" stored="true" required="false" multiValued="true"/>
<field name="dnslongitude" type="stringfloat" indexed="true" stored="true" required="false"/>
<field name="userAgentlatitude" type="stringfloat" indexed="true" stored="true" required="false"/>
<field name="isBotowningComm" type="booleaninteger" indexed="true" stored="true" required="false" multiValued="true"/>
<field name="bundleNameowningColl" type="stringinteger" indexed="true" stored="true" required="false" multiValued="true" />

The combination of type and id determine which resource (either community, collection, item page or file download) has been requested.

Web user interface for DSpace statistics

In the XMLUI, statistics can be accessed from the lower end of the navigation menu. In the JSPUI, a view statistics button appears on the bottom of pages for which statistics are available.

If you are not seeing these links or buttons, it's likely that they are only enabled for administrators in your installation. Change the configuration parameter "statistics.item.authorization.admin" to false in order to make statistics visible for all repository visitors.

Home page

Starting from the repository homepage, the statistics page displays the top 10 most popular items of the entire repository.

Community home page

The following statistics are available for the community home pages:

Total visits of the current community home page
Visits of the community home page over a timespan of the last 7 months
Top 10 country from where the visits originate
Top 10 cities from where the visits originate

Collection home page

The following statistics are available for the collection home pages:

Total visits of the current collection home page
Visits of the collection home over a timespan of the last 7 months
Top 10 country from where the visits originate
Top 10 cities from where the visits originate

Item home page

The following statistics are available for the item home pages:

Total visits of the item
Total visits for the bitstreams attached to the item
Visits of the item over a timespan of the last 7 months
Top 10 country views from where the visits originate
Top 10 cities from where the visits originate

Usage Event Logging and Usage Statistics Gathering

The DSpace Statistics Implementation is a Client/Server architecture based on Solr for collecting usage events in the JSPUI and XMLUI user interface applications of DSpace. Solr runs as a separate webapplication and an instance of Apache Http Client is utilized to allow parallel requests to log statistics events into this Solr instance.

Configuration settings for Statistics

In the {dspace.dir}/config/modules/solr-statistics.cfg file review the following fields to make sure they are uncommented:

="true"/>
<field name="owningItem" type="integer" indexed="true" stored="true" required="false" multiValued="true"/>
<field name="dns" type="string" indexed="true" stored="true" required="false"/>
<field name="userAgent" type="string" indexed="true" stored="true" required="false"/>
<field name="isBot" type="boolean" indexed="true" stored="true" required="false"/>
<field name="referrer" type="string" indexed="true" stored="true" required="false"/>
<field name="uid" type="uuid" indexed="true" stored="true" default="NEW" />
<field name="statistics_type" type="string" indexed="true" stored="true" required="true" default="view" />

The combination of type and id determines which resource (either community, collection, item page or file download) has been requested.

Unique stored fields for bitstream downloads

Code Block

language	html/xml

<field name="bundleName" type="string" indexed="true" stored="true" required="false" multiValued="true" />

Unique stored fields for search queries

Code Block

language	html/xml

<field name="query" type="string" indexed="true" stored="true" required="false" multiValued="true"/>
<field name="scopeType" type="integer" indexed="true" stored="true" required="false" />
<field name="scopeId" type="integer" indexed="true" stored="true" required="false" />
<field name="rpp" type="integer" indexed="true" stored="true" required="false" />
<field name="sortBy" type="string" indexed="true" stored="true" required="false" />
<field name="sortOrder" type="string" indexed="true" stored="true" required="false" />
<field name="page" type="integer" indexed="true" stored="true" required="false" />

Unique stored fields for workflow events

Code Block

language	html/xml

<field name="workflowStep" type="string" indexed="true" stored="true" required="false" multiValued="true"/>
<field name="previousWorkflowStep" type="string" indexed="true" stored="true" required="false" multiValued="true"/>
<field name="owner" type="string" indexed="true" stored="true" required="false" multiValued="true"/>
<field name="submitter" type="integer" indexed="true" stored="true" required="false" />
<field name="actor" type="integer" indexed="true" stored="true" required="false" />
<field name="workflowItemId" type="integer" indexed="true" stored="true" required="false" />

Web User Interface Elements

Pageview and Download statistics

In the XMLUI, pageview and download statistics can be accessed from the lower end of the navigation menu. In the JSPUI, a view statistics button appears on the bottom of pages for which statistics are available.

If you are not seeing these links or buttons, it's likely that they are only enabled for administrators in your installation. Change the configuration parameter "authorization.admin.usage" in usage-statistics.cfg to false in order to make statistics visible for all repository visitors.

Home page

Starting from the repository homepage, the statistics page displays the top 10 most popular items of the entire repository.

Community home page

The following statistics are available for the community home pages:

Total visits of the current community home page
Visits of the community home page over a timespan of the last 7 months
Top 10 country from where the visits originate
Top 10 cities from where the visits originate

Collection home page

The following statistics are available for the collection home pages:

Total visits of the current collection home page
Visits of the collection home over a timespan of the last 7 months
Top 10 country from where the visits originate
Top 10 cities from where the visits originate

Item home page

The following statistics are available for the item home pages:

Total visits of the item
Total visits for the bitstreams attached to the item
Visits of the item over a timespan of the last 7 months
Top 10 country views from where the visits originate
Top 10 cities from where the visits originate

Search Query Statistics

In the XMLUI, search query statistics can be accessed from the lower end of the navigation menu.

If you are not seeing the link labelled "search statistics", it is likely that they are only enabled for administrators in your installation. Change the configuration parameter "authorization.admin.search" in usage-statistics.cfg to false in order to make statistics visible for all repository visitors.

The dropdown on top of the page allows you to modify the time frame for the displayed statistics.

The Pageviews/Search column tracks the amount of pages visited after a particular search term. Therefor a zero in this column means that after executing a search for a specific keyword, not a single user has clicked a single result in the list.

If you are using Discovery, note that clicking the facets also counts as a search, because clicking a facet sends a search query to the Discovery index.

Image Added

Workflow Event Statistics

In the XMLUI, search query statistics can be accessed from the lower end of the navigation menu.

If you are not seeing the link labelled "Workflow statistics", it is likely that they are only enabled for administrators in your installation. Change the configuration parameter "authorization.admin.workflow" in usage-statistics.cfg to false in order to make statistics visible for all repository visitors.

The dropdown on top of the page allows you to modify the time frame for the displayed statistics.

Image Added

Architecture

The DSpace Statistics Implementation is a Client/Server architecture based on Solr for collecting usage events in the JSPUI and XMLUI user interface applications of DSpace. Solr runs as a separate webapplication and an instance of Apache Http Client is utilized to allow parallel requests to log statistics events into this Solr instance.

Configuration settings for Statistics

In the {dspace.dir}/config/modules/solr-statistics.cfg file review the following fields to make sure they are uncommented:

Is used by the SolrLogger Client class to connect to the Solr server over http and perform updates and queries. In most cases, this can (and should) be set to localhost (or 127.0.0.1).
To determine the correct path, you can use a tool like wget to see where Solr is responding on your server. For example, you'd want to send a query to Solr like the following:

Property:

server

Example Values:

server = http://127.0.0.1/solr/statistics
server = ${solr.server}/statistics

Informational Note:

Is used by the SolrLogger Client class to connect to the Solr server over http and perform updates and queries. In most cases, this can (and should) be set to localhost (or 127.0.0.1).

To determine the correct path, you can use a tool like wget to see where Solr is responding on your server. For example, you'd want to send a query to Solr like the following:

Code Block
wget http://127.0.0.1/solr/statistics/select?q=:

Assuming you get an HTTP 200 OK response, then you should set solr.log.server to the '/statistics' URL of 'http://127.0.0.1/solr/statistics' (essentially removing the "/select?q=:" query off the end of the responding URL.)

Property:

query.filter.bundles

Example
Value:

query.filter.bundles=ORIGINAL

Informational
Note:

A comma seperated list that contains the bundles for which the file statistics will be displayed.

Property:

solr.statistics.query.filter.spiderIp

Example Value:

solr.statistics.query.filter.spiderIp = false

Informational Note:

If true, statistics queries will filter out spider IPs -- use with caution, as this often results in extremely long query strings.

Property:

solr.statistics.query.filter.isBot

Example Value:

solr.statistics.query.filter.isBot = true

Informational Note:

If true, statistics queries will filter out events flagged with the "isBot" field. This is the recommended method of filtering spiders from statistics.

Property:

server

Example Value:

server = http://127.0.0.1/solr/statistics

Informational Note:

Code Block
wget http://127.0.0.1/solr/statistics/select?q=:

Assuming you get an HTTP 200 OK response, then you should set solr.log.server to the '/statistics' URL of 'http://127.0.0.1/solr/statistics' (essentially removing the "/select?q=:" query off the end of the responding URL.)

Property:

spiderips.urls

Example Value:

spiderips.urls =

Code Block


http://iplists.com/google.txt, \
http://iplists.com/inktomi.txt, \
http://iplists.com/lycos.txt, \
http://iplists.com/infoseek.txt, \
http://iplists.com/altavista.txt, \
http://iplists.com/excite.txt, \
http://iplists.com/misc.txt, \
http://iplists.com/non_engines.txt

Informational Note:

List of URLs to download spiders files into [dspace]/config/spiders. These files contain lists of known spider IPs and are utilized by the SolrLogger to flag usage events with an "isBot" field, or ignore them entirely.

The "stats-util" command can be used to force an update of spider files, regenerate "isBot" fields on indexed events, and delete spiders from the index. For usage, run:

Code Block
dspace stats-util -h

from your [dspace]/bin directory

In the {dspace.dir}/config/modules/usage-statistics.cfg file review the following fields to make sure they are uncommented:

statistics.item. solrstatisticslogBots solrstatisticslogBots this property is false, and IP is detected as a spider, the event is not logged.
When this property is set to true, the event will be logged with the "isBot" field set to true.
(see solr.statistics.query.filter.* for query filter options) solrstatistics.query.filter.spiderIp solrstatistics.query.filter.spiderIp = false solr.statistics.query.filter.isBot

Property:	dbfile
Example Value:	dbfile = ${dspace.dir}/config/GeoLiteCity.dat
Informational Note:	The following referes to the GeoLiteCity database file utilized by the LocationUtils to calculate the location of client requests based on IP address. During the Ant build process (both fresh_install and update) this file will be downloaded from http://www.maxmind.com/app/geolitecity if a new version has been published or it is absent from your [dspace]/config directory.

Property:	resolver.timeout
Example Value:	resolver.timeout = 200
Informational Note:	Timeout in milliseconds for DNS resolution of origin hosts/IPs. Setting this value too high may result in solr exhausting your connection pool.

Property:	useProxies
Example Value:	useProxies = true
Informational Note:	Will cause Statistics logging to look for X-Forward URI to detect clients IP that have accessed it through a Proxy service (e.g. the Apache mod_proxy). Allows detection of client IP when accessing DSpace. [Note: This setting is found in the DSpace Logging section of dspace.cfg]

Property:	authorization.admin.usage
Example Value:	statistics.item.authorization.admin.usage = true
Informational Note:	When set to true, only general administrators, collection and community administrators are able to access the pageview and download statistics from the web user interface. As a result, the links to access statistics are hidden for non logged-in admin users. Setting this property to "false" will display the links to access statistics to anyone, making them publicly available.

Property:	authorization.	admin.	search
Example Value:	authorization.	admin.	search = true
Informational Note:	When	set to	true, only system, collection or community administrators are able to access statistics on search queries.

Property:	authorization.	admin.workflow
Example Value:	authorization.	admin.workflow = true
Informational Note:	If true, statistics queries will filter out spider IPs -- use with caution, as this often results in extremely long query strings.	Property:	When set to true, only system, collection or community administrators are able to access statistics on workflow events.

Property:	logBots
Example Value:	solr.statistics.query.filter.isBot logBots = true
Informational Note:	When this property is set to false, and IP is detected as a spider, the event is not logged. When this property is set to true, the event will be logged If true, statistics queries will filter out events flagged with the "isBot" field . This is the recommended method of filtering spiders from statistics.	Property:	set to true. (see solr.statistics.query.filter.bundles
Example Value:	query.filter.bundles=ORIGINAL
Informational Note:	A comma seperated list that contains the bundles for which the file statistics will be displayed.

Upgrade Process for Statistics

Example of rebuild and redeploy DSpace (only if you have configured your distribution in this manner)

First approach the traditional DSpace build process for updating

Code Block
cd [dspace-source]/dspace mvn package cd [dspace-source]/dspace/target/dspace-<version>-build.dir ant -Dconfig=[dspace]/config/dspace.cfg update cp -R [dspace]/webapps/* [TOMCAT]/webapps

The last step is only used if you do not follow the recommended practice of configuring [dspace]/webapps as location for webapps in your servlet container (Tomcat, Resin or Jetty). If you only need to build the statistics, and don't make any changes to other web applications, you can replace the copy step above with:

Code Block
cp -R dspace/webapps/solr TOMCAT/webapps

Again, only if you are not mounting [dspace]/webapps directly into your Tomcat, Resin or Jetty host (the recommended practice)

Restart your webapps (Tomcat/Jetty/Resin)

Older setting that are not related to the new 1.6 Statistics

The following Dspace.cfg fields are only applicable to the older statistics solution.

Code Block


###### Statistical Report Configuration Settings ######

 # should the stats be publicly available?  should be set to false if you only
 # want administrators to access the stats, or you do not intend to generate
 # any
 report.public = false

 # directory where live reports are stored
 report.dir = ${dspace.dir}/reports/

* for query filter options)

Pre-1.6 Statistics settings

Older versions of DSpace featured static reports generated from the log files. They still persist in DSpace today but are completely independent from the SOLR based statistics.
The following configuration parameters applicable to these reports can be found in dspace.cfg.

Code Block

###### Statistical Report Configuration Settings ######

 # should the stats be publicly available?  should be set to false if you only
 # want administrators to access the stats, or you do not intend to generate
 # any
 report.public = false

 # directory where live reports are stored
 report.dir = ${dspace.dir}/reports/

These fields are not used by the new 1.6 Statistics, but are only related to the Statistics from previous DSpace releases

Upgrade Process for Statistics

Example of rebuild and redeploy DSpace (only if you have configured your distribution in this manner)

First approach the traditional DSpace build process for updating

Code Block
cd [dspace-source]/dspace mvn package cd [dspace-source]/dspace/target/dspace-<version>-build.dir ant -Dconfig=[dspace]/config/dspace.cfg update cp -R [dspace]/webapps/* [TOMCAT]/webapps

The last step is only used if you do not follow the recommended practice of configuring [dspace]/webapps as location for webapps in your servlet container (Tomcat, Resin or Jetty). If you only need to build the statistics, and don't make any changes to other web applications, you can replace the copy step above with:

Code Block
cp -R dspace/webapps/solr TOMCAT/webapps

Again, only if you are not mounting [dspace]/webapps directly into your Tomcat, Resin or Jetty host (the recommended practice)

Restart your webapps (Tomcat/Jetty/Resin)These fields are not used by the new 1.6 Statistics, but are only related to the Statistics from previous DSpace releases

Statistics Administration

...

The command line interface (CLI) scripts can be used to clean the usage database from additional spider traffic and other maintenance tasks. In DSpace 3.0, a script has been added to split up the monolithic SOLR core into individual cores each containing a year of statistics.

Statistics differences between DSpace 1.7.x and 1.8.0

...

When a backup has been made start the Tomcat/Jetty/Resin server program.
The update script has one optional command which will if given not only update the broken file statistics but also delete file statistics for files that where removed from the system (if this option isn't active these statistics will receive the "BITSTREAM_DELETED" bundle name).

Code Block
#The -r is optional [dspace]/bin/dspace stats-util -b -r

...

If required, the solr server can be optimized by running

Code Block
{dspace.dir}/bin/stats-util -o

...

In DSpace 1.6.x, each solr event was committed to the solr server individually. For high load DSpace installations, this would result in a huge load of small solr commits resulting in a very high load on the solr server.
This has been resolved in dspace 1.7 by only committing usage events to the solr server every 15 minutes. This will result in a delay of the storage of a usage event of maximum 15 minutes. If required, this value can be altered by changing the maxTime property in the

Code Block
{dspace.dir}/solr/statistics/conf/solrconfig.xml

...

Modifying the number of months, for which statistics are displayed

Modify line 178 205 in the StatisticsTransformer.java file

https://github.com/DSpace/DSpace/blob/dspace-xmlui3_x/dspace-xmlui-api/src/main/java/org/dspace/app/xmlui/aspect/statistics/StatisticsTransformer.javajava#L205

-6 is the default setting, displaying the past 6 months of statistics. When reducing this to a smaller natural number, less months are being displayed.

Related: DatasetTimeGenerator Javadoc

Custom Reporting - Querying SOLR Directly

...

Top downloaded items by a specific user

Query:

Code Block
http://localhost:8080/solr/statistics/select?indent=on&version=2.2&start=0&rows=10&fl=*%2Cscore&qt=standard&wt=standard&explainOther=&hl.fl=&facet=true&facet.field=epersonid&q=type:0

...

facet.field=epersonid — You want to group by epersonid, which is the user id.
type:0 — Interested in bitstreams only

Code Block


<lst name="facet_counts">
    <lst name="facet_fields">
        <lst name="epersonid">
            <int name="66">1167</int>

<int name="117">251</int>

<int name="52">42</int>

<int name="19">36</int>

<int name="88">20</int>

<int name="112">18</int>

<int name="110">9</int>

<int name="96">0</int>

</lst>
    </lst>
</lst>

...

You have two options to install/update this file:

Attempt to re-run the automatic installer from your DSpace Source Directory ([dspace-source]). This will attempt to automatically download the database file, unzip it and install it into the proper location:
Code Block
ant update_geolite
- NOTE: If the location of the GeoLite Database file is known to have changed, you can also run this auto-installer by passing it the new URL of the GeoLite Database File: ant -Dgeolite=[full-URL-of-geolite] update_geolite
OR, you can manually install the file by performing these steps yourself:
- First, download the latest GeoLite Database file from http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
- Next, unzip that file to create a file named GeoLiteCity.dat
- Finally, move or copy that file to your DSpace installation, so that it is located at [dspace]/config/GeoLiteCity.dat.

...

All Versions

DSpace Documentation

Page tree

Versions Compared

Old Version 5

New Version Current

Key

What is exactly being logged ?

Common stored fields for all usage events

Web user interface for DSpace statistics

Home page

Community home page

Collection home page

Item home page

Usage Event Logging and Usage Statistics Gathering

Configuration settings for Statistics

Unique stored fields for bitstream downloads

Unique stored fields for search queries

Unique stored fields for workflow events

Web User Interface Elements

Pageview and Download statistics

Home page

Community home page

Collection home page

Item home page

Search Query Statistics

Workflow Event Statistics

Architecture

Configuration settings for Statistics

Upgrade Process for Statistics

Older setting that are not related to the new 1.6 Statistics

Pre-1.6 Statistics settings

Upgrade Process for Statistics

Statistics Administration

Statistics differences between DSpace 1.7.x and 1.8.0

Modifying the number of months, for which statistics are displayed

Custom Reporting - Querying SOLR Directly

Top downloaded items by a specific user