Page History
...
| Table of Contents | ||||||
|---|---|---|---|---|---|---|
|
What is exactly being logged ?
DSpace 1.6 and newer
After the introduction of the SOLR Statistics logging in DSpace 1.6, every pageview and file download is logged in a dedicated SOLR statistics core.
...
| Code Block | ||
|---|---|---|
| ||
<field name="statistics_type" type="string" indexed="true" stored="true" required="true" /> |
Common stored fields for all usage events
| Code Block | ||
|---|---|---|
| ||
<field name="type" type="integer" indexed="true" stored="true" required="true" /> <field name="id" type="integer" indexed="true" stored="true" required="true" /> <field name="ip" type="string" indexed="true" stored="true" required="false" /> <field name="time" type="date" indexed="true" stored="true" required="true" /> <field name="epersonid" type="integer" indexed="true" stored="true" required="false" /> <field name="continent" type="string" indexed="true" stored="true" required="false"/> <field name="country" type="string" indexed="true" stored="true" required="false"/> <field name="countryCode" type="string" indexed="true" stored="true" required="false"/> <field name="city" type="string" indexed="true" stored="true" required="false"/> <field name="longitude" type="float" indexed="true" stored="true" required="false"/> <field name="latitude" type="float" indexed="true" stored="true" required="false"/> <field name="owningComm" type="integer" indexed="true" stored="true" required="false" multiValued="true"/> <field name="owningColl" type="integer" indexed="true" stored="true" required="false" multiValued="true"/> <field name="owningItem" type="integer" indexed="true" stored="true" required="false" multiValued="true"/> <field name="dns" type="string" indexed="true" stored="true" required="false"/> <field name="userAgent" type="string" indexed="true" stored="true" required="false"/> <field name="isBot" type="boolean" indexed="true" stored="true" required="false"/> <field name="referrer" type="string" indexed="true" stored="true" required="false"/> <field name="uid" type="uuid" indexed="true" stored="true" default="NEW" /> <field name="statistics_type" type="string" indexed="true" stored="true" required="true" default="view" /> |
The combination of type and id determines which resource (either community, collection, item page or file download) has been requested.
Unique stored fields for bitstream downloads
| Code Block | ||
|---|---|---|
| ||
<field name="bundleName" type="string" indexed="true" stored="true" required="false" multiValued="true" /> |
Unique stored fields for search queries
| Code Block | ||
|---|---|---|
| ||
<field name="query" type="string" indexed="true" stored="true" required="false" multiValued="true"/> <field name="scopeType" type="integer" indexed="true" stored="true" required="false" /> <field name="scopeId" type="integer" indexed="true" stored="true" required="false" /> <field name="rpp" type="integer" indexed="true" stored="true" required="false" /> <field name="sortBy" type="string" indexed="true" stored="true" required="false" /> <field name="sortOrder" type="string" indexed="true" stored="true" required="false" /> <field name="page" type="integer" indexed="true" stored="true" required="false" /> |
Unique stored fields for workflow events
| Code Block | ||
|---|---|---|
| ||
<field name="workflowStep" type="string" indexed="true" stored="true" required="false" multiValued="true"/> <field name="previousWorkflowStep" type="string" indexed="true" stored="true" required="false" multiValued="true"/> <field name="owner" type="string" indexed="true" stored="true" required="false" multiValued="true"/> <field name="submitter" type="integer" indexed="true" stored="true" required="false" /> <field name="actor" type="integer" indexed="true" stored="true" required="false" /> <field name="workflowItemId" type="integer" indexed="true" stored="true" required="false" /> |
Web User Interface Elements
Pageview and Download statistics
In the XMLUI, pageview and download statistics can be accessed from the lower end of the navigation menu. In the JSPUI, a view statistics button appears on the bottom of pages for which statistics are available.
If you are not seeing these links or buttons, it's likely that they are only enabled for administrators in your installation. Change the configuration parameter "authorization.admin.usage" in usage-statistics.cfg to false in order to make statistics visible for all repository visitors.
Home page
Starting from the repository homepage, the statistics page displays the top 10 most popular items of the entire repository.
Community home page
The following statistics are available for the community home pages:
- Total visits of the current community home page
- Visits of the community home page over a timespan of the last 7 months
- Top 10 country from where the visits originate
- Top 10 cities from where the visits originate
Collection home page
The following statistics are available for the collection home pages:
- Total visits of the current collection home page
- Visits of the collection home over a timespan of the last 7 months
- Top 10 country from where the visits originate
- Top 10 cities from where the visits originate
Item home page
The following statistics are available for the item home pages:
- Total visits of the item
- Total visits for the bitstreams attached to the item
- Visits of the item over a timespan of the last 7 months
- Top 10 country views from where the visits originate
- Top 10 cities from where the visits originate
Search Query Statistics
In the XMLUI, search query statistics can be accessed from the lower end of the navigation menu.
...
If you are using Discovery, note that clicking the facets also counts as a search, because clicking a facet sends a search query to the Discovery index.
Workflow Event Statistics
In the XMLUI, search query statistics can be accessed from the lower end of the navigation menu.
...
The dropdown on top of the page allows you to modify the time frame for the displayed statistics.
Architecture
The DSpace Statistics Implementation is a Client/Server architecture based on Solr for collecting usage events in the JSPUI and XMLUI user interface applications of DSpace. Solr runs as a separate webapplication and an instance of Apache Http Client is utilized to allow parallel requests to log statistics events into this Solr instance.
Configuration settings for Statistics
In the {dspace.dir}/config/modules/solr-statistics.cfg file review the following fields to make sure they are uncommented:
...
Property: | dbfile |
Example Value: | dbfile = ${dspace.dir}/config/GeoLiteCity.dat |
Informational Note: | The following referes to the GeoLiteCity database file utilized by the LocationUtils to calculate the location of client requests based on IP address. During the Ant build process (both fresh_install and update) this file will be downloaded from http://www.maxmind.com/app/geolitecity if a new version has been published or it is absent from your [dspace]/config directory. |
Property: | resolver.timeout |
Example Value: | resolver.timeout = 200 |
Informational Note: | Timeout in milliseconds for DNS resolution of origin hosts/IPs. Setting this value too high may result in solr exhausting your connection pool. |
Property: | useProxies |
Example Value: | useProxies = true |
Informational Note: | Will cause Statistics logging to look for X-Forward URI to detect clients IP that have accessed it through a Proxy service (e.g. the Apache mod_proxy). Allows detection of client IP when accessing DSpace. [Note: This setting is found in the DSpace Logging section of dspace.cfg] |
Property: | authorization.admin.usage |
Example Value: | authorization.admin.usage = true |
Informational Note: | When set to true, only general administrators, collection and community administrators are able to access the pageview and download statistics from the web user interface. As a result, the links to access statistics are hidden for non logged-in admin users. Setting this property to "false" will display the links to access statistics to anyone, making them publicly available. |
Property: | authorization.admin.search |
Example Value: | authorization.admin.search = true |
Informational Note: | When set to true, only system, collection or community administrators are able to access statistics on search queries. |
Property: | authorization.admin.workflow |
Example Value: | authorization.admin.workflow = true |
Informational Note: | When set to true, only system, collection or community administrators are able to access statistics on workflow events. |
Property: | logBots |
Example Value: | logBots = true |
Informational Note: | When this property is set to false, and IP is detected as a spider, the event is not logged. |
Pre-1.6 Statistics settings
Older versions of DSpace featured static reports generated from the log files. They still persist in DSpace today but are completely independent from the SOLR based statistics.
The following configuration parameters applicable to these reports can be found in dspace.cfg.
...
These fields are not used by the new 1.6 Statistics, but are only related to the Statistics from previous DSpace releases
Upgrade Process for Statistics
Example of rebuild and redeploy DSpace (only if you have configured your distribution in this manner)
...
Restart your webapps (Tomcat/Jetty/Resin)
Statistics Administration
Converting older DSpace logs into SOLR usage data
If you have upgraded from a previous version of DSpace, converting older log files ensures that you carry over older usage stats from before the upgrade.
Statistics Client Utility
The command line interface (CLI) scripts can be used to clean the usage database from additional spider traffic and other maintenance tasks. In DSpace 3.0, a script has been added to split up the monolithic SOLR core into individual cores each containing a year of statistics.
Statistics differences between DSpace 1.7.x and 1.8.0
Displayed file statistics bundle configurable
In DSpace 1.6.x & 1.7.x the file download statistics were generated without regard to the bundle in which the file was located. In DSpace 1.8.0 it is possible to configure the bundles for which the file statistics are to be shown by using the query.filter.bundles property. If required the old file statistics can also be upgraded to include the bundle name so that the old file statistics are fixed.
...
| Code Block |
|---|
#The -r is optional [dspace]/bin/dspace stats-util -b -r |
Statistics differences between DSpace 1.6.x and 1.7.0
SOLR optimization added
If required, the solr server can be optimized by running
...
More information on how these solr server optimizations work can be found here: http://wiki.apache.org/solr/SolrPerformanceFactors#Optimization_Considerations.
SOLR Autocommit
In DSpace 1.6.x, each solr event was committed to the solr server individually. For high load DSpace installations, this would result in a huge load of small solr commits resulting in a very high load on the solr server.
This has been resolved in dspace 1.7 by only committing usage events to the solr server every 15 minutes. This will result in a delay of the storage of a usage event of maximum 15 minutes. If required, this value can be altered by changing the maxTime property in the
| Code Block |
|---|
{dspace.dir}/solr/statistics/conf/solrconfig.xml
|
Web UI Statistics Modification (XMLUI Only)
Modifying the number of months, for which statistics are displayed
Modify line 205 in the StatisticsTransformer.java file
...
Related: DatasetTimeGenerator Javadoc
Custom Reporting - Querying SOLR Directly
When the web user interface does not offer you the statistics you need, you can greatly expand the reports by querying the SOLR index directly.
Resources
- http://www.lucidimagination.com/Community/Hear-from-the-Experts/Articles/Faceted-Search-Solr
- http://my.safaribooksonline.com/9781847195883/Cover
Examples
Top downloaded items by a specific user
Query:
| Code Block |
|---|
http://localhost:8080/solr/statistics/select?indent=on&version=2.2&start=0&rows=10&fl=*%2Cscore&qt=standard&wt=standard&explainOther=&hl.fl=&facet=true&facet.field=epersonid&q=type:0 |
...
| Code Block |
|---|
<lst name="facet_counts">
<lst name="facet_fields">
<lst name="epersonid">
<int name="66">1167</int>
<int name="117">251</int>
<int name="52">42</int>
<int name="19">36</int>
<int name="88">20</int>
<int name="112">18</int>
<int name="110">9</int>
<int name="96">0</int>
</lst>
</lst>
</lst>
|
Manually Installing/Updating GeoLite Database File
The GeoLite Database file (at [dspace]/config/GeoLiteCity.dat) is used by the Statistics engine to generate location/country based reports. (Note: If you are not using DSpace Statistics, this file is not needed.)
...
Overview
Content Tools