Page History
...
With the release of DSpace 1.6, new statistics software component was added. DSpace's use of Solr SOLR for statistics makes it possible to have a database of statistics. With this This in mind, there is the issue of the older log files and how a site can use them. The following command process is able to convert the existing log files and then import them for Solr SOLR use. The user will need to perform this only once.
The Log Converter command program converts log files from dspace.log into an intermediate format that can be inserted into SolrSOLR.
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="00cfea82d03385a6-19e64561-4d864b77-8b4995b1-3018e2462c2c2047a2bd8888"><ac:plain-text-body><![CDATA[ | Command used: | | ]]></ac:plain-text-body></ac:structured-macro> |
Java class: | org.dspace.statistics.util.ClassicDSpaceLogConverter | ||
Arguments short and long forms): | Description | ||
| Input file . Read from standard input if omitted or "-". | ||
| Output file . Written to standard output if omitted or "-". | ||
| Adds a wildcard at the end of input and output, so it would mean dspace.log* would be converted. (For example, the following files would be included because of this argument: dspace.log, dspace.log.1, dspace.log.2, dspace.log.3, etc.) | ||
| If the log files have been created with DSpace 1.6 | ||
| Display verbose output (helpful for debugging) | ||
| Help |
The Log Importer command loads into Solr the intermediate log files that have been created by the Log Converteraforementioned script into SOLR.
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="06aab97626db2cd2-612eb292-40e541c3-8c768a2d-1bdfc09d95c0eacc8aa5b0ff"><ac:plain-text-body><![CDATA[ | Command used: | | ]]></ac:plain-text-body></ac:structured-macro> |
Java class: | org.dspace.statistics.util.StatisticsImporter | ||
Arguments (short and long forms): | Description | ||
| input file . Read from standard input if omitted or "-". | ||
| Adds a wildcard at the end of the input, so it would mean dspace.log* would be imported | ||
| To skip the reverse DNS lookups that work out where a user is from. (The DNS lookup finds the information about the host from its IP address, such as geographical location, etc. This can be slow, and wouldn't work on a server not connected to the internet.) | ||
| Display verbose ouput (helpful for debugging) | ||
| For developers: allows you to import a log file from another system, so because the handles won't exist, it looks up random items in your local system to add hits to instead. | ||
| Help |
Although the DSpace Log Converter Convertor applies basic spider filtering (Googlebotgooglebot, Yahoo!, Slurp, MSNbotyahoo slurp, msnbot), it is far from complete. Please refer to Filtering and Pruning Spiders for spider removal operations, after converting your old logs.
...
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="5bd305e837feebe8-78ad7845-42784a45-b33a8873-a49f3347acc49860fc797e9d"><ac:plain-text-body><![CDATA[ | Command used: | | ]]></ac:plain-text-body></ac:structured-macro> |
Java class: | org.dspace.statistics.util.StatisticsClient | ||
Arguments (short and long forms): | Description | ||
| Update Spider IP Files from internet into | ||
| Delete Spiders in Solr By isBot Flag. Will prune out all records that have | ||
| Delete Spiders in Solr By IP Address. Will prune out all records that have IPs IP's that match spider IPs. | ||
| Update isBog Flag in Solr. Marks any records currently stored in statistics that have IP addresses matched in spiders files | ||
| Calls up this brief help table at command line. |
...
The usage of these options is open for the user to choose. , If you wish they want to keep spider entries entires in your their repository, they can just mark them using "-m
" and they will be excluded from statistics queries when "solr.statistics.query.filter.isBot = true
" in the dspace.cfg
.
If you they want to keep the spiders out of the Solr solr repository, they can run just use the "-i
" option and they will be removed immediately.
Wiki Markup |
---|
There are guards in place to control what can be defined as an IP range for a bot., in In {{\[dspace\]/config/spiders}}, spider IP address ranges have to be at least 3 subnet sections in length (12123.34123.56)123 and IP Ranges can only be on the smallest subnet \[123.123.123.0 - 123.123.123.255\]. If not, loading that row will cause exceptions in the DSpacedspace logs and exclude that IP entry. |
Routine
...
SOLR Index Maintenance
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="408535ed6af6a90c-052055ca-402c458f-9e0997b9-9c81da25fb0fc04a41d9b001"><ac:plain-text-body><![CDATA[ | Command used: | | ]]></ac:plain-text-body></ac:structured-macro> |
Java class: | org.dspace.statistics.util.StatisticsClient | ||
Arguments (short and long forms): | Description | ||
| Run maintenance on the Solr SOLR index. Recommended to run daily, to prevent your servlet container from running out of memory |
Notes:
The usage of this this option is strongly recommended. You , you should run this command script daily (from crontab or your system's scheduler), to prevent your servlet container from running out of memory.
Yearly solr sharding
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="9f6356a0-ae1b-45a1-87ad-c8e6ec333f18"><ac:plain-text-body><![CDATA[ | Command used: | | ]]></ac:plain-text-body></ac:structured-macro> |
Java class: | org.dspace.statistics.util.StatisticsClient | ||
Arguments (short and long forms): | Description | ||
-s or shard-solr-index | Splits the data in the main core up into a separate solr core for each year, this will upgrade the performance of the solr. |
Notes:
The usage of this this option is strongly recommended, you should run this script once a year at the start of a new year (from crontab or your system's scheduler), doing this will ensure that the solr performance stays solid.
Wiki Markup |
---|
This script will split up the current solr data core into one core for each year. Data cores are located in the \[dspace.dir\]/solr directory, these cores will NOT be defined in the solr.xml since they will be loaded on run time. The loading on run time is handled by the _static_ method located in the _org.dspace.statistics.SolrLogger_ class. These cores are stored in the _statisticYearCores_ list each time a query is made to the solr these cores are added as shards by the _addAdditionalSolrYearCores_ method. The cores share the configuration of the original _statistics_ core so updating by using ant isn't an issue. The actual sharding of the of the original solr core into the year cores is done in the _shardSolrIndex_ method in the _org.dspace.statistics.SolrLogger_ class, the sharding is done by first running a facet on the time to get the facets split by year. Once we have our years from our logs we query the main solr data server for all information on each year & download these as csv's. When we have all data for one year we upload it to the newly created core of that year by using the [update csv|http://wiki.apache.org/solr/UpdateCSV] handler. One all data of one year has been uploaded that data is removed from the main solr (by doing it this way if our solr crashes we do not need to start from scratch). |