Page History

...

The goal of the curation system ('CS') is to provide a simple, extensible , way to manage routine content operations on a repository. These operations are known to CS as 'tasks', and they can operate on any DSpaceObject (i.e. subclasses of DSpaceObject) - which means Communities, Collections, and Items - viz. core data model objects. Tasks may elect to work on only one type of DSpace object - typically an Item - and in this case they may simply ignore other data types (tasks have the ability to 'skip' objects for any reason). The DSpace core distribution will provide a number of useful tasks, but the system is designed to encourage local extension - tasks can be written for any purpose, and placed in any java package. This gives DSpace sites the ability to customize the behavior of their repository without having to alter - and therefore manage synchronization with - the DSpace source code. What sorts of activities are appropriate for tasks?

...

• apply a virus scan to item bitstreams (this will be our example below)
• profile a collection based on format types - good for identifying format migrations
• ensure a given set of metadata fields are present in every item, or even that they have particular values
• call a network service to enhance/replace/normalize an itemsitem's metadata or content
• ensure all item bitstreams are readable and their checksums agree with the ingest values

Since tasks have access to, and can modify, DSpace content, performing tasks is considered an administrative function to be available only to knowledgeable collection editors, repository administrators, sysadmins, etc. No tasks are exposed in the public interfaces.

Activation

Wiki MarkupFor CS to run a task, the code for the task must of course be included with other deployed code (to {{\[dspace\]/lib}}, WAR, etc) but it must also be declared and given a name. This is done via a configuration property in {{\[dspace\]/config/modules/curate.cfg}} as follows:

plugin.named.org.dspace.curate.CurationTask = \
org.dspace.curate.ProfileFormats = profileformats, \
org.dspace.curate.RequiredMetadata = requiredmetadata, \
org.dspace.curate.ClamScan = vscan

For each activated task, a key-value pair is added. The key is the fully qualified class name and the value is the taskname used elsewhere to configure the use of the task, as will be seen below. Note that the curate.cfg configuration file, while in the config directory, is located under 'modules'. The intent is that tasks, as well as any configuration they require, will be optional 'add-ons' to the basic system configuration. Adding or removing tasks has no impact on dspace.cfg.

Wiki MarkupFor many For many tasks, this activation configuration is all that will be required to use it. But for others, the task needs specific configuration itself. A concrete example is described below, but note that these task-specific configuration property files also reside in {{\[dspace\]/config/modules}}

Writing your own tasks

A task is just a java class that can contain arbitrary code, but it must have 2 properties:

...

 [dspace]/bin/dspace curate -t vscan -i 123456789/4 

As with other command-line tools, these invocations could be placed in a cron table and run on a fixed schedule, or run on demand by an administrator.

In the admin UI

In the XMLUI, there is a 'Curate' tab (appearing within the 'Edit  Community/Collection/Item') that exposes a drop-down list of configured tasks, with a button to 'perform' the task, or queue it for later operation (see section below). Not all activated tasks need appear in the Curate tab - you filter them by means of a configuration property. This property also permits you to assign to the task a more user-friendly name than the PluginManager _taskname_. The property resides in {{\[dspace\]/config/modules/curate.cfg}}:

ui.tasknames = \
     profileformats = Profile Bitstream Formats, \
     requiredmetadata = Check for Required Metadata

When a task is selected from the drop-down list and performed, the tab displays both a phrase interpreting the 'status code' of the task execution, and the 'result' message if any has been defined. When the task has been queued, an acknowlegement appears instead. You may configure the words used for status codes in curate.cfg (for clarity, language localization, etc):

The complete list of arguments:

-t taskname: name of task to perform
-T filename: name of file containing list of tasknames
-e epersonID: (email address) will be superuser if unspecified
-i identifier: Id of object to curate. May be (1) a handle (2) a workflow Id or (3) 'all' to operate on the whole repository
-q queue: name of queue to process - -i and -q are mutually exclusive
-v emit verbose output
-r - emit reporting to standard out

As with other command-line tools, these invocations could be placed in a cron table and run on a fixed schedule, or run on demand by an administrator.

In the admin UI

In the XMLUI, there is a 'Curate' tab (appearing within the 'Edit Community/Collection/Item') that exposes a drop-down list of configured tasks, with a button to 'perform' the task, or queue it for later operation (see section below). Not all activated tasks need appear in the Curate tab - you filter them by means of a configuration property. This property also permits you to assign to the task a more user-friendly name than the PluginManager taskname. The property resides in [dspace]/config/modules/curate.cfg:

ui.tasknames

ui.statusmessages = \
    -3 = Unknown Task, \
    -2 = No Status Set profileformats = Profile Bitstream Formats, \
     -1requiredmetadata = Error,Check \
for Required Metadata

When a task is selected from the drop-down list and performed, the tab displays both a phrase interpreting the 'status code' of the task execution, and the 'result' message if any has been defined. When the task has been queued, an acknowledgement appears instead. You may configure the words used for status codes in curate.cfg (for clarity, language localization, etc):

ui.statusmessages =    0 = Success, \
     1 = Fail, \
     2 = Skip, \
    -3 other= =Unknown InvalidTask, Status

In workflow

CS provides the ability to attach any number of tasks to standard DSpace workflows. Using a configuration file {{\[dspace\]/config/workflow-curation.xml}}, you can declaratively (without coding) wire tasks to any step in a workflow. An example:

\
    -2 = No Status Set, \
    -1 = Error, \
     0 = Success, \
     1 = Fail, \
     2 = Skip, \
     other = Invalid Status

In workflow

CS provides the ability to attach any number of tasks to standard DSpace workflows. Using a configuration file [dspace]/config/workflow-curation.xml, you can declaratively (without coding) wire tasks to any step in a workflow. An example:

<taskset-map>
   <mapping collection-handle="default" taskset="cautious" />
</taskset-map>
<tasksets>
  <taskset name="cautious">
    <flowstep name="step1">
      

<taskset-map>
   <mapping collection-handle="default" taskset="cautious" />
</taskset-map>
<tasksets>
  <taskset name="cautious">
    <flowstep name="step1">
      <task name="vscan">
        <workflow>reject</workflow>
        <notify on="fail">$flowgroup</notify>
        <notify on="fail">$colladmin</notify>
        <notify on="error">$siteadmin</notify>
      </task>
    </flowstep>
  </taskset>
</tasksets>

This markup would cause a virus scan to occur during step one of workflow for any collection, and automatically reject any submissions with infected files. It would further notify (via email) both the reviewers (step 1 group), and the collection administrators, if either of these are defined. If it could not perform the scan, the site administrator would be notified.

Wiki MarkupThe notifications use the same procedures that other workflow notifications do - namely email. There is a new email template defined for curation task use: {{\[dspace\]/config/emails/flowtask_notify}}. This may be language-localized or otherwise modified like any other email template.

Like configurable submission, you can assign these task rules per collection, as well as having a default for any collection.

...

Few assumptions are made by CS about what the 'outcome' of a task may be (if any) - it. could e.g. produce a report to a temporary file. , it could modify DSpace content silently, etc But the CS runtime does provide a few pieces of information that whenever a task can assignis performed:

Status Code

This was mentioned above. This is returned to CS whenever a task is called. The complete list of values:

      -3 NOTASK - CS could not find the requested task
      -2 UNSET  - task did not return a status code because it has not yet run
      -1 ERROR - task could not be performed
       0 SUCCESS - task performed successfully
       1 FAIL - task performed, but failed
       2 SKIP - task not performed due to object not being eligible

Result String

In the administrative UI, this code is translated into the word or phrase configured by the ui.statusmessages property (discussed above) for display.

Result String

The task may define a string indicating details of the outcome. The task may define a string indicating details of the outcome. This result is displayed, e.g. in the 'curation widget' described above:

...

CS does not interpret or assign result strings, the task does it. A task may not assign a result, but the 'best practice' for tasks is to assign one whenever possible.

Reporting Stream

This is not currently fully implemented, just writes to standard out. But if more details should be recorded, they can For very fine-grained information, a task may write to a reporting stream. This stream is sent to standard out, so is only available when running a task from the command line. Unlike the result string, there is no limit to the amount of data that may be pushed to this stream.

All 3 are The status code, and the result string are accessed (or set) by methods on the Curation object:

     Curator curator = new Curator();
     curator.addTask("vscan").curate(coll);
     int status = curator.getStatus("vscan");
     String result - curator.getResult("vscan");

Task Annotations

...

DSpace 1.7 bundles a few tasks and activates two (2) by default to demonstrate the use of the curation system. These may be removed (deactivated by means of configuration) if desired without affecting system integrity. Each task is briefly described here.

Bitstream Format Profiler

The task with the task name taskname 'formatprofiler' (in the admin UI it is labelled labeled "Profile Bitstream Formats") examines all the bitstreams in an item any and produces a table ("profile") which is assigned to the result string. It is activated by default, and is configured to display in the administrative UI. The result string has the layout:

     10 (K) Portable Network Graphics
     5  (S) Plain Text

...

    U  Unsupported
    K  Known
    S  Supported

The profiler will operate on any DSpace object. If the object is an item, then only that item's bitstreams are profiled; if a collection, all the bitsteams of all the items; if a community, all the items of all the collections of the community.

Required Metadata

profiler will operate on any DSpace object. If the object is an item, then only that item's bitstreams are profiled; if a collection, all the bitstreams of all the items; if a community, all the items of all the collections of the community.

Required Metadata

The 'requiredmetadata' task examines item metadata and determines whether fields that the web submission (input-forms.xml) marks as required are present. It sets the result string to indicate either that all required fields are present, or constructs a list of metadata elements that are required but missing. When the task is performed on an item, it will display the result for that item. When performed on a collection or community, the task be performed on each item, and will display the last item result. If all items in the community or collection have all required fields, that will be the last in the collection. If the task fails for any item (i.e. the item lacks all required fields), the process is halted. This way the results for the 'failed' items are not lostThe 'requiredmetadata' task examines item metadata and determines whether fields that the web submission (input-forms.xml) marks as required are present. It sets the result string to either indicate that all required fields are present, or constructs a list of metadata elements that are required but missing.

Virus Scan

The 'vscan' task performs a virus scan on the bitstreams of items using the ClamAV software product.
Clam AntiVirus is an open source (GPL) anti-virus toolkit for UNIX. A port for Windows is also available. The virus scanning curation task interacts with the ClamAV virus scanning service to scan the bitstreams contained in items, reporting on infection(s). Like other curation tasks, it can be run against a container or item, in the GUI or from the command line. It should be installed according to the documentation at http://www.clamav.net. It should not be installed in the dspace installation directory. You may install it on the same machine as your dspace installation, or on another machine which has been configured properly.

...

This plugin requires a ClamAV daemon installed and configured for TCP sockets. Instructions for installing ClamAV (http://www.clamav.net/doc/latest/clamdoc.pdf )

NOTICE: The following directions assume there is a properly installed and configured clamav daemon. Refer to links above for more information about ClamAV.
The Clam anti-virus database must be updated regularly to maintain the most current level of anti-virus protection. Please refer to the ClamAV documentation for instructions about maintaining the anti-virus database.

DSpace Configuration

Wiki MarkupIn {{\[dspace\]/config/modules/curate.cfg}}, activate the task:

• Add the plugin to the comma separated list of curation tasks.

...

ui.tasknames = \
profileformats = Profile Bitstream Formats, \
requiredmetadata = Check for Required Metadata, \
vscan = Scan for Viruses

Wiki MarkupIn {{\[dspace\]/config/modules}}, edit configuration file clamav.cfg:

service.host = 127.0.0.1
Change if not running on the same host as your DSpace installation.
service.port = 3310
Change if not using standard ClamAV port
socket.timeout = 120
Change if longer timeout needed
scan.failfast = false
Change only if items have large numbers of bitstreams

...