DSpace System Documentation: System Administration

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

...

Table of Contents:

Community and Collection Structure Importer

This CLI tool gives you the ability to import a community and collection structure directory from a source XML file.

...

[dspace]/bin/dspace packager -e [user-email] -p [parent-handle] -t [packager-name] /full/path/to/package

Wiki MarkupWhere _\[user-email\]_ is the e-mail address of the E-Person under whose authority this runs; _\[parent-handle\]_ is the Handle of the Parent Object into which the package is ingested, _\[packager-name\]_ is the plugin name of the package ingester to use, and _/full/path/to/package_ is the path to the file to ingest (or _"-"_ to read from the standard input).

Here is an example that loads a PDF file with internal metadata as a package:

...

Restoring/Replacing using Packages

...

[dspace]/bin/dspace packager -d -e [user-email] -i [handle] -t [packager-name] [file-path]

Wiki MarkupWhere _\[user-email\]_ is the e-mail address of the E-Person under whose authority this runs; _\[handle\]_ is the Handle of the Object to disseminate; _\[packager-name\]_ is the plugin name of the package disseminator to use; and _\[file-path\]_ is the path to the file to create (or _"-"_ to write to the standard output). For example:

 [dspace]/bin/dspace packager -d -t METS -e admin@myu.edu -i 4321/4567 4567.zip

...

The above code will export the object of the given handle (4321/4567) into a METS file named "4567.zip". In addition it would export all children objects to the same directory as the "4567.zip" file.

METS packages

Archival Information Packages (AIPs)

As of Since DSpace 1.4 release, the software includes a package disseminator and matching ingester for the DSpace METS SIP (Submission Information Package) format. They were created to help end users prepare sets of digital resources and metadata for submission to the archive using well-defined standards such as METS, MODS, and PREMIS. The plugin name is METS by default, and it uses MODS for descriptive metadata.7, DSpace now can backup and restore all of its contents as a set of AIP Files. This includes all Communities, Collections, Items, Groups and People in the system.

This feature came out of a requirement for DSpace to better integrate with DuraCloud (The DSpace METS SIP profile is available at: http://www.dspace.org/standards/METS/SIP/profilev1p0/metsipv1p0.pdf .

Item Importer and Exporter

DSpace has a set of command line tools for importing and exporting items in batches, using the DSpace simple archive format. The tools are not terribly robust, but are useful and are easily modified. They also give a good demonstration of how to implement your own item importer if desired.

DSpace Simple Archive Format

duracloud.org), and other backup storage systems. One of these requirements is to be able to essentially "backup" local DSpace contents into the cloud (as a type of offsite backup), and "restore" those contents at a later time.

Essentially, this means DSpace can export the entire hierarchy (i.e. bitstreams, metadata and relationships between Communities/Collections/Items) into a relatively standard format (a METS-based, AIP format). This entire hierarchy can also be re-imported into DSpace in the same format (essentially a restore of that content in the same or different DSpace installation).

For more information, see the section on AIP backup & Restore for DSpace.

METS packages

Since DSpace 1.4 release, the software includes a package disseminator and matching ingester for the DSpace METS SIP (Submission Information Package) format. They were created to help end users prepare sets of digital resources and metadata for submission to the archive using well-defined standards such as METS, MODS, and PREMIS. The plugin name is METS by default, and it uses MODS for descriptive metadata.

The DSpace METS SIP profile is available at: https://wiki.duraspace.org/display/DSPACE/DSpaceMETSSIPProfile

Item Importer and Exporter

DSpace has a set of command line tools for importing and exporting items in batches, using the DSpace simple archive format. The tools are not terribly robust, but are useful and are easily modified. They also give a good demonstration of how to implement your own item importer if desired.

DSpace Simple Archive Format

The basic concept behind the DSpace's simple archive format is to create an archive, which is directory full of items, The basic concept behind the DSpace's simple archive format is to create an archive, which is directory full of items, with a subdirectory per item. Each item directory contains a file for the item's descriptive metadata, and the files that make up the item.

archive_directory/
    item_000/
        dublin_core.xml         -- qualified Dublin Core metadata for metadata fields belonging to the dc schema
        metadata_[prefix].xml   -- metadata in another schema, the prefix is the name of the schema as registered with the metadata registry
        contents                -- text file containing one line per filename
        file_1.doc              -- files to be added as bitstreams to the item
        file_2.pdf
    item_001/
        dublin_core.xml
        contents
        file_1.png
        ...

Wiki MarkupThe _dublinThe dublin_core.xml_ or _metadata_\[prefix\].xml_file has the following format, where each metadata element has it's own entry within a _<dcvalue>_ tagset. There are currently three tag attributes available in the _<dcvalue>_ tagset:

• <element> - the Dublin Core element
• <qualifier> - the element's qualifier
• <language> - (optional)ISO language code for element

  Code Block
  <dublin_core> <dcvalue element="title" qualifier="none">A Tale of Two Cities</dcvalue> <dcvalue element="date" qualifier="issued">1990</dcvalue> <dcvalue element="title" qualifier="alternate" language="fr">J'aime les Printemps</dcvalue> </dublin_core>

  (Note the optional language tag attribute which notifies the system that the optional title is in French.)

...

The bitstream name may optionally be followed by any of the sequencefollowing:

• \tbundle:BUNDLENAME
• \tpermissions:PERMISSIONS
• \tdescription:DESCRIPTION
• \tprimary:true

Where bundlenamewhere '\t' is the tab character.

and 'bundlenameBUNDLENAME' is replaced by the name of the bundle to which the bitstream should be added. If no bundle is specified, the bitstream will be added to the 'ORIGINAL' bundle.

Wiki Markup
Configuring _metadata-\[prefix\].xml_ for Different Schema

Without specifying the bundle, items will go into the default bundle, ORIGINAL.

'PERMISSIONS'  is text with the following format: -[r|w] 'group name'

'DESCRIPTION' is text of the files description.

Primary is used to specify the primary bitstream.

Configuring metadata-[prefix].xml for Different Schema

It is possible to use other Schema such as EAD, VRA Core, etc. Make sure you have defined the new scheme in the DSpace Metada Schema Registry.

1. Create a separate file for the other schema named "metadata{-[prefix}].xml_", where the {prefix} is replaced with the schema's prefix.
2. Inside the xml file use the dame Dublin Core syntax, but on the <dublin_core> element include the attribute "schema={prefix}".
3. Here is an example for ETD metadata, which would be in the file "metadata_etd.xml":

   Code Block
   <xml<?xml version="1.0" encoding="UTF-8"?> <dublin_core schema="etd"> <dcvalue element="degree" qualifier="department">Computer Science</dcvalue> <dcvalue element="degree" qualifier="level">Masters</dcvalue> <dcvalue element="degree" qualifier="grantor">Texas A & M</dcvalue> </dublin_core>

Importing Items

Before running the item importer over items previously exported from a DSpace instance, please first refer to Transferring Items Between DSpace Instances.

...

• eperson
• Collection ID (either Handle (e.g. 123456789/14) or Database ID (e.g. 2)
• Source directory where the items reside
• Mapfile. Since you don't have one, you need to determine where it will be (e.g. /Import/Col_14/mapfile)
  At the command line:

[dspace]/bin/dspace import --add --eperson=joe@user.com --collection=CollectionID --source=items_dir --mapfile=mapfile

or by using the short form: Wiki Markup_\

[dspace

...

]/bin/dspace import 

...

-a 

...

-e joe@user.com 

...

-c CollectionID 

...

-s items_dir 

...

-m mapfile

...

The above command would cycle through the archive directory's items, import them, and then generate a map file which stores the mapping of item directories to item handles. SAVE THIS MAP FILE. Using the map file you can use it for replacing or deleting (unimporting) the file.

Testing. You can add --test (or -t) to the command to simulate the entire import process without actually doing the import. This is extremely useful for verifying your import files before doing the actual import.

...

Replacing existing items is relatively easy. Remember that mapfile you were supposed to save? Now you will use it. The command (in short form): Wiki Markup_\

[dspace

...

]/bin/dspace import 

...

-r 

...

-e joe@user.com 

...

-c collectionID 

...

-s items_dir 

...

-m mapfile

...

Long form: Wiki Markup_\

[dspace

...

]/bin/dspace import 

...

--replace 

...

--eperson=joe@user.com 

...

--collection=collectionID 

...

--source=items_dire 

...

--mapfile=mapfile

...

Deleting or Unimporting Items in a Collection

You are able to unimport or delete items provided you have the mapfile. Remember that mapfile you were supposed to save? The command is (in short form): Wiki Markup_\

[dspace

...

]/bin/dspace import 

...

-d 

...

-m mapfile

...

In long form: Wiki Markup_\

[dspace]/bin/dspace import 

...

--delete 

...

--mapfile mapfile

...

...

Other Options

• Workflow. The importer usually bypasses any workflow assigned to a collection. But add the --workflow

...

• (-w

...

• ) argument will route the imported items through the workflow system.

• Templates. If you have templates that have constant data and you wish to apply that data during batch importing, add the --template

...

• (-p

...

• ) argument.

• Resume. If, during importing, you have an error and the import is aborted, you can use the --resume

...

• (-R

...

• ) flag that you can try to resume the import where you left off after you fix the error.

Exporting Items

The item exporter can export a single item or a collection of items, and creates a DSpace simple archive for each item to be exported.

Exporting a Collection

To export a collection's items you type at the CLI:unmigrated-wiki-markup

...

[dspace

...

]/bin/dspace export 

...

--type=COLLECTION 

...

--id=collID 

...

--dest=dest_dir 

...

--number=seq_num

Short form: Wiki Markup_\

[dspace

...

]/bin/dspace export 

...

-t COLLECTION 

...

-d CollID or Handle 

...

-d /path/to/destination 

...

-n Some_number

...

Exporting a Single Item

The keyword COLLECTION means that you intend to export an entire collection. The ID can either be the database ID or the handle. The exporter will begin numbering the simple archives with the sequence number that you supply. To export a single item use the keyword ITEM and give the item ID as an argument: Wiki Markup_\[dspace\

[dspace]/bin/dspace export 

...

--type=ITEM 

...

--id=itemID 

...

--dest=dest_dir 

...

--number=seq_num

...

Short form: Wiki Markup_\

[dspace

...

]/bin/dspace export 

...

-t ITEM 

...

-i itemID or Handle 

...

-d /path/to/destination 

...

-n some_

...

number

Each exported item will have an additional file in its directory, named 'handle'. This will contain the handle that was assigned to the item, and this file will be read by the importer so that items exported and then imported to another machine will retain the item's original handle.

The -m Arugment Argument

Using the -m argument will export the item/collection and also perform the migration step. It will perform the same process that the next section Transferring Items Between DSpace Instances performs. We recommend that the next section be read in conjunction with this flag being used.

Transferring Items Between DSpace Instances

Migration of Data
Where items are to be moved between DSpace instances (for example from a test DSpace into a production DSpace) the item exporter and item importer can be used in conjunction with a script to assist in this process.

...

• date.accessioned
• date.available
• date.issued
• description.provenance
• format.extent
• format.mimetype
• identifier.uri
  In order to avoid duplication of this metadata, run

[dspace]/bin/dspace_migrate </path/to/exported item directory>

prior to running the item importer. This will remove the above metadata items, except for date.issued - if the item has been published or publicly distributed before and identifier.uri - if it is not the handle, from the dublin_core.xml file and remove all handle files. It will then be safe to run the item exporter.

...

For metadata, ItemUpdate can perform 'add' and 'delete' actions on specified metadta metadata elements. For bitstreams, 'add' and 'delete' are similarly available. All these actions can be combined in a single batch run.

...

One probable scenario for using this tool is where there is an external primary data source for which the DSpace instance is a secondary or down-stream system. Metadata and/or bitstream content changes in the primary system can be exported to the simple archive format to be used by ItemUpdate to synchronize the changes.

Wiki MarkupA note on terminology: *item* refers to a DSpace item. *metadata element* refers generally to a qualified or unqualified element in a schema in the form _\[schema\].\[element\].\[qualifier\]_ or _\[schema\].\[element\]_ and occasionally in a more specific way to the second part of that form. *metadata field* refers to a specific instance pairing a metadata element to a value.

DSpace simple Archive Format

...

The optional suppress_undo file is a flag to indicate that the 'undo archive' should not be written to disk. This file is usually written by the application in an undo archive to prevent a recursive undo. This file is an addition to the Archive format specifically for ItemUpdate.

ItemUpdate Commands

...

CLI Examples

Adding Metadata: Wiki Markup_\

[dspace

...

]/bin/dspace itemupdate 

...

-e joe@user.com 

...

-s 

...

[path/to/archive

...

] 

...

-a dc.description

...

This will add from your archive the dc element description based on the handle from the URI (since the -i argument wasn't used).

Registering (Not Importing) Bitstreams

...

• -r indicates this is a file to be registered
• -s n indicates the asset store number (n)
• -f filepath indicates the path and name of the content file to be registered (filepath)
• \t is a tab character
• bundle:bundlename is an optional bundle name
• Wiki Markup_permissions: \ -\[r\|w\] 'group name'_ is an optional read or write permission that can be attached to the bitstream
• description: some text is an optional description field to add to the file
  The bundle, that is everything after the filepath, is optional and is normally not used.

The command line for registration is just like the one for regular import: Wiki Markup_\

[dspace

...

]/bin/dspace import 

...

-a 

...

-e joe@user.com 

...

-c collectionID 

...

-s items_dir 

...

-m mapfile

...

(or by using the long form) Wiki Markup_\

[dspace

...

]/bin/dspace import 

...

--add 

...

--eperson=joe@user.com 

...

--collection=collectionID 

...

--source=items_dir 

...

--map=mapfile

...

The --workflow and --test flags will function as described in Importing Items.

The --delete flag will function as described in Importing Items but the registered content files will not be removed from storage. See Deleting Registered Items.

The --replace flag will function as described in Importing Items but care should be taken to consider different cases and implications. With old items and new items being registered or ingested normally, there are four combinations or cases to consider. Foremost, an old registered item deleted from DSpace using --replace will not be removed from the storage. See Deleting Registered Items. where is resides. A new item added to DSpace using --replace will be ingested normally or will be registered depending on whether or not it is marked in the contents files with the -r.

Internal Identification and Retrieval of Registered Items

...

If a registered item is deleted from DSpace, either interactively or by using the -delete or -replace flags described in Importing Items, the item will disappear from DSpace but it's registered content files will remain in place just as they were prior to registration. Bitstreams not registered but added by DSpace as part of registration, such as license.txt files, will be deleted.

METS Tools

...

The experimental (incomplete) METS export tool The experimental (incomplete) METS export tool writes DSpace items to a filesystem with the metadata held in a more standard format based on METS.

The Export Tool

This tool is obsolete, and does not export a complete AIP. It's use is strongly deprecated.

...

<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="401ab3ac-ae97-4823-a94e-c7525e1554a0"><ac:plain-text-body><![CDATA[

...

Command used:

...

_[dspace]_/bin/dspace mets-export

...

]]></ac:plain-text-body></ac:structured-macro>

. Its use is strongly discouraged. Please use the Package Importer and Exporter instead.

The following are examples of the types of process the METS tool can provide.

Exporting an individual item. From the CLI:

[dspace]/bin/dspace org.dspace.app.mets.METSExport -i [handle] -d /path/to/destination

Exporting a collection. From the CLI:

[dspace]/bin/dspace org.dspace.app.mets.METSExport -c [handle] -d /path/to/destination

Exporting all the items in DSpace. From the CLI:

[dspace]/bin/dspace 

...

Java class:

org.dspace.app.mets.METSExport

...

Arguments short and (long) forms:

...

Description

...

a or -all

...

Export all items in the archive.

...

c or -collection

...

Handle of the collection to export.

...

d or -destination

...

Destination directory.

...

i or -item

...

Handle of the item to export.

...

h or -help

...

Help

 -a -d /path/to/destination

Limitations

• No corresponding import tool yet
• No structmap section
• Some technical metadata not written, e.g. the primary bitstream in a bundle, original filenames or descriptions.
• Only the MIME type is stored, not the (finer grained) bitstream format.
• Dublin Core to MODS mapping is very simple, probably needs verification

MediaFilters: Transforming DSpace Content

DSpace can apply filters to content/bitstreams, creating new content. Filters are included that extract text for full-text searching, and create thumbnails for items that contain images. The media filters are controlled by the MediaFilterManager which traverses the asset store, invoking the MediaFilter or FormatFilter classes on bitstreams. The media filter plugin configuration filter.plugins in dspace.cfg contains a list of all enabled media/format filter plugins (see Configuring Media Filters for more information). The media filter system is intended to be run from the command line (or regularly as a cron task):

[dspace]/bin/dspace filter-media

With no options, this traverses the asset store, applying media filters to bitstreams, and skipping bitstreams that have already been filtered.

Available Command-Line Options:

• Help : [dspace]/bin/dspace filter-media -h
  • Display help message describing all command-line options.
• Force mode : [dspace]/bin/dspace filter-media -f
  • Apply filters to ALL bitstreams, even if they've already been filtered. If they've already been filtered, the previously filtered content is overwritten.
• Identifier mode : [dspace]/bin/dspace filter-media -i 123456789/2
  • Restrict processing to the community, collection, or item named by the identifier - by default, all bitstreams of all items in the repository are processed. The identifier must be a Handle, not a DB key. This option may be combined with any other option.
• Maximum mode : [dspace]/bin/dspace filter-media -m 1000
  • Suspend operation after the specified maximum number of items have been processed - by default, no limit exists. This option may be combined with any other option.
• No-Index mode : [dspace]/bin/dspace filter-media -n
  • Suppress index creation - by default, a new search index is created for full-text searching. This option suppresses index creation if you intend to run index-update elsewhere.
• Plugin mode : [dspace]/bin/dspace filter-media -p "PDF Text Extractor","Word Text Extractor"
  • Apply ONLY the filter plugin(s) listed (separated by commas). By default all named filters listed in the filter.plugins field of dspace.cfg are applied. This option may be combined with any other option. WARNING: multiple plugin names must be separated by a comma (i.e. ',') and NOT a comma followed by a space (i.e. ', ').
• Skip mode : [dspace]/bin/dspace filter-media -s 123456789/9,123456789/100
  • SKIP the listed identifiers (separated by commas) during processing. The identifiers must be Handles (not DB Keys). They may refer to items, collections or communities which should be skipped. This option may be combined with any other option. WARNING: multiple identifiers must be separated by a comma (i.e. ',') and NOT a comma followed by a space (i.e. ', ').
  • NOTE: If you have a large number of identifiers to skip, you may maintain this comma-separated list within a separate file (e.g. filter-skiplist.txt). Use the following format to call the program. Please note the use of the "grave" or "tick" (`_) symbol and do not use the single quotation. _
    • [dspace]/bin/dspace filter-media -s `less filter-skiplist.txt`
• Verbose mode : [dspace]/bin/dspace filter-media -v
  • Verbose mode - print all extracted text and other filter details to STDOUT.
    Adding your own filters is done by creating a class which implements the org.dspace.app.mediafilter.FormatFilter interface. See the Creating a new Media Filter topic and comments in the source file FormatFilter.java for more information. In theory filters could be implemented in any programming language (C, Perl, etc.) However, they need to be invoked by the Java code in the Media Filter class that you create.

Sub-Community Management

DSpace provides an administrative tool‚ 'CommunityFiliator'‚ for managing community sub-structure. Normally this structure seldom changes, but prior to the 1.2 release sub-communities were not supported, so this tool could be used to place existing pre-1.2 communities into a hierarchy. It has two operations, either establishing a community to sub-community relationship, or dis-establishing an existing relationship.

The familiar parent/child metaphor can be used to explain how it works. Every community in DSpace can be either a 'parent' community‚ meaning it has at least one sub-community, or a 'child' community‚ meaning it is a sub-community of another community, or both or neither. In these terms, an 'orphan' is a community that lacks a parent (although it can be a parent); 'orphans' are referred to as 'top-level' communities in the DSpace user-interface, since there is no parent community 'above' them. The first operation‚ establishing a parent/child relationship - can take place between any community and an orphan. The second operation - removing a parent/child relationship‚ will make the child an orphan.

Set a parent/child relationship, issue the following at the CLI:

dspace community-filiator --set --parent=parentID --child=childID

(or using the short form)

[dspace]/bin/dspace community-filiator -s -p parentID -c childID

where 's' or '-set' means establish a relationship whereby the community identified by the '-p' parameter becomes the parent of the community identified by the '-c' parameter. Both the 'parentID' and 'childID' values may be handles or database IDs.

The reverse operation looks like this:

[dspace]/bin/dspace community-filiator --remove --parent=parentID --child=childID

(or using the short form)

[dspace]/bin/dspace community-filiator -r -p parentID -c childID

where 'r' or '-remove' means dis-establish the current relationship in which the community identified by 'parentID' is the parent of the community identified by 'childID'. The outcome will be that the 'childID' community will become an orphan, i.e. a top-level community.

If the required constraints of operation are violated, an error message will appear explaining the problem, and no change will be made. An example in a removal operation, where the stated child community does not have the stated parent community as its parent: "Error, child community not a child of parent community".

It is possible to effect arbitrary changes to the community hierarchy by chaining the basic operations together. For example, to move a child community from one parent to another, simply perform a 'remove' from its current parent (which will leave it an orphan), followed by a 'set' to its new parent.

It is important to understand that when any operation is performed, all the sub-structure of the child community follows it. Thus, if a child has itself children (sub-communities), or collections, they will all move with it to its new 'location' in the community tree.

Batch Metadata Editing

DSpace provides a batch metadata editing tool. The batch editing tool is able to produce a comma delimited file in the CVS format. The batch editing tool facilitates the user to perform the following:

• Batch editing of metadata (e.g. perform an external spell check)
• Batch additions of metadata (e.g. add an abstract to a set of items, add controlled vocabulary such as LCSH)
• Batch find and replace of metadata values (e.g. correct misspelled surname across several records)
• Mass move items between collections
• Enable the batch addition of new items (without bitstreams) via a CSV file
• Re-order the values in a list (e.g. authors)

  Export Function

The following table summarizes the basics.

Exporting Process

To run the batch editing exporter, at the command line:

[dspace]/bin/dspace metadata-export -f name_of_file.csv -i 1023/24

Example:

[dspace]/bin/dspace metadata-export -f /batch_export/col_14.csv -i /1989.1/24

In the above example we have requested that a collection, assigned handle '1989.1/24' export the entire collection to the file 'col_14.cvs' found in the '/batch_export' directory.

Import Function

The following table summarizes the basics.

The following are examples of the types of process the METS tool can provide.

Exporting an individual item. From the CLI:

__\[dspace\]___/bin/dspace mets-export \-i ___\[handle\] \-d /path/to/destination__

Exporting a collection. From the CLI:

_\[dspace\]/bin/dspace mets-export \-c \[handle\] \-d /path/to/destination_

Exporting all the items in DSpace. From the CLI:

_\[dspace\]/bin/dspace mets-export \-a \-d /path/to/destination_

The AIP Format

Note that this tool is deprecated, and the output format is not a true AIP

Each exported item is written to a separate directory, created under the base directory specified in the command-line arguments, or in the current directory if --destination is omitted. The name of each directory is the Handle, URL-encoded so that the directory name is 'legal'.

Within each item directory is a mets.xml file which contains the METS-encoded metadata for the item. Bitstreams in the item are also stored in the directory. Their filenames are their MD5 checksums, firstly for easy integrity checking, and also to avoid any problems with 'special characters' in the filenames that were legal on the original filing system they came from but are illegal in the server filing system. The mets.xml file includes XLink pointers to these bitstream files.

An example AIP might look like this:

• hdl%3A123456789%2F8/
  • mets.xml – METS metadata
  • 184BE84F293342 – bitstream
  • 3F9AD0389CB821
  • 135FB82113C32D
    The contents of the METS in the mets.xml file are as follows:

• A dmdSec (descriptive metadata section) containing the item's metadata in Metadata Object Description Schema (MODS) XML. The Dublin Core descriptive metadata is mapped to MODS since there is no official qualified Dublin Core XML schema in existence as of yet, and the Library Application Profile of DC that DSpace uses includes some qualifiers that are not part of the DCMI Metadata Terms.
• An amdSec (administrative metadata section), which contains the a rights metadata element, which in turn contains the base64-encoded deposit license (the license the submitter granted as part of the submission process).
• A fileSec containing a list of the bitstreams in the item. Each bundle constitutes a fileGrp. Each bitstream is represented by a file element, which contains an FLocat element with a simple XLink to the bitstream in the same directory as the mets.xml file. The file attributes consist of most of the basic technical metadata for the bitstream. Additionally, for those bitstreams that are thumbnails or text extracted from another bitstream in the item, those 'derived' bitstreams have the same GROUPID as the bitstream they were derived from, in order that clients understand that there is a relationship.The OWNERID of each file is the 'persistent' bitstream identifier assigned by the DSpace instance. The ID and GROUPID attributes consist of the item's Handle, together with the bitstream's sequence ID, which underscores used in place of dots and slashes. For example, a bitstream with sequence ID 24, in the item hdl:123.456/789 will have the ID123_456_789_24. This is because ID and GROUPID attributes must be of type xsd:id.

Limitations

• No corresponding import tool yet
• No structmap section
• Some technical metadata not written, e.g. the primary bitstream in a bundle, original filenames or descriptions.
• Only the MIME type is stored, not the (finer grained) bitstream format.
• Dublin Core to MODS mapping is very simple, probably needs verification

MediaFilters: Transforming DSpace Content

DSpace can apply filters to content/bitstreams, creating new content. Filters are included that extract text for full-text searching, and create thumbnails for items that contain images. The media filters are controlled by the MediaFilterManager which traverses the asset store, invoking the MediaFilter or FormatFilter classes on bitstreams. The media filter plugin configuration filter.plugins in dspace.cfg contains a list of all enabled media/format filter plugins (see Configuring Media Filters for more information). The media filter system is intended to be run from the command line (or regularly as a cron task):

[dspace]/bin/filter-media

With no options, this traverses the asset store, applying media filters to bitstreams, and skipping bitstreams that have already been filtered.

Available Command-Line Options:

• Wiki Markup
  *Help* : _\[dspace\]/bin/dspace filter-media \-h_

  • Display help message describing all command-line options.

• Wiki Markup
  *Force mode* : _\[dspace\]/bin/dspace filter-media \-f_

  • Apply filters to ALL bitstreams, even if they've already been filtered. If they've already been filtered, the previously filtered content is overwritten.

• Wiki Markup
  *Identifier mode* : _\[dspace\]/bin/dspace filter-media \-i 123456789/2_

  • Restrict processing to the community, collection, or item named by the identifier - by default, all bitstreams of all items in the repository are processed. The identifier must be a Handle, not a DB key. This option may be combined with any other option.

• Wiki Markup
  *Maximum mode* : _\[dspace\]/bin/dspace filter-media \-m 1000_

  • Suspend operation after the specified maximum number of items have been processed - by default, no limit exists. This option may be combined with any other option.

• Wiki Markup
  *No-Index mode* : _\[dspace\]/bin/dspace filter-media \-n_

  • Suppress index creation - by default, a new search index is created for full-text searching. This option suppresses index creation if you intend to run index-update elsewhere.

• Wiki Markup
  *Plugin mode* : _\[dspace\]/bin/dspace filter-media \-p "PDF Text Extractor","Word Text Extractor"_

  • Apply ONLY the filter plugin(s) listed (separated by commas). By default all named filters listed in the filter.plugins field of dspace.cfg are applied. This option may be combined with any other option. WARNING: multiple plugin names must be separated by a comma (i.e. ',') and NOT a comma followed by a space (i.e. ', ').

• Wiki Markup
  *Skip mode* : _\[dspace\]/bin/dspace filter-media \-s 123456789/9,123456789/100_

  • SKIP the listed identifiers (separated by commas) during processing. The identifiers must be Handles (not DB Keys). They may refer to items, collections or communities which should be skipped. This option may be combined with any other option. WARNING: multiple identifiers must be separated by a comma (i.e. ',') and NOT a comma followed by a space (i.e. ', ').
  • NOTE: If you have a large number of identifiers to skip, you may maintain this comma-separated list within a separate file (e.g. filter-skiplist.txt). Use the following format to call the program. Please note the use of the "grave" or "tick" (`_) symbol and do not use the single quotation. _

    • Wiki Markup
      _\[dspace\]/bin/dspace filter-media \-s `less filter-skiplist.txt`_

• Wiki Markup
  *Verbose mode* : _\[dspace\]/bin/dspace filter-media \-v_

  • Verbose mode - print all extracted text and other filter details to STDOUT.
    Adding your own filters is done by creating a class which implements the org.dspace.app.mediafilter.FormatFilter interface. See the Creating a new Media Filter topic and comments in the source file FormatFilter.java for more information. In theory filters could be implemented in any programming language (C, Perl, etc.) However, they need to be invoked by the Java code in the Media Filter class that you create.

Sub-Community Management

DSpace provides an administrative tool—'CommunityFiliator'—for managing community sub-structure. Normally this structure seldom changes, but prior to the 1.2 release sub-communities were not supported, so this tool could be used to place existing pre-1.2 communities into a hierarchy. It has two operations, either establishing a community to sub-community relationship, or dis-establishing an existing relationship.

The familiar parent/child metaphor can be used to explain how it works. Every community in DSpace can be either a 'parent' community—meaning it has at least one sub-community, or a 'child' community—meaning it is a sub-community of another community, or both or neither. In these terms, an 'orphan' is a community that lacks a parent (although it can be a parent); 'orphans' are referred to as 'top-level' communities in the DSpace user-interface, since there is no parent community 'above' them. The first operation—establishing a parent/child relationship - can take place between any community and an orphan. The second operation - removing a parent/child relationship—will make the child an orphan.

Set a parent/child relationship, issue the following at the CLI:

dsrun org.dspace.administer.CommunityFiliator --set --parent=parentID --child=childID

(or using the short form)

_\[dspace\]/bin dspace community-filiator \-s \-p parentID \-c childID_

where 's' or '-set' means establish a relationship whereby the community identified by the '-p' parameter becomes the parent of the community identified by the '-c' parameter. Both the 'parentID' and 'childID' values may be handles or database IDs.

The reverse operation looks like this:

_\[dspace\]/bin dspace community-filiator \--remove \--parent=parentID \--child=childID_

(or using the short form)

_\[dspace\]/bin dspace community-filiator \-r \-p parentID \-c childID_

where 'r' or '-remove' means dis-establish the current relationship in which the community identified by 'parentID' is the parent of the community identified by 'childID'. The outcome will be that the 'childID' community will become an orphan, i.e. a top-level community.

If the required constraints of operation are violated, an error message will appear explaining the problem, and no change will be made. An example in a removal operation, where the stated child community does not have the stated parent community as its parent: "Error, child community not a child of parent community".

It is possible to effect arbitrary changes to the community hierarchy by chaining the basic operations together. For example, to move a child community from one parent to another, simply perform a 'remove' from its current parent (which will leave it an orphan), followed by a 'set' to its new parent.

It is important to understand that when any operation is performed, all the sub-structure of the child community follows it. Thus, if a child has itself children (sub-communities), or collections, they will all move with it to its new 'location' in the community tree.

Batch Metadata Editing

DSpace provides a batch metadata editing tool. The batch editing tool is able to produce a comma delimited file in the CVS format. The batch editing tool facilitates the user to perform the following:

• Batch editing of metadata (e.g. perform an external spell check)
• Batch additions of metadata (e.g. add an abstract to a set of items, add controlled vocabulary such as LCSH)
• Batch find and replace of metadata values (e.g. correct misspelled surname across several records)
• Mass move items between collections
• Enable the batch addition of new items (without bitstreams) via a CSV file
• Re-order the values in a list (e.g. authors)

  Export Function

The following table summarizes the basics.

Exporting Process

To run the batch editing exporter, at the command line:

\_\[dspace\]/bin/dspace metadata-export \-f name_of_file.csv \-i 1023/24 _

Example:

_\[dspace\]/bin/dspace metadata-export \-f /batch_export/col_14.csv \-i /1989.1/24_

In the above example we have requested that a collection, assigned handle '1989.1/24' export the entire collection to the file 'col_14.cvs' found in the '/batch_export' directory.

Import Function

The following table summarizes the basics.

Silent Mode should be used carefully. It is possible (and probable) that you can overlay the wrong data and cause irreparable damage to the database.

Importing Process

To run the batch importer, at the command line:

\_\[dspace\]/bin/dspace metadata-import \-f name_of_file.csv _

Example

_\[dspace\]/bin/dspace metadata-import \-f /dImport/col_14.csv_

If you are wishing to upload new metadata without bistreams, at the command line:

_\[dspace\]/bin/dspace/metadata-import \-f /dImport/new_file.csv \-e joe@user.com \-w \-n \-t_

In the above example we threw in all the arguments. This would add the metadata and engage the workflow, notification, and templates to all be applied to the items that are being added.

The CSV Files

The csv files that this tool can import and export abide by the RFC4180 CSV format http://www.ietf.org/rfc/rfc4180.txt. This means that new lines, and embedded commas can be included by wrapping elements in double quotes. Double quotes can be included by using two double quotes. The code does all this for you, and any good csv editor such as Excel or OpenOffice will comply with this convention.

File Structure. The first row of the csv must define the metadata values that the rest of the csv represents. The first column must always be "id" which refers to the item'id. All other columns are optional. The other columns contain the dublin core metadata fields that the data is to reside.

A typical heading row looks like:

id,collection,dc.title,dc.contributor,dc.date.issued,etc,etc,etc.

Subsequent rows in the csv file relate to items. A typical row might look like:

350,2292,Item title,"Smith, John",2008

If you want to store multiple values for a given metadata element, they can be separated with the double-pipe '||' (or another character that you defined in your _dspace.cfg _file. For example:

Horses||Dogs||Cats

Silent Mode should be used carefully. It is possible (and probable) that you can overlay the wrong data and cause irreparable damage to the database.

Importing Process

To run the batch importer, at the command line:

[dspace]/bin/dspace metadata-import -f name_of_file.csv 

Example

[dspace]/bin/dspace metadata-import -f /dImport/col_14.csv

If you are wishing to upload new metadata without bitstreams, at the command line:

[dspace]/bin/dspace/metadata-import -f /dImport/new_file.csv -e joe@user.com -w -n -t

In the above example we threw in all the arguments. This would add the metadata and engage the workflow, notification, and templates to all be applied to the items that are being added.

The CSV Files

The csv files that this tool can import and export abide by the RFC4180 CSV format http://www.ietf.org/rfc/rfc4180.txt. This means that new lines, and embedded commas can be included by wrapping elements in double quotes. Double quotes can be included by using two double quotes. The code does all this for you, and any good csv editor such as Excel or OpenOffice will comply with this convention.

File Structure. The first row of the csv must define the metadata values that the rest of the csv represents. The first column must always be "id" which refers to the item's id. All other columns are optional. The other columns contain the dublin core metadata fields that the data is to reside.

A typical heading row looks like:

id,collection,dc.title,dc.contributor,dc.date.issued,etc,etc,etc.

Subsequent rows in the csv file relate to items. A typical row might look like:

350,2292,Item title,"Smith, John",2008

If you want to store multiple values for a given metadata element, they can be separated with the double-pipe '||' (or another character that you defined in your _dspace.cfg _file. For example:

Horses||Dogs||Cats

Elements are Elements are stored in the database in the order that they appear in the csv file. You can use this to order elements where order may matter, such as authors, or controlled vocabulary such as Library of Congress Subject Headings.

When importing a csv file, the importer will overlay the data onto what is already in the repository to determine the differences. It only acts on the contents of the cvs csv file, rather than on the complete item metadata. This means that the CSV file that is exported can be manipulated quite substantially before being re-imported. Rows (items) or Columns (metadata elements) can be removed and will be ignored. For example, if you only want to edit item abstracts, you can remove all of the other columns and just leave the abstract column. (You do need to leave the ID column intact. This is mandatory).

...

Checksum Checker is program that can run to verify the checksum of every item within DSpace. Checksum Checker was designed with the idea that most System Administrators will run it from the cron. Depending on the size of the repository choose the options wisely.

There are three aspects of the Checksum Checker's operation that can be configured:

...

Available command line options

• Wiki Markup\*Limited-count mode: *_\[dspace\]/bin/dspace checker \ -c_ To check a specific number of bitstreams. The \_-c_ option if followed by an integer, the number of bitstreams to check. Example: _\[dspace/bin/dspace checker \ -c 10_ This is particularly useful for checking that the checker is executing properly. The Checksum Checker's default execution mode is to check a single bitstream, as if the option was \_-c 1_ Wiki Markup
• *Duration mode:*_\[dspace\]/bin/dspace checker \ -d_ To run the Check for a specific period of time with a time argument. You may use any of the time arguments below: Example: \_\[dspace/bin/dspace checker \ -d 2h_ (Checker will run for 2 hours) \|

  s

  \|

  Seconds

  \|

  m	Minutes
  h	Hours
  d	Days
  w	Weeks
  y	Years

  The checker will keep starting new bitstream checks for the specific durations, so actual execution duration will be slightly longer than the specified duration. Bear this in mind when scheduling checks.unmigrated-wiki-markup
• *Specific Bistream Bitstream mode:*_\[dspace\]/bin/dspace checker \ -b_ Checker will only look at the internal bitsteam bitstream IDs. Example: \_\[dspace\]/bin/dspace checker \ -b 112 113 4567_ Checker will only check bitstream IDs 112, 113 and 4567.unmigrated-wiki-markup
• *Specific Handle mode:*_\[dspace\]/bin/dspace checker \ -a_Checkr Checker will only check bistreams bitstreams within the Community, Community or the item itself. Example: \_\[dspace\]/bin/dspace checker \ -a 123456/999_ Checker will only check this handle. If it is a Collection or Community, it will run through the entire Collection or Community.The Checkunmigrated-wiki-markup
• *Looping mode:*_\[dspace\]/bin/dspace checker \ -l_ or \_\[dspace\]/bin/dspace checker \ -L_ There are two modes. The lowercase 'el' (-l) specifies to check every bitstream in the repository once. This is recommended for smaller repositories who are able to loop through all their content in just a few hours maximum. An uppercase 'L' (-L) specifies to continuously loops through the repository. This is not recommended for most repository systems. *Cron Jobs*. For large repositories that cannot be completely checked in a couple of hours, we recommend the \ -d option in cron.
• Wiki Markup*Pruning mode:*\_\[dspace\]/bin/dspace checker \ -p_ The Checksum Checker will store the result of every check in the checksum_histroy history table. By default, successful checksum matches that are eight weeks old or older will be deleted when the \ -p option is used. (Unsuccessful ones will be retained indefinitelindefinitely). Without this option, the retention settings are ignored and the database table may grow rather large\!

Checker Results Pruning

As stated above in "Pruning mode", the checksum_history table can get rather large, and that running the checker with the -p assists in the size of the checksum_history being kept manageable. The amount of time for which results are retained in the checksum_history table can be modified by one of two methods:

...

1. Editing the retention policies in _\[dspace\]/config/dspace.cfg_ See Chapter 5 Configuration for the property keys. OR
2. Pass in a properties file containting containing retention policies when using the -p option.To do this, create a file with the following two property keys:

   Code Block
   checker.retention.default = 10y checker.retention.CHECKSUM_MATCH = 8w

   You can use the table above for your time units. At the command line: Code Block[dspace]/bin/dspace checker -p retention_file_name <ENTER>

Checker Reporting

...

Checksum Checker uses log4j to report its results. By default it will report to a log called _\[dspace\]/log/checker.log_, and it will report only on bitstreams for which the newly calculated checksum does not match the stored checksum. To report on all bitstreams checked regardless of outcome, use the _\-v_ (verbose) command line option:

Wiki Markup_\[dspace\]/bin/dspace checker \ -l \ -v_ (This will loop through the repository once and report in detail about every bitstream checked.unmigrated-wiki-markup

To change the location of the log, or to modify the prefix used on each line of output, edit the _\[dspace\]/config/templates/log4j.properties_ file and run _\[dspace\]/bin/install_configs_.

Cron or Automatic Execution of Checksum Checker

You should schedule the Checksum Checker to run automatically, based on how frequently you backup your DSpace instance (and how long you keep those backups). The size of your repository is also a factor. For very large repositories, you may need to schedule it to run for an hour (e.g. -d 1h option) each evening to ensure it makes it through your entire repository within a week or so. Smaller repositories can likely get by with just running it weekly.

Unix, Linux, or MAC OS. You can schedule it by adding a cron entry similar to the following to the crontab for the user who installed DSpace:unmigrated-wiki-markup

...

0 4 

...

*

...

 * 0 

...

[dspace

...

]/bin/dspace checker 

...

-d2h 

...

-p

...

The above cron entry would schedule the checker to run the checker every Sunday at 400 (4:00 a.m.) for 2 hours. It also specifies to 'prune' the database based on the retention settings in dspace.cfg.

Windows OS. You will be unable to use the checker shell script. Instead, you should use Windows Schedule Tasks to schedule the following command to run at the appropriate times: Wiki Markup_''\

[dspace

...

]

...

/bin/

...

dspace checker -d2h 

...

-p

_ (This command should appear on a single line).

Automated Checksum Checkers' Results

Optionally, you may choose to receive automated emails listing the Checksum Checkers' results. Schedule it to run after the Checksum Checker has completed its processing (otherwise the email may not contain all the results).

...

it to run after the Checksum Checker has completed its processing (otherwise the email may not contain all the results).

You can also combine options (e.g. -m -c) for combined reports.

Cron. Follow the same steps above as you would running checker in cron. Change the time but match the regularity. Remember to schedule this *after* Checksum Checker has run..

Embargo

If you have implemented the Embargo feature, you will need to run it periodically to check for Items with expired embargoes and lift them.

...

run it periodically to check for Items with expired embargoes and lift them.

You must run the Embargo Lifter task periodically to check for items with expired embargoes and lift them from being embargoed. For example, to check the status, at the CLI: Wiki Markup_\

[dspace

...

]/bin/dspace embargo-lifter 

...

-c

...

To lift the actual embargoes on those items that meet the time criteria, at the CLI: Wiki Markup_\

[dspace

...

]/bin/dspace embargo-lifter 

...

-l

...

...

Browse Index Creation

To create all the various browse indexes that you define in the Configuration Section (Chapter 5) there are a variety of options available to you. You can see these options below in the command table.

...

5) there are a variety of options available to you. You can see these options below in the command table.

...

Running the Indexing Programs

Wiki Markup*Complete Index Regeneration*. By running _\[dspace\]/bin/dspace index-init_ you will completely regenerate your indexes, tearing down all old tables and reconstructing with the new cofiguration. Running this is the same as: Wiki Markup_\[dspace\new configuration.

[dspace]/bin/

...

dspace

...

...

index-init

Updating the Indexes. By running [dspace]/bin/dspace index-update you will reindex your full browse wihtout without modifying the table structure. (This should be your default approach if indexing, for example, via a cron job periodically). Running this is the same as: Wiki Markup_\

[dspace

...

]/bin/

...

dspace

...

...

index-

...

update

Destroy and rebuild. You can destroy and rebuild the database, but do not do the indexing. Output the SQL to do this to the screen and a file, as well as executing it against the database, while being verbose. At the CLI screen: Wiki Markup_\

[dspace

...

]/bin/

...

dspace index \-r \-t \-p \-v \-x \-o myfile.sql

...

...

Indexing Customization

DSpace provides robust browse indexing. It is possible to expand upon the default indexes delivered at the time of the installation. The System Administrator should review "Defining the Indexes" from the Chapter 5. Configuration to become familiar with the property keys and the definitions used therein before attempting heavy customizations.

...

• Add new browse indexes besides the four that are delivered upon installation. Examples:
  • Series
  • Specific subject fields (Library of Congress Subject Headings.(It is possible to create a browse index based on a controlled vocabulary or thesauristhesaurus.)
  • Other metadata schema fields
• Combine metadata fields into one browse
• Combine different metadata schemas in one browse
  Examples of new browse indexes that are possible.(The system administrator is reminded to read the section on Defining the Indexes in Chapter 5. Configuration.)

...

Remember to run index-init after adding any new defitions definitions in the dspace.cfg to have the indexes created and the data indexed.

...

With the release of DSpace 1.6, new statistics software component was added. DSpace's use of SOLR for statics makes it possible to have a database of statistics. This in mind, there is the issue of the older log files and how a site can use them. The following command process is able to convert the existing log files and then import them for SOLR use. The user will need to perform this only once.

The Log Converter program converts log files from dspace.log into an intermediate format that can be inserted into SOLR.

...

program converts log files from dspace.log into an intermediate format that can be inserted into SOLR.

The command loads the intermediate log files that have been created by the aforementioned script into SOLR.

...

The command loads the intermediate log files that have been created by the aforementioned script into SOLR.

Although the DSpace Log Convertor applies basic spider filtering (googlebot, yahoo slurp, msnbot), it is far from complete. Please refer to Statistics Client (8.15) for spider removal operations, after converting your old logs.

Client Statistics

...

converting your old logs.

Client Statistics

...

The usage of these options is open for the user to choose, If they want to keep spider entires in their repository, they can just mark them using "-m" and they will be excluded from statistics queries when "solr.statistics.query.filter.isBot = true" in the dspace.cfg.

If they want to keep the spiders out of the solr repository, they can run just use the "-i" option and they will be removed immediately.

Wiki MarkupThere are guards in place to control what can be defined as an IP range for a bot, in _\[dspace\]/config/spiders_, spider IP address ranges have to be at least 3 subnet sections in length 123.123.123 and IP Ranges can only be on the smallest subnet \ [123.123.123.0 - 123.123.123.255\]. If not, loading that row will cause exceptions in the dspace logs and exclude that IP entry.