Page History

...

The Replication Task Suite is a DSpace 1.8 Add-On which provides a set of curation system tasks to assist in performing replication (backup/restore/audit) of DSpace contents to other locations. The DSpace content is packaged in containers known as AIPs (OAIS speak: 'archival information packages'). By default, AIPs are generated in the default DSpace AIP Format (the same format used by the AIP Backup and Restore tool). If desired, there is an option to generate BagIt-based AIPs instead of using the default DSpace AIP format.

This Add-On integrates DSpace 1.8 with DuraCloud for users that wish to easily back up their content into DuraCloud directly from their DSpace administrative interface.

...

An "Early Access" release of the Replication Task Suite is available to install via:

...

New Development of the Replication Task Suite has been moved to GitHub: https://github.com/DSpace/dspace-replicate

Prerequisites

...

Installation

Supported DSpace Versions

The Replication Task Suite currently supports the following versions of DSpace software:

...

...

...

...

...

...

Installation instructions for each version are included below:

• Installation on DSpace 1.8.x
• Installation on DSpace 3.x

User Interface Compatibility Notes

As the Replication Suite is just a suite of Curation System tasks, it may be called

Developers may obtain an early version of the soon-to-be DSpace 1.8.1 release by accessing the 1.8 Bug-fix Branch in the DSpace SVN: http://scm.dspace.org/svn/repo/dspace/branches/dspace-1_8_x/

Because of enhancements to the Curation System in DSpace 1.8.0, the Replication Suite is only compatible with a DSpace 1.8.x System.

User Interface Compatibility Notes

As the Replication Suite is just a suite of Curation System tasks, it may be called (like any Curation Tasks) from the following locations:

• From the Command Line
• From the Admin UI (XMLUI Only)
• From Item Approval Workflow
• From custom Java code

For more information see the Curation System details on Task Invocation.

Installation

...

on DSpace 1.8.x

1. In your DSpace Source directory ([dspace-src]), you will modify two Maven pom.

Maven-based Installation

1. In your DSpace Source directory ([dspace-src]), you will modify two Maven pom.xml files:
   
   • [dspace-src]/dspace/pom.xml (This POM controls dependencies of CommandLine scripts. Modifying it will let you run dspace-replicate from commandline)

   • [dspace-src]/dspace/modules/xmlui/pom.xml (This POM controls dependencies of XMLUI. Modifying it will let you run dspace-replicate from XMLUI)

2. For both each of these pom.xml files, add the following <dependency> section at the end of the existing <dependencies> section (just before the closing </dependencies> tag):.

   Code Block
   <dependency><dependencies> <groupId>org.dspace</groupId> <artifactId>dspace-replicate</artifactId> <version>1.0-EA</version> </dependency> ... <!-- Adding this dependency will install the Replication Task Suite Addon --> <dependency> <groupId>org.dspace</groupId> <artifactId>dspace-replicate</artifactId> <version>1.0</version> </dependency> </dependencies>

3. Once you've finished modifying both pom.xml files, rebuild DSpace by running the following from your [dspace-src]/dspace/ folder:

   Code Block
   mvn clean package

4. You will need to update your existing DSpace 1.8.x installation, by running the following from your [dspace-src]/dspace/target/dspace-1.8.x-SNAPSHOT[version]-build/ directory

   Code Block
   ant update

   Note
   Alternatively, if you don't want to do a full DSpace update, you can just update your existing binaries & webapps by running the following two commands: ant update_code (Updates the existing [dspace]/lib/ directory) ant update_webapps (Updates the existing [dspace]/webapp/ directory)

   Copy

5. Follow the instructions in the Configuration section below in order to enable & configure the Replication

   Suite's configuration files to your DSpace configuration directory
   • Replication Suite Configuration File: Copy [dspace-replicate]/config/modules/replicate.cfg to your [dspace]/config/modules/ directory
   • METS-specific AIP Configuration Settings: Copy [dspace-replicate]/config/modules/replicate-mets.cfg to your [dspace]/config/modules/ directory
   • DuraCloud Configuration File: Copy [dspace-replicate]/config/modules/duracloud.cfg to your [dspace]/config/modules/ directory
6. Finally, follow the Configurationsettings instructions below to configure the Replication Suite based on your usage needs.
   • There is a sample curate.cfg file provided in [dspace-replicate]/config/modules/curate.cfg which can be used as a reference. It is pre-configured to use the DSpace AIP Format (METS-based packaging).

Manual Installation

1. Download the Replication Suite code
   • Downloadable Zip: https://github.com/DSpace/dspace-replicate/tags
   • Download "Early Access" release via SVN: http://scm.dspace.org/svn/repo/modules/dspace-replicate/tags/dspace-replicate-1.0-EA/
   • Experimental Trunk code is available via GitHub at: https://github.com/DSpace/dspace-replicate

2. Build/Compile the Replication Suite, by running the following from the root directory

   Code Block
   mvn package

3. Copy the generated JAR files to your DSpace 1.8 installation.
   1. There are a total of 5 JARs that will need to be copied to your [dspace]/lib/
      • [dspace-replicate]/target/dspace-replicate-[version].jar (The Replication Suite Plugin)
      • [dspace-replicate]/target/lib/common-[version].jar (DuraCloud common libraries - required for DuraCloud integration)
      • [dspace-replicate]/target/lib/commons-compress-[version].jar (Apache Commons Compress - prerequisite for Replication Suite plugin)
      • [dspace-replicate]/target/lib/storageprovider-[version].jar (DuraCloud storage provider libraries - required for DuraCloud integration)
      • [dspace-replicate]/target/lib/storeclient-[version].jar (DuraCloud store client libraries - required for DuraCloud integration)
   2. Also, copy the above 5 JARs also to your XMLUI web application's WEB-INF/lib directory (e.g. [dspace]/webapps/xmlui/WEB-INF/lib/)
4. Copy the Replication Suite's configuration files to your DSpace configuration directory
   • Replication Suite Configuration File: Copy [dspace-replicate]/config/modules/replicate.cfg to your [dspace]/config/modules/ directory
   • METS-specific AIP Configuration Settings: Copy [dspace-replicate]/config/modules/replicate-mets.cfg to your [dspace]/config/modules/ directory
   • DuraCloud Configuration File: Copy [dspace-replicate]/config/modules/duracloud.cfg to your [dspace]/config/modules/ directory
5. Finally, follow the Configurationsettings instructions below to configure the Replication Suite based on your usage needs.
   • There is a sample curate.cfg file provided in [dspace-replicate]/config/modules/curate.cfg which can be used as a reference. It is pre-configured to use the DSpace AIP Format (METS-based packaging).

Configuration

Configuration of the Replication Task Suite is based entirely on your local institution's backup, restore and preservation needs.

Before getting started, you may wish to determine the answers to the following questions:

1. #AIP Format Options: Does you institution want to backup using the default DSpace AIP format (METS packaging)? Or would you rather utilize the new BagIt AIP Format?
2. #Storage Options: Does you institution plan to use the Replication Suite to backup to a local/mounted drive? Or would you like to connect it to a DuraCloud account?
3. #Additional Options: Do you plan to use Checkm manifests for checksum auditing?

AIP Format Options

One of the first questions to ask yourself is the format you wish to utilize for your AIPs.

There are two options:

1. DSpace AIP Format (METS-based) (default) - This is the same AIP format utilized by the DSpace AIP Backup and Restore feature, so it is 100% compatible with that existing feature. In fact when using this format, the Replication Task Suite just "wraps" calls to the AIP Backup and Restore feature itself.
2. BagIt AIP Format - This is a new AIP format provided by the Replication Task Suite. It generates AIPs in the BagIt File Packaging Format. Institutions which already are familiar with BagIt or use it elsewhere may find this format preferrable.

For more information on the tasks available based on your AIP format choice, please see the #Problem Statement & Usage Examples section below. This section also provides good examples of how to use each of the tasks available to you in the Replication Task Suite.

Configuring usage of DSpace default AIP Format (METS-based)

This section goes through the steps of configuring the Replication Suite to use the default DSpace AIP format, which utilizes METS packaging.

1. Task Suite Add-On.

    

Installation on DSpace 3.x

1. In your DSpace Source directory ([dspace-src]), you will need to modify the following POM file:
   

   • [dspace-src]/dspace/modules/additions/pom.xml (This POM will ensure that the "dspace-replicate" dependency is made available to commandline and ALL DSpace interfaces)

2. For this pom.xml file, add the following <dependency> section at the end of the existing <dependencies> section (just before the closing </dependencies> tag).

   Code Block
   <dependencies> ... <!-- Adding this dependency will install the Replication Task Suite Addon --> <dependency> <groupId>org.dspace</groupId> <artifactId>dspace-replicate</artifactId> <version>3.0-SNAPSHOT</version> </dependency> </dependencies>

   Note
   title Replication Task Suite version 3.0-SNAPSHOT is nearly identical to 1.0 stable
   The 3.0-SNAPSHOT version of the Replication Task Suite is nearly identical to the 1.0 stable version.  The only changes are very minor bug fixes to allow for the Replication Task Suite to be compatible with the new DSpace 3.x API.  So, even though this is a "-SNAPSHOT" version, you should still find it to be stable.  A "3.0-EA1" (Early Access #1) version will be released in the near future after more extensive testing is performed.

3. Once you've finished modifying both pom.xml files, rebuild DSpace by running the following from your [dspace-src]/dspace/ folder:

   Code Block
   mvn clean package

4. You will need to update your existing DSpace 3.x installation, by running the following from your [dspace-src]/dspace/target/dspace-[version]-build/ directory

   Code Block
   ant update

   Note
   Alternatively, if you don't want to do a full DSpace update, you can just update your existing binaries & webapps by running the following two commands: ant update_code (Updates the existing [dspace]/lib/ directory) ant update_webapps (Updates the existing [dspace]/webapp/ directory)

5. Follow the instructions in the Configuration section below in order to enable & configure the Replication Task Suite Add-On.

Configuration

Configuration of the Replication Task Suite is based entirely on your local institution's backup, restore and preservation needs.

Before getting started, you may wish to determine the answers to the following questions:

1. AIP Format Options: Does you institution want to backup using the default DSpace AIP format (METS packaging)? Or would you rather utilize the new BagIt AIP Format?
2. Storage Options: Does you institution plan to use the Replication Suite to backup to a local/mounted drive? Or would you like to connect it to a DuraCloud account?
3. Additional Options: Do you plan to use Checkm manifests for checksum auditing?

AIP Format Options

One of the first questions to ask yourself is the format you wish to utilize for your AIPs.

There are two options:

1. DSpace AIP Format (METS-based) (default) - This is the same AIP format utilized by the DSpace AIP Backup and Restore feature, so it is 100% compatible with that existing feature. In fact when using this format, the Replication Task Suite just "wraps" calls to the AIP Backup and Restore feature itself.
2. BagIt AIP Format - This is a new AIP format provided by the Replication Task Suite. It generates AIPs in the BagIt File Packaging Format. Institutions which already are familiar with BagIt or use it elsewhere may find this format preferrable.

For more information on the tasks available based on your AIP format choice, please see the Problem Statement & Usage Examples section below. This section also provides good examples of how to use each of the tasks available to you in the Replication Task Suite.

Configuring usage of DSpace default AIP Format (METS-based)

This section goes through the steps of configuring the Replication Suite to use the default DSpace AIP format, which utilizes METS packaging.

1. General Curation Configuration: First, in your [dspace]/config/modules/curate.cfg you will want to enable & configure the METS-based replication tasks. (NOTE: there is a sample curate.cfg file provided in [dspace-replicate]/config/modules/curate.cfgwhich is pre-configured to use METS-based AIPs).

   • Enable the Replication Tasks: In the list of "Task Class implementations" (plugin.named.org.dspace.curate.CurationTask), add the following.
     REMEMBER to add a comma and backslash (", \") after each line (except the final line).

     Code Block
     plugin.named.org.dspace.curate.CurationTask = \ ... (YOUR

   General Curation Configuration: First, in your [dspace]/config/modules/curate.cfg you will want to enable & configure the METS-based replication tasks. (NOTE: there is a sample curate.cfg file provided in [dspace-replicate]/config/modules/curate.cfgwhich is pre-configured to use METS-based AIPs).

   • Enable the Replication Tasks: In the list of "Task Class implementations" (plugin.named.org.dspace.curate.CurationTask), add the following.
     REMEMBER to add a comma and backslash (", \") after each line (except the final line).

     Code Block

     plugin.named.org.dspace.curate.CurationTask = \ ... (YOUR EXISTING TASKS) ... , \ org.dspace.ctask.replicate.EstimateAIPSize = estaipsize, \ org.dspace.ctask.replicate.ReadOdometer = readodometer, \ org.dspace.ctask.replicate.TransmitAIP = transmitaip, \ org.dspace.ctask.replicate.VerifyAIP = verifyaip, \ org.dspace.ctask.replicate.FetchAIP = fetchaip, \ org.dspace.ctask.replicate.CompareWithAIP = auditaip, \ org.dspace.ctask.replicate.RemoveAIP = removeaip, \ org.dspace.ctask.replicate.METSRestoreFromAIP = restorefromaip, \ org.dspace.ctask.replicate.METSRestoreFromAIP = replacewithaip, \ org.dspace.ctask.replicate.METSRestoreFromAIP = restorekeepexisting, \ org.dspace.ctask.replicate.METSRestoreFromAIP = restoresinglefromaip, \ org.dspace.ctask.replicate.METSRestoreFromAIP = replacesinglewithaip

   • Give Each Task a Human-Friendly Task Name: Under the ui.tasknames setting, give each of the above Tasks a human-friendy name. Here are some recommended values, but you are welcome to tweak them.
     REMEMBER to add a comma and backslash (", \") after each line (except the final line).

     Code Block

     ui.tasknames = \ ... (YOUR EXISTING TASK NAMES) ... , \ estaipsize = Estimate Storage Space for AIP(s), \ readodometer = Read Odometer, \ transmitaip = Transmit AIP(s) to Storage, \ verifyaip = Verify AIP(s) exist in Storage, \ fetchaip = Fetch AIP(s) from Storage, \ auditaip = Audit against AIP(s), \ removeaip = Remove AIP(s) from Storage, \ restorefromaip = Restore Missing Object(s) from AIP(s), \ replacewithaip = Replace Existing Object(s) with AIP(s), \ restorekeepexisting = Restore Missing Object(s) but Keep Existing Objects,\ restoresinglefromaip = Restore Single Object from AIP, \ replacesinglewithaip = Replace Single Object with AIP

   • Optionally Create a Task Group: Finally, if you'd like to create a Task Group for these tasks, you can create a group named "replicate" and add them all to it. The below is just an example for how you may wish to set the ui.taskgroups and ui.taskgroup.*settings. It creates two Task Groups: (1) a "General Purpose Tasks" group for a few default DSpace Curation Tasks, and (2) a "Replication Suite Tasks" group for all these new Replication tasks.

     Code Block

     # Tasks may be organized into named groups which display together in UI drop-downs ui.taskgroups = \ general = General Purpose Tasks, replicate = Replication Suite Tasks # Group membership is defined using comma-separated lists of task names, one property per group ui.taskgroup.general = profileformats, requiredmetadata, checklinks ui.taskgroup.replicate = estaipsize, readodometer, transmitaip, verifyaip, fetchaip, auditaip, removeaip, restorefromaip, replacewithaip, restorekeepexisting, restoresinglefromaip, replacesinglewithaip

2. Replication Suite Configuration: Next, in your [dspace]/config/modules/replicate.cfgyou will want to ensure it is setup to properly use METS-based AIPs. Under the "AIP Packaging Settings" you'll want the following settings enabled:

   Code Block

   # Package type. Permitted values: 'mets', 'bagit' # mets = Generate default DSpace AIPs as described in: https://wiki.duraspace.org/display/DSDOC18/AIP+Backup+and+Restore # bagit = Generate AIPs based on the BagIt packaging format: https://wiki.ucop.edu/display/Curation/BagIt packer.pkgtype = mets # Format of package compression. Permitted values: 'zip' or 'tgz' # for 'mets' packages, only 'zip' is supported packer.archfmt = zip # Whether or not the name packages with a DSpace type prefix. # When 'true', package files are named [type]@[handle].[format] (e.g. ITEM@123456789-1.zip) # When 'false', package files are named [handle].[format] (e.g. 123456789-1.zip) # Defaults to 'true'. For 'mets' packages, this must be 'true'. packer.typeprefix = true

3. Optionally tweak the AIP Restore/Replace settings: Optionally, you can decide to tweak the way AIPs are restored or replaced (using AIP Backup and Restore). These settings normally should not need to be tweaked, but are available in the [dspace]/config/modules/replicate-mets.cfg configuration file. See that configuration file for more details.

...

1. General Curation Configuration: First, in your [dspace]/config/modules/curate.cfg you will want to enable & configure the BagIt-based replication tasks. (NOTE: there is a sample curate.cfg file provided in [dspace-replicate]/config/modules/curate.cfg which provides example settings).
   

   • Enable the Replication Tasks: In the list of "Task Class implementations" (plugin.named.org.dspace.curate.CurationTask), add the following.
     REMEMBER to add a comma and backslash (", \") after each line (except the final line).

     Code Block

     plugin.named.org.dspace.curate.CurationTask = \ ... (YOUR EXISTING TASKS) ... , \ org.dspace.ctask.replicate.EstimateAIPSize = estaipsize, \ org.dspace.ctask.replicate.ReadOdometer = readodometer, \ org.dspace.ctask.replicate.TransmitAIP = transmitaip, \ org.dspace.ctask.replicate.VerifyAIP = verifyaip, \ org.dspace.ctask.replicate.FetchAIP = fetchaip, \ org.dspace.ctask.replicate.CompareWithAIP = auditaip, \ org.dspace.ctask.replicate.RemoveAIP = removeaip, \ org.dspace.ctask.replicate.BagItRestoreFromAIP = restorefromaip, \ org.dspace.ctask.replicate.BagItReplaceWithAIP = replacewithaip

   • Give Each Task a Human-Friendly Task Name: Under the ui.tasknames setting, give each of the above Tasks a human-friendy name. Here are some recommended values, but you are welcome to tweak them.
     REMEMBER to add a comma and backslash (", \") after each line (except the final line).

     Code Block

     ui.tasknames = \ ... (YOUR EXISTING TASK NAMES) ... , \ estaipsize = Estimate Storage Space for AIP(s), \ readodometer = Read Odometer, \ transmitaip = Transmit AIP(s) to Storage, \ verifyaip = Verify AIP(s) exist in Storage, \ fetchaip = Fetch AIP(s) from Storage, \ auditaip = Audit/Compare against AIP(s), \ removeaip = Remove AIP(s) from Storage, \ restorefromaip = Restore Missing Object(s) from AIP(s), \ replacewithaip = Replace Existing Object(s) with AIP(s)

   • Optionally Create a Task Group: Finally, if you'd like to create a Task Group for these tasks, you can create a group named "replicate" and add them all to it. The below is just an example for how you may wish to set the ui.taskgroups and ui.taskgroup.*settings. It creates two Task Groups: (1) a "General Purpose Tasks" group for a few default DSpace Curation Tasks, and (2) a "Replication Suite Tasks" group for all these new Replication tasks.

     Code Block

     # Tasks may be organized into named groups which display together in UI drop-downs ui.taskgroups = \ general = General Purpose Tasks, replicate = Replication Suite Tasks # Group membership is defined using comma-separated lists of task names, one property per group ui.taskgroup.general = profileformats, requiredmetadata, checklinks ui.taskgroup.replicate = estaipsize, readodometer, transmitaip, verifyaip, fetchaip, auditaip, removeaip, restorefromaip, replacewithaip

2. Replication Suite Configuration: Next, in your [dspace]/config/modules/replicate.cfgyou will want to ensure it is setup to properly use BagIt-based AIPs. Under the "AIP Packaging Settings" you'll want the following settings enabled:

   Code Block
   # Package type. Permitted values: 'mets', 'bagit' # mets = Generate default DSpace AIPs as described in: https://wiki.duraspace.org/display/DSDOC18/AIP+Backup+and+Restore # bagit = Generate AIPs based on the BagIt packaging format: https://wiki.ucop.edu/display/Curation/BagIt packer.pkgtype = bagit

...

The default configuration (LocalObjectStore) simply writes the AIPs to the local directory configured by the 'store.dir' property in replicate.cfg. This is not intended to be a production-grade solution, since a failure in the DSpace asset store could likely also affect this storage. It is provided mostly as a way to begin to work with the replication tasks without worrying about finding a storage provider.

For replicating in earnest, a service like DuraCloud is recommended (DuraCloudObjectStore. Such a service has the additional benefits of providing offsite storage/replication while also providing additional preservation management tools. Note that this service must be established and provisioned prior to use. For more information on DuraCloud see: http://www.duracloud.org

Alternatively, the MountableObjectStore option may be used if you wish to keep your AIP storage more "local" (e.g. on a local SAN or storage network). This option acts similar to the default configuration (in that it writes to the local directory configured by the 'store.dir' property in replicate.cfg). But, the expectation is that directory is actually a mounted storage drive, so AIPs are written in such a way as to support more complex storage architectures (e.g. an NFS-mounted store).

More information about each of these storage options (and how to configure them) is available in the Storage Options configuration section.

Codebase / Development

1. Download the Replication Suite code from GitHub: https://github.com/DSpace/dspace-replicate
   

   1. Checkout the branch you wish to develop against.  For example, to checkout the 1.x branch of the codebase:

      Code Block
      git checkout dspace-replicate-1.x

2. Build/Compile the Replication Suite, by running the following from the root directory

   Code Block
   mvn package

3. The code will be compiled into a JAR and all its dependencies will also be copied to your "target" directory
   
   1. The main dspace-replicate.jar will be compiled to:
      
      • [dspace-replicate]/target/dspace-replicate-[version].jar (The Replication Suite Plugin)
   2. There will also be a total of 4 dependency JARs that will be copied to:
      
      • [dspace-replicate]/target/lib/common-[version].jar (DuraCloud common libraries - required for DuraCloud integration)
      • [dspace-replicate]/target/lib/commons-compress-[version].jar (Apache Commons Compress - prerequisite for Replication Suite plugin)
      • [dspace-replicate]/target/lib/storageprovider-[version].jar (DuraCloud storage provider libraries - required for DuraCloud integration)
      • [dspace-replicate]/target/lib/storeclient-[version].

...

1. 1. • jar (DuraCloud store client libraries - required for DuraCloud integration)
   2. Also, copy the above 5 JARs also to your XMLUI web application's WEB-INF/lib directory (e.g. [dspace]/webapps/xmlui/WEB-INF/lib/)
2. Once the codebase is compiled, you can install it by following the Installation instructions above.  
   
   1. Alternatively, you may temporarily copy all 5 JARs (dspace-replicate + dependency JARs) to the following locations for testing purposes only:
      
      • DSpace "lib" folder (e.g. [dspace]/lib/) - This will make the Replication Task Suite available via the commandline
      • DSpace XMLUI "lib" folder (e.g. [dspace]/webapps/xmlui/WEB-INF/lib/) - This will make the Replication Task Suite available via the XMLUI.
   2. You will also need to follow the Configuration instructions above in order to properly enable & configure the Replication Task Suite

For replicating in earnest, a service like DuraCloud is recommended (DuraCloudObjectStore. Such a service has the additional benefits of providing offsite storage/replication while also providing additional preservation management tools. Note that this service must be established and provisioned prior to use. For more information on DuraCloud see: http://www.duracloud.org

Alternatively, the MountableObjectStore option may be used if you wish to keep your AIP storage more "local" (e.g. on a local SAN or storage network). This option acts similar to the default configuration (in that it writes to the local directory configured by the 'store.dir' property in replicate.cfg). But, the expectation is that directory is actually a mounted storage drive, so AIPs are written in such a way as to support more complex storage architectures (e.g. an NFS-mounted store).

...

1. 1. .