Page History

...

The Replication Task Suite is a DSpace Add-On which provides a set of curation system tasks to assist in performing replication (backup/restore/audit) of DSpace contents to other locations. The DSpace content is packaged in containers known as AIPs (OAIS speak: 'archival information packages'). By default, AIPs are generated in the default DSpace AIP Format (the same format used by the AIP Backup and Restore tool). If desired, there is an option to generate BagIt-based AIPs instead of using the default DSpace AIP format.

...

As the Replication Suite is just a suite of Curation System tasks, it may be called (like any Curation Tasks) from the following locations:

...

For more information see the Curation System details on Task Invocation.

Installation on DSpace 1.8.x

...

1. General Curation Configuration: First, in your [dspace]/config/modules/curate.cfg you will want to enable & configure the METS-based replication tasks. (NOTE: there is a sample curate.cfg file provided in https://github.com/DSpace/dspace-replicate/tree/master/config/modules which is pre-configured to use METS-based AIPs).
   

   • Enable the Replication Tasks: In the list of "Task Class implementations" (plugin.named.org.dspace.curate.CurationTask), add the following.
     REMEMBER to add a comma and backslash (", \") after each line (except the final line).

     Code Block

     plugin.named.org.dspace.curate.CurationTask = \ ... (YOUR EXISTING TASKS) ... , \ org.dspace.ctask.replicate.EstimateAIPSize = estaipsize, \ org.dspace.ctask.replicate.ReadOdometer = readodometer, \ org.dspace.ctask.replicate.TransmitAIP = transmitaip, \ org.dspace.ctask.replicate.TransmitSingleAIP = transmitsingleaip, \  org.dspace.ctask.replicate.VerifyAIP = verifyaip, \ org.dspace.ctask.replicate.FetchAIP = fetchaip, \ org.dspace.ctask.replicate.CompareWithAIP = auditaip, \ org.dspace.ctask.replicate.RemoveAIP = removeaip, \ org.dspace.ctask.replicate.METSRestoreFromAIP = restorefromaip, \ org.dspace.ctask.replicate.METSRestoreFromAIP = replacewithaip, \ org.dspace.ctask.replicate.METSRestoreFromAIP = restorekeepexisting, \ org.dspace.ctask.replicate.METSRestoreFromAIP = restoresinglefromaip, \ org.dspace.ctask.replicate.METSRestoreFromAIP = replacesinglewithaip

   • Give Each Task a Human-Friendly Task Name: Under the ui.tasknames setting, give each of the above Tasks a human-friendy name. Here are some recommended values, but you are welcome to tweak them.
     REMEMBER to add a comma and backslash (", \") after each line (except the final line).

     Code Block

     ui.tasknames = \ ... (YOUR EXISTING TASK NAMES) ... , \ estaipsize = Estimate Storage Space for AIP(s), \ readodometer = Read Odometer, \ transmitaip = Transmit AIP(s) to Storage, \ verifyaip = Verify AIP(s) exist in Storage, \ fetchaip = Fetch AIP(s) from Storage, \ auditaip = Audit against AIP(s), \ removeaip = Remove AIP(s) from Storage, \ restorefromaip = Restore Missing Object(s) from AIP(s), \ replacewithaip = Replace Existing Object(s) with AIP(s), \ restorekeepexisting = Restore Missing Object(s) but Keep Existing Objects,\ restoresinglefromaip = Restore Single Object from AIP, \ replacesinglewithaip = Replace Single Object with AIP

   • Optionally Create a Task Group: Finally, if you'd like to create a Task Group for these tasks, you can create a group named "replicate" and add them all to it. The below is just an example for how you may wish to set the ui.taskgroups and ui.taskgroup.* settings. It creates two Task Groups: (1) a "General Purpose Tasks" group for a few default DSpace Curation Tasks, and (2) a "Replication Suite Tasks" group for all these new Replication tasks.

     Code Block

     # Tasks may be organized into named groups which display together in UI drop-downs ui.taskgroups = \ general = General Purpose Tasks, replicate = Replication Suite Tasks # Group membership is defined using comma-separated lists of task names, one property per group ui.taskgroup.general = profileformats, requiredmetadata, checklinks ui.taskgroup.replicate = estaipsize, readodometer, transmitaip, verifyaip, fetchaip, auditaip, removeaip, restorefromaip, replacewithaip, restorekeepexisting, restoresinglefromaip, replacesinglewithaip

2. Replication Suite Configuration: Next, in your [dspace]/config/modules/replicate.cfg you will want to ensure it is setup to properly use METS-based AIPs. Under the "AIP Packaging Settings" you'll want the following settings enabled:

   Code Block

   # Package type. Permitted values: 'mets', 'bagit' # mets = Generate default DSpace AIPs as described in: https://wiki.duraspace.org/display/DSDOC18/AIP+Backup+and+Restore # bagit = Generate AIPs based on the BagIt packaging format: https://wiki.ucop.edu/display/Curation/BagIt packer.pkgtype = mets # Format of package compression. Permitted values: 'zip' or 'tgz' # for 'mets' packages, only 'zip' is supported packer.archfmt = zip # Whether or not the name packages with a DSpace type prefix. # When 'true', package files are named [type]@[handle].[format] (e.g. ITEM@123456789-1.zip) # When 'false', package files are named [handle].[format] (e.g. 123456789-1.zip) # Defaults to 'true'. For 'mets' packages, this must be 'true'. packer.typeprefix = true

3. Optionally tweak the AIP Restore/Replace settings: Optionally, you can decide to tweak the way AIPs are restored or replaced (using AIP Backup and Restore). These settings normally should not need to be tweaked, but are available in the [dspace]/config/modules/replicate-mets.cfg configuration file. See that configuration file for more details.

...

1. General Curation Configuration: First, in your [dspace]/config/modules/curate.cfg you will want to enable & configure the BagIt-based replication tasks. (NOTE: there is a sample curate.cfg file provided in  in https://github.com/DSpace/dspace-replicate/tree/master/config/modules which provides example settings, though they are all commented out by default).
   

   • Enable the Replication Tasks: In the list of "Task Class implementations" (plugin.named.org.dspace.curate.CurationTask), add the following.
     REMEMBER to add a comma and backslash (", \") after each line (except the final line).

     Code Block

     plugin.named.org.dspace.curate.CurationTask = \ ... (YOUR EXISTING TASKS) ... , \ org.dspace.ctask.replicate.EstimateAIPSize = estaipsize, \ org.dspace.ctask.replicate.ReadOdometer = readodometer, \ org.dspace.ctask.replicate.TransmitAIP = transmitaip, \ org.dspace.ctask.replicate.VerifyAIP = verifyaip, \ org.dspace.ctask.replicate.FetchAIP = fetchaip, \ org.dspace.ctask.replicate.CompareWithAIP = auditaip, \ org.dspace.ctask.replicate.RemoveAIP = removeaip, \ org.dspace.ctask.replicate.BagItRestoreFromAIP = restorefromaip, \ org.dspace.ctask.replicate.BagItReplaceWithAIP = replacewithaip

   • Give Each Task a Human-Friendly Task Name: Under the ui.tasknames setting, give each of the above Tasks a human-friendy name. Here are some recommended values, but you are welcome to tweak them.
     REMEMBER to add a comma and backslash (", \") after each line (except the final line).

     Code Block

     ui.tasknames = \ ... (YOUR EXISTING TASK NAMES) ... , \ estaipsize = Estimate Storage Space for AIP(s), \ readodometer = Read Odometer, \ transmitaip = Transmit AIP(s) to Storage, \ verifyaip = Verify AIP(s) exist in Storage, \ fetchaip = Fetch AIP(s) from Storage, \ auditaip = Audit/Compare against AIP(s), \ removeaip = Remove AIP(s) from Storage, \ restorefromaip = Restore Missing Object(s) from AIP(s), \ replacewithaip = Replace Existing Object(s) with AIP(s)

   • Optionally Create a Task Group: Finally, if you'd like to create a Task Group for these tasks, you can create a group named "replicate" and add them all to it. The below is just an example for how you may wish to set the ui.taskgroups and ui.taskgroup.*settings. It creates two Task Groups: (1) a "General Purpose Tasks" group for a few default DSpace Curation Tasks, and (2) a "Replication Suite Tasks" group for all these new Replication tasks.

     Code Block

     # Tasks may be organized into named groups which display together in UI drop-downs ui.taskgroups = \ general = General Purpose Tasks, replicate = Replication Suite Tasks # Group membership is defined using comma-separated lists of task names, one property per group ui.taskgroup.general = profileformats, requiredmetadata, checklinks ui.taskgroup.replicate = estaipsize, readodometer, transmitaip, verifyaip, fetchaip, auditaip, removeaip, restorefromaip, replacewithaip

2. Replication Suite Configuration: Next, in your [dspace]/config/modules/replicate.cfg you will want to ensure it is setup to properly use BagIt-based AIPs. Under the "AIP Packaging Settings" you'll want the following settings enabled:

   Code Block
   # Package type. Permitted values: 'mets', 'bagit' # mets = Generate default DSpace AIPs as described in: https://wiki.duraspace.org/display/DSDOC18/AIP+Backup+and+Restore # bagit = Generate AIPs based on the BagIt packaging format: https://wiki.ucop.edu/display/Curation/BagIt packer.pkgtype = bagit

...

1. General Curation Configuration: First, in your [dspace]/config/modules/curate.cfg you will want to enable & configure the Checkm Manifest tasks. (NOTE: there is a sample curate.cfg file provided in [https://github.com/DSpace/dspace-replicate]/tree/master/config/modules/curate.cfg which provides example settings, though they are all commented out by default).
   

   • Enable the Checkm Tasks: In the list of "Task Class implementations" (plugin.named.org.dspace.curate.CurationTask), add the following.
     REMEMBER to add a comma and backslash (", \") after each line (except the final line).

     Code Block

     plugin.named.org.dspace.curate.CurationTask = \ ... (YOUR EXISTING TASKS) ... , \ org.dspace.ctask.replicate.checkm.TransmitManifest = transmitmanifest, \ org.dspace.ctask.replicate.checkm.VerifyManifest = verifymanifest, \ org.dspace.ctask.replicate.checkm.FetchManifest = fetchmanifest, \ org.dspace.ctask.replicate.checkm.CompareWithManifest = auditmanifest, \ org.dspace.ctask.replicate.checkm.RemoveManifest = removemanifest

   • Give Each Task a Human-Friendly Task Name: Under the ui.tasknames setting, give each of the above Tasks a human-friendy name. Here are some recommended values, but you are welcome to tweak them.
     REMEMBER to add a comma and backslash (", \") after each line (except the final line).

     Code Block
     ui.tasknames = \ ... (YOUR EXISTING TASK NAMES) ... , \ transmitmanifest = Transmit Checkm Manifest to Storage, \ verifymanifest = Verify Checkm Manifest exists in Storage, \ fetchmanifest = Fetch Checkm Manifest from Storage, \ auditmanifest = Audit against Checkm Manifest, \ removemanifest = Remove Checkm Manifest from Storage

   • Optionally Create a Task Group: Finally, if you'd like to create a Task Group for these tasks, you can create a group named "checkm" and add them all to it. The below is just an example for how you may wish to set the ui.taskgroups and ui.taskgroup.* settings. It creates two Task Groups: (1) a "General Purpose Tasks" group for a few default DSpace Curation Tasks, and (2) a "Checkm Validation Tasks" group for all these new Replication tasks.

     Code Block

     # Tasks may be organized into named groups which display together in UI drop-downs ui.taskgroups = \ general = General Purpose Tasks, checkm = Checkm Validation Tasks # Group membership is defined using comma-separated lists of task names, one property per group ui.taskgroup.general = profileformats, requiredmetadata, checklinks ui.taskgroup.checkm = transmitmanifest, verifymanifest, fetchmanifest, auditmanifest, removemanifest

...

The default values will create a METS-based AIP in the default DSpace AIP Format, compressed into a 'zip' archive. The other alternative supported by the replication task suite is Library of Congress 'Bagit' packaging, which may compressed either into a 'zip' file or a 'tgz' ('gzipped tar'), a compression standard more common in Unix systems.

...

• Restore Single Object from AIP (restoresinglefromaip)
  
  • This task acts the same as the default "restorefromaip" task, but it does NOT restore any child objects. So, if it is run on a collection, just the collection itself will be restored (items in that collection will not be restored).
• Restore Missing Object(s) but Keep Existing Objects (restorekeepexisting)
  
  • This task acts similar to the default "restorefromaip" task, but it attempts to skip over any objects which already exist in the repository. In other words, an error is not thrown if an object already exists – rather that entire object (and all its child objects) are skipped over during processing and left unchanged. This mode is identical to the "Keep Existing" mode of the DSpace AIP Backup and Restore tool.

Replacing Object(s)

...