<?xml version="1.0" encoding="utf-8"?>
<html>
PLEDGE AIP Prototype
This page describes a prototype AIP implementation planned as
part of the PLEDGE project.
Since the PLEDGE project only needs AIPs
to replicate them under the direction of a policy engine,
it was not necessary to create an AIP-based asset store.
...
- AIP is a package describing one archival object.
- Archival object may be*Item*, Collection, or Community. Bitstreams are included in an Item's AIP.
- Each AIP is logically self-contained, can be restored without rest of the archive.
- AIP profile trades off favoring completeness and accuracy rather than presenting the semantics of an object in a standard format. It conforms to the quirks of DSpace's internal object model rather than attempting to produce a universally understandable representation of the object.
- An AIP can serve as a DIP, especially when transferring custody of objects to another DSpace implementation, but it is not intended to be a general-purpose DIP (the DSpace METS SIP profile is better for that).
- The implementation is layered on top of DSpace 1.4 plus the EventSystemPrototype with minimal other changes to the source.
- Restoration of an archive from AIPs is not perfectly complete; it is intended to recover from catastropic loss of content and metadata, not restore the exact same archive as before. Some information (e.g. access controls) would be lost.
- This prototype does NOT attempt to redefine the asset store in terms of AIPs, as in the AssetStore proposals.
...
- Although this prototype does not implement AIP-centric storage, it can be leveraged toward that goal.
- This design does not consider any of the proposals for versioning DSpace objects.
- See the ObjectUri proposal that establishes a pattern for generating URIs for pieces of the DSpace object model. This AIP implementation relies on it to name Bitstream asset-store files for internal AIPs.
- The impact on performance was a consideration:
- Creating or rewriting an AIP manifest is too expensive to do synchronously as part of every data model change.
- The solution is to let AIP updates lag behind real-time changes by some amount. Possible implementations include:
- Use asynchronous consumer (see Event System Prototype) to update AIPs at intervals and/or in a separate JVM.
- Periodically run the
Code Block AIPManager
command-line application (e.g. from cron) to update all "stale" AIPs.
- How best to record descriptive metadata? crosswalk to a standard like MODS, best for archival and external consumption? Save DIM directly for maximum completeness and accuracy of restoration in another DSpace? Include both?
- Some archival objects have elements that will probably be left out of the AIP prototype, e.g. template Items for Collections.
- Internal AIPs are susceptible to a race condition: Whenever any "member" elements of an object (e.g. the Bitstreams of an Item, or the Items of a Collection) are permanently deleted from the archive, the AIP for the parent object will contain unresolvable references until it is updated. If the internal AIP – which, remember, depends on the asset store for its components – is ingested at this point, the ingest will fail until the AIP is corrected.
AIP Details: METS Usage
Code Block mets
elementCode Block @PROFILE
fixed value="http://www.dspace.org/schema/aip/1.0/mets.xsd" (this is how we identify an AIP manifest)code@OBJID
URN-format persistent identifier (Handle) if available, or else a unique identifier.Code Block @LABEL
title if availablecode@TYPE
DSpace object type, one of "DSpace ITEM", "DSpace COLLECTION", "DSpace COMMUNITY".Code Block @ID
is a globally unique identifier, such as
{{dspace67075091976862014717971209717749394363{{.Code Block dspace67075091976862014717971209717749394363
- <font color="RED">@IDs should be used wherever available, I'll put a note about forming IDs in the profile spec.</font>
OK, but how unique does the ID have to be, just within the document or amongst all other AIP documentes? --lcs
<font color="RED">I can't imagine a scenario where we would reference an ID within a METS document from without that document, except for perhaps theCode Block mets@ID
{{. I would say the
Code Block {{mets@ID
should be unique amongst all AIP documents, but the other IDs should just be unique within the document.</font>
- <font color="RED">@IDs should be used wherever available, I'll put a note about forming IDs in the profile spec.</font>
Code Block mets/metsHdr
elementCode Block @CREATEDATE
timestamp that AIP was created.code@LASTMODDATE
last-modified date on Item, or nothing for other objects.- <font color="RED">mets defines these attributes as describing the METS document itself, we use them to describe the AIP, which sometimes we think of as the METS document, but more often think of as the 'package' – i.e. the METS document and all the files. I don't have a problem with the use Larry put forth, but we need to mention it in a prolife. I wonder if these dates shouldn't rather be in a techMD section, or maybe both.</font>
Code Block agent
element:- code
@ROLE
= "CUSTODIAN",Code Block @TYPE
= "OTHER",code@OTHERTYPE
= "DSpace Archive",codename
= Site handle.
Code Block mets/dmdSec
element- object's descriptive metadata crosswalked to MODS (or whatever the METS default is)
- <font color="RED">See link to RW's Comments Page below for notes on use of MODS</font>
- object's descriptive metadata in DSpace native DIM intermediate format, to serve as a complete and precise record for restoration or ingestion into another DSpace.
- <font color="RED">We should require mets/dmdSec@OTHERMDTYPE if @MDTYPE = "OTHER"</font>
- When the
Code Block mdWrap
Code Block @TYPE
value isCode Block OTHER
{{, the element MUST include a value for the
Code Block {{@OTHERTYPE
attribute which names the crosswalk that produced (or interprets) that metadata, e.g. {{AIP-TECHMD{{.Code Block
Code Block - object's descriptive metadata crosswalked to MODS (or whatever the METS default is)
mets/amdSec
element - admin (technical, source, rights, and provenance) metadata for the entire archival object.Code Block rightsMD
elements of the following TYPEs:Code Block DSpaceDepositLicense
if the object has a deposit license, it is contained here.codeCreativeCommonsRDF
If the object is an Item with a Creative Commons license expressed in RDF, it is included here.codeCreativeCommonsText
If the object is an Item with a Creative Commons license in plain text, it is included here.
sourceMD
elements - recorded twice, once in DSpace native format, once in PREMIS:
<font COLOR="RED">NOTE: PREMIS is only implemented for Bitstreams at the moment, and for the forseeable future.</font>- DSpace native format: MDTYPE="OTHER" OTHERMDTYPE="
{{AIP-TECHMD{{" (see Crosswalks section below for details'')Code Block - PREMIS expression of this technical metadata for archival object. (To be done later.)
- <font color="RED">RW:Comment AIP Object (Item-Collection-Community)-specific Metadata in PREMIS To see an example of the PREMIS version of this metadata, SEE link to RW Comments section page below</font>
Code Block - DSpace native format: MDTYPE="OTHER" OTHERMDTYPE="
digiprovMD
- When History data is available, includes a section of
Code Block TYPE="DSpaceHistory"
containing an RDF/XML rendition of the history data for the object. For internal AIPs, the history is stored in an external bitstream in the asset store; for self-contained packages it is a file in the package.
- When History data is available, includes a section of
Code Block mets/amdSec
elements - technical metadata for each of an Items's Bitstreams, both in PREMIS and DIM formats- code
techMD
element - PREMIS technical metadata, <font color="red">expanded from</font> SIP, for each of an Item's Bitstreams.Code Block sourceMD
element, type is AIP-TECHMD.- Bitstream-specific metadata not <font color="RED">all of which is</font> explicitly encoded in PREMIS, i.e.code
name
( {{dc.title{{)Code Block Code Block description
( {{dc.descripton{{)Code Block Code Block userFormatDescription
( {{dc.format{{)Code Block - BitstreamFormat, including short name, MIME type, extension. (
{{dc.format.medium{{)Code Block - <font color="RED">RW:Comment – Bitstream Technical Metadata
- Why are we recording the file format support status? That's a DSpace property, rather than an Item property. Do DSpace instances rely on objects to tell them their support status?</font>
- Format support and other properties of the BitstreamFormat are recorded here in case the Item is restored in an empty DSpace that doesn't have that format yet, and the relevant bits of the format entry have to be reconstructed from the AIP. --lcs
- <font color="RED">To see an example of the changes to the PREMIS version of this metadata, SEE link to RW Comments section page below</font>
- <font color="RED">RW:Comment – Bitstream Technical Metadata
Code Block mets/fileSec
element- For archival objects of type ITEM:
- Each distinct Bundle in an Item goes into a
{{fileGrp{{.Code Block fileGrp
- <font color="RED">Did the "ORIGINAL" bundle get renamed "CONTENT"?</font>
Not in DSpace 1.4_ atUSE is set to the exact Bundle name in an AIP. --lcs
- <font color="RED">Did the "ORIGINAL" bundle get renamed "CONTENT"?</font>
- Bitstreams in bundles become
Code Block file
elements under
.code{{fileGrp{{.Code Block fileGrp
file/@SEQ
contains the Bitstream sequence ID- <font color="RED">
Code Block file@CREATED
and
{{file@SIZE{{</font>Code Block file@SIZE
- <font color="RED">The DSpace SIP calls for the use of
Code Block @CREATED
for the file element, AIP examples do not use
{{@CREATED{{, but do useCode Block @CREATED
{{@SIZE{{, which is not recommended by SIP.</font>Code Block @SIZE
Since Bitstreams don't have any dates (neither created nor last-modified) the atCREATED cannot be set on dissemination. --lcs
- <font color="RED">The DSpace SIP calls for the use of
- Each distinct Bundle in an Item goes into a
Code Block mets/fileSec/fileGrp/file
element- Set @SIZE to length of the bitstream. There is a redundant value in the techMD but it is more accessible here.
- Set @MIMETYPE, @CHECKSUM, @CHECKSUMTYPE to corresponding bitstream values. There is redundant info in the techMD.
- SET @SEQ to bitstream's SequenceID if it has one.
- For archival objects of types COLLECTION and COMMUNITY:
- Only if the object has a logo bitstream, there is a
Code Block fileSec
with oneCode Block fileGrp
child of {{@TYPE="LOGO"{{.Code Block - The
Code Block fileGrp
contains oneCode Block file
element, representing the logo Bitstream. It has the same file format, checksum, etc fields as the Item content bitstreams, but does not include metadata section references or a SequenceID. - See the main
Code Block structMap
for the reference to this file.
- Only if the object has a logo bitstream, there is a
- For archival objects of type ITEM:
mets/structMap
- Primary structure map,Code Block @LABEL="DSpace
Object",
@TYPE="LOGICAL"
- For COLLECTION objects: Top-level
Code Block div
has one child:- code
div
withCode Block @TYPE="MEMBERS"
{{. For every Item in the Collection, it contains a
Code Block {{div
with anCode Block mptr
linking to the Handle of that Item. ItsCode Block @LOCTYPE="HANDLE"
{{, and
Code Block {{@xlink:href
value is the raw Handle.
- If Collection has a Logo bitstream, there is an
Code Block fptr
reference to it in the very first
{{div{{.Code Block div
- For COMMUNITY objects: Top-level
Code Block div
has two children:Code Block div
withCode Block @TYPE="SUBCOMMUNITIES"
{{. For every Sub-Community in the Community it contains a
Code Block {{div
with anCode Block mptr
linking to the Handle of that Community. ItsCode Block @LOCTYPE="HANDLE"
{{, and
Code Block {{@xlink:href
value is the raw Handle.Code Block div
withCode Block @TYPE="COLLECTIONS"
{{. For every child Collection, it contains a
Code Block {{div
with anCode Block mptr
linking to the Handle of that Collection. ItsCode Block @LOCTYPE="HANDLE"
{{, and
Code Block {{@xlink:href
value is the raw Handle.
- If Community has a Logo bitstream, there is an
Code Block fptr
reference to it in the very first
{{div{{.Code Block div
- ITEM objects have the same kind of simple structure map as SIP/DIP: top level
Code Block div
with aCode Block div
under it for each visible Bitstream.- If Item has primary bitstream, put it in first
{{structMap/div/fptr{{.Code Block
- If Item has primary bitstream, put it in first
Code Block - For COLLECTION objects: Top-level
mets/structMap
- Structure Map to indicate object's Parent- Contains one
Code Block div
element which has the unique attribute valueCode Block TYPE="AIP
Parent
Link"
to identify it as the older of the parent pointer.- It contains a
Code Block mptr
element whoseCode Block xlink:href
attribute value is the raw Handle of the parent object, e.g. {{1721.1/4321{{.<p>In order to restore a DSpace archive from internal AIPs in the asset store, the parent of each object must be available at the surface level of the METS document so the object can be instantiated under its correct parent before the metadata (which may also name the parent) is crosswalked.Code Block
- It contains a
- Contains one
...
The following steps have been tested for a very small archive
and successfully restored the RDBMS tables from internal AIPs in the
asset store. Note that this is a coarse overview and does not
consider error-handling.
Restoration
- Run
Code Block /dspace/bin/cleanup
to clear out unused bitstreams from the asset store. - Shut down your servlet container, if necessary.
- Remove the search indices:
Code Block rm
/dspace/search/*
- If your archive is configured to use History, save the old History by renaming its directory, and create a new, empty History directory
e.g.Code Block mv
history
history.old
;
mkdir
history
- Start with an empty database. Either:
- Backup the current state of the RDBMS, and destroy it with
e.g.Code Block drop
database
dspace;
- Simply change your DSpace configuration to point to a different database instance, if you have room for another database.
- Backup the current state of the RDBMS, and destroy it with
- Create a new, empty database:code
createdb
-U
dspace
-E
UNICODE
dspace
- Run the scripts in your install directory to initialize the DB:
Code Block ant
setup_database
load_registries
- Back in the DSpace run directory, create an admin user:
Code Block /dspace/bin/create-administrator
- Initialize the search and browse indices:
Code Block /dspace/bin/index-all
- In your DSpace configuration, ensure that the AIP restoration application will run with History turned off:
- Set up a separate dispatcher for the AIPManager application:
Code Block aipManager.dispatcher
=
restore
- Ensure that the
Code Block restore
Dispatcher does NOT call the History consumer, although it should call the search and browse consumers synchronously:Code Block event.dispatcher.restore.class
=
org.dspace.event.BasicDispatcher<br>event.dispatcher.restore.consumers
=
search:sync,
browse:sync
- Set up a separate dispatcher for the AIPManager application:
- Rebuild the Bitstream table:
Code Block /dspace/bin/dsrun
org.dspace.administer.RebuildBitstreamTable
-r
- Rebuild the InternalAIP table:
Code Block /dspace/bin/dsrun
org.dspace.administer.AIPManager
-c
-a
-f
-v
-e
''admin-user''
- Restore archive from the internal AIPs:
Code Block /dspace/bin/dsrun
org.dspace.administer.AIPManager
-r
-a
-v
-e
''admin-user''
At each stage, carefully monitor the output and the DSpace log for indications of errors. You can retry the restore of an internal AIP, or even the whole set of them, if necessary; it automatically skips any objects that already exist.
...
To create an internal AIP, just add the package parameter
...
internal=true
...
to the command.
The resulting "package" will be a METS manifest document, e.g.
...
To ingest an AIP and create a new object under a parent of your choice, add the
...
ignoreParent
...
and
...
ignoreHandle
...
package parameters to the command:
...
Then apply the changes to your DSpace installation directory:
NOTE: The interface of
...
org.dspace.content.packager.PackageIngester
...
has been changed slightly. This will break any existing package ingesters, although the ones in the DSpace core have been fixed. Look at the changes to e.g.
...
org.dspace.content.packager.PDFPackager
...
for an example of how to update your code. The changes are quite minimal.
- Unpack the new source Zip file in your install directory with
{{unzip{{.Code Block unzip
- For each of the "diff" files, in order, go to your install directory and apply the diff with the command:
Code Block patch
-p
0
-l
<
''diff-file''
- Build and install the code:
Code Block ant
install_code
build_wars
- Ensure the configuration changes in
Code Block config/dspace.cfg
get propagated to your run-time config file. - Ensure the new files in
Code Block config/crosswalks
are installed in your run-time directory. - Apply the database change by running the SQL code in the file:
Code Block etc/database_schema_14-15.sql
- Be sure to install the new WAR file(s) in your servlet container.
- Test by updating internal AIPs as shown above
...
The following configuration keys apply to the AIP packager and management infrastructure.
They may also require certain crosswalk plugins to be configured,
but that is a separate issue that is addressed in the sample DSpace
configuration supplied with the system source.
Code Block aipManager.dispatcher
name of the Event Dispatcher for the AIPManager application; when restoring an archive from AIPs, it is best to set this to a dispatcher that calls the search and browse consumers, but not History.Code Block aip.packager
plugin name of the Packager used to ingest and disseminate AIPs; by default it is
{{AIP{{.Code Block AIP
Code Block mets.dspaceAIP.ingest.crosswalk.''mdSecType''
crosswalk plugin (either XML or Stream-oriented) to be called to interpret the given mdSec type. To ignore a section, set it toCode Block NULLSTREAM
(for stream data) orCode Block NIL
for XML.Code Block aip.disseminate.''mdSecName''
Sets the type name and crosswalks associated with each metadata section under the METS {{amdSec: sourceMD, techMD, rightsMD, digiprovMD{{. Value is comma-separated list of mdSecType:pluginName specifiers. For example:Code Block Code Block aip.disseminate.techMD
=
PREMIS
Code Block aip.disseminate.dmd
Sets the crosswalks and type names of descriptive metadata sections to include; value format is the same as the admin MD sections.Code Block aip.ingest.createEperson
When value is "true", AIP ingester will create an EPerson if needed so it can set the Submitter of a newly-created Item to the "correct" value. An EPerson created this way cannot login. Default is false.
...
- EventSystemPrototype prerequisite to applying the AIP patches.
- HistorySystemPrototype can be applied to add history data to AIPs
...