...

The Fedora 3→ 6 Migration Path

Overview

This migration path relies on the migration-utils tool.

The To walk this migration path you will need to download and run the  migration-utils tool directly against the can migrate either exported FOXML (in archive or migration format), or directly migrate the Fedora 3 data stored on-disk.  We recommend the direct Fedora 3 filesystem migration.

The representation of your Fedora 3 repository.  In other words it is not necessary to export your Fedora 3 repository.  The fcrepo- migration-utils tool will not make any changes to your Fedora 3 repository  – it only reads data from Fedora 3. 

repository so it is not necessary to make a backup before starting.  You will need to make sure that you have sufficient storage space available in the target Fedora 6 (OCFL) directory, as the migration will effectively create a copy of your Fedora 3 repository in the Fedora 6 format which conforms to the Oxford Common File Layout (OCFL) specification. 

Once your Fedora 3 repository has been migrated to Fedora 6, you must may start up an instance of Fedora 6 on top of the newly created OCFL directory tree.  As Fedora 6 starts up it will automatically rebuild internal indices by scanning the OCFL tree.  This index initialization can take a few seconds or several hours depending on the size of your repository.

Note that the Fedora 3 migration utility will not migrate the following Fedora 3 features:

• Disseminators (disseminator datastreams will be migrated, but they will no longer function in Fedora 6)
• Redirect (R) and external (E) datastream content (URLs will be copied)
• Solr index
• Resource index
• XACML policies (objects with POLICY datastreams attached will be migrated, but they will no longer control access in Fedora 6)

Prerequisites

• A host machine with filesystem access to both the Fedora 3 disk storage (or exported FOXML files) and the target OCFL disk storage.
  The target allocated disk space should be roughly four times the size of your Fedora 3 repository occupied disk space.
• Java 11 installed on the host machine.
• Knowledge of the Fedora 3 filesystem layout (legacy or Akubra), if migrating directly from Fedora 3 data on disk.

Running

1. Download the latest version of the  migration-utils utility: https://github.com/fcrepo-exts/migration-utils/releases

2.  Follow Follow the instructions in the migration-utils README: https://github.com/fcrepo-exts/migration-utils

3. Start up Fedora on top of your newly created Fedora 6 OCFL-compliant repository using the -Dfcrepo.home configuration property..

The migration-utils tool is a java command-line tool.  Make sure you run the utility with Java 11.

Hints and warnings

• We recommend that you shut down Fedora 3 repository or put it into read-only mode, if you plan on migrating directly from Fedora 3 data on disk, to avoid issues of changes occurring while data is being written.
• When performing a sizable migration, run the utility in a screen or tmux session that you can detach from.  Try not to shut down or reboot the host while the tool is running.
• If the migration is interrupted, you can pick up where the migration left off by relaunching the tool with the --resume flag (keeping the all the other parameters the same).
• Redirect the output of the utility to a log file that you can analyze at your leisure during and after the migration.
• Errors:  the migration tool may encounter problems copying some objects or datastreams.  The tool by default will halt at the first error;  if you wish to migrate as much as possible then go back and address errors, run the tool with the --continue-on-error flag.  Objects or datastreams with errors will not be written to the OCFL repository;  they will be skipped, and the next object in the list will be processed.
  Objects that could not be migrated will provoke a stack dump in the log, marked with the string ERROR.  They can be extracted from the log at a later date and fixed, then migrated individually.
• Warning:  the migration utility is not idempotent!  This means that if you run the utility twice over the same content to the same target, you will wind up with an OCFL repository with duplicate versions of datastreams.  For the purposes of testing, delete or move out of the way previous migration attempts before running a new migration.
  However, you can migrate new objects to an already-existing OCFL repository, which means you can plan your migration in stages, if you desire.  Note that you will need to rebuild your Fedora 6 index after the new additions.
• When adding new or fixed objects to the OCFL repository, make sure to regenerate a fresh datastream index.
• As many files are created per object and per datastream version, make sure to allocate enough inodes on your system to allow for all the files (note:  this should only be necessary for extremely large repositories, more than 4 million objects, for example).
• See Fedora 3 to 6 Migration Community Updates for examples of medium- and large-scale migrations performed with the migrations tool, with benchmark data and detailed notes.

Example migration workflow

Select 1000 pids or so from your Fedora 3 repository, and migrate just those objects to an OCFL repository.  Start up Fedora 6 on the sample OCFL tree, and examine.  You may also look at the contents of the OCFL repository on the filesystem – the structure and contents are easily visible to the naked eye.  Use the sample run to determine the disk space required, and time needed to migrate your whole repository.

Full migration:

1. Shut down  your Fedora 3 repository or put it in offline mode.
2. For a partial migration, create a file with a list of PIDs to be migrated (one PID per line).  You may create several PID list files, one for each batch of objects to migrate.  You can migrate just those objects in the file with the tool's --pid-file parameter.
3. Run the migration utility, redirecting the output to a log file.  Run it with the --continue-on-error flag.
4. After the migration has finished, look for objects with errors in the log file.  Fix them, if possible, and create a new PID file with just the PIDs of the fixed objects.
5. Delete the datastream index directory, and re-run the migration utility, using the PID file with the fixed PIDs.
6. Repeat steps 4 and 5 until all objects in the PID batch are successfully migrated (or dropped from migration).
7. If running partial migrations in batches, repeat the above steps with the next batch of pids.

Once you are satisfied with the migration, start up Fedora 6 on top of your new OCFL repositoryBelow you will find an example migration.  Please refer to the migration-utils README for a full discussion of the available options.

A Sample Migration From 3→6

...