Page tree
Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 15 Next »

NLM

Observations

  1. External datastreams.  Most of our binaries are of type E external.  The migration tool migrates the Fedora objects, but not the type E external binaries (as expected).  Thus we are left with object structure, metadata and RDF in OCFL format, but not the actual binaries themselves.  If, how, when and where to migrate external binaries to an OCFL structure is TBD, but a major consideration for us in adopting OCFL.
  2. Speed. The tool migrates objects at the rate of 15K-40K objects per hour. This should be manageable for our purpose.
  3. For the "citations" repository, it consistently takes 30 minutes to build the datastream index before starting the migration. This server has 3.8M managed datastreams (1 per object). The option to cache this index when resuming migrations is helpful.
  4. CPU time. Consumes about 30%.
  5. Layout. In flat and pairtree migrations the PID is used to form the path; for example PID nlm:nlmuid-101588995-bk (stored FOXML file name nlm_nlmuid-101588995-bk) becomes /ocfl/nl/m+/nl/mu/id/-1/01/58/89/95/-b/k/5-bk. Characters such as – are problematic in Linux.  See  FCREPO-3180 - Getting issue details... STATUS .
  6. It would be nice to declare use of another field, or input map, to dictate the value to use for layout path generation. For example, it may be nice to use 101588995_bk to generate a path for PID nlm:nlmuid-101588995-bk.  Also included in  FCREPO-3180 - Getting issue details... STATUS .
  7. Migrated datastreams have no file extension. It would be nice if migrated datastreams have a file extension inferred from the MIME type; e.g. DC.xml instead of just DC, and OCR.txt instead of just OCR. This should particularly help out with in-line XML datastreams.   FCREPO-3181 - Getting issue details... STATUS
  8. OCFL versions appear to be created based on datastream timestamps. Each unique timestamp creates a new OCFL version, even if they were part of the same Fedora version in the AUDIT trail and differed only by milliseconds.
  9. Add XML declarations for migrated in-line datastreams.   FCREPO-3197 - Getting issue details... STATUS

Issues

  1. Occasional "Unable to delete staging" messages, which stops a migration.  See  FCREPO-3187 - Getting issue details... STATUS  and  FCREPO-3191 - Getting issue details... STATUS .
  2. Migrated objects from the "collections" repository generate approx. 40 files in OCFL from a single Fedora 3 FOXML file.  This repo with 4M objects could generate 160M files.  We have already run out of filesystem inodes when attempting migration runs.  It may be useful to optimize the number of generated files to mitigate this issue.
  3. Migrated external datastreams are not resolvable.  FCREPO-3198 - Getting issue details... STATUS

Migration Tests

This section logs migration tests for our two different repository environments, "collections" and "citations".

Repository environment #1: "collections". 

Approx. 4.3M records in legacy format, mostly books, pages and still images. Objects have many datastreams, binaries are generally type E external.

Fedora 3.8.1.  VM with 4 cores, 8 GB RAM.  CentOS release 6.10 (Final), Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz

Number

of objects

Execution

Time

Source

Layout

Dest.

Layout

Migration

tool version

Notes
10004 minlegacypairtree11/26/191K fedora items produced 42K+ files
10003 minlegacytruncated11/26/191K fedora items produced 43K+ files
100,0006.5 hourslegacyflat11/26/19
1 million~3 dayslegacypairtree11/26/19Execution crashed twice for "unable to delete staging" file issues, resume option had no issues running
full run (4,656,669 items)7 dayslegacypairtree2/4/20No issues observed for successful full migration run.  Required deployment of new filesystem with large inode limit.


Repository environment #2: "citations". 

Approx. 3.8M records in akubra format, all citations with one small type M XML datastream (the citation payload).

Fedora 3.8.1.  VM with 1 core, 8 GB RAM.  CentOS release 6.10 (Final), Intel(R) Xeon(R) CPU E5-2690 v3 @ 2.60GHz

Number

of objects

Execution

Time

Source

Layout

Dest.

Layout

Migration

tool version

Notes
200032 minakubrapairtree11/26/19Includes 30 min to build the index.  Hung on completion-could not delete index.
10,00042 minakubraflat11/26/19Includes 30 min to build the index.  Hung on completion-could not delete index.
554,69513 hoursakubraflat11/26/19Attempted to migrate 1M records.  Includes 30 min to build the index.  Crashed due to 

UnrecognizedPropertyException.

full run (3,830,777 items)5 daysakubratruncated2/4/20No issues observed for successful full migration run.


Brown Univ


University of Wisconsin - Madison

Observations

  1. Storage environment:  for the purposes of this test (and for our real migration), we are migrating from one CIFS-mounted remote filesystem to another CIFS-mounted remote filesystem.
  2. Speed: The tool migrates approximately 1700 objects/hr.  At this rate, it will take approximately 10 days to migrate the entire repository,
  3. Datastream index:  takes about 67 minutes to build.
  4. CPU time. Consumes about 15%.
  5. Source layout.  Akubra hash storage, using the pattern "#/##/##" for both datastreams and objects.
  6. OCFL storage:  Pairtree.  It will be good when the OCFL storage profile specification is set and incorporated into migration-utils, so that we can define the OCFL layout, similar to how we can specify the Akubra filesystem layout.

Issues

Migration Tests

UW Digital Collections Center Production Repository

Fedora 3: Approx. 390,000 objects (382GB): mostly books, pages and still images, with some audio, video, and PDF resources.  Approximately 2.33M datastreams (6.3TB). Content objects have one binary datastream and 5 XML metadata datatstreams.  Container objects have ~5 XML metadata datastreams.  All datastreams are either inline or managed (no external or redirect datastreams).

Fedora 3.8.1.  Migration run on desktop workstation with 8 cores, 16 GB RAM.  CentOS Linux release 7.7.1908 (Core), Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz

Command run:

UW Madison migration-util command line
$ java -jar target/migration-utils-4.4.1-SNAPSHOT-driver.jar --source-type=akubra --datastreams-dir=/fedora3-prod/fedora/datastreams --objects-dir=/fedora3-prod/fedora/objects --target-dir=/fedora-migration-test --layout=pairtree --index-dir=/var/tmp/datastream-index


Number

of objects

Execution

Time

Source

Layout

Dest.

Layout

Migration

tool version

Notes
Datastream index1 hr 7 minutesAkubrapairtree02/02/20 (cd7ece7)233MB
100037 minAkubrapairtree02/02/20 (cd7ece7)1K fedora items produced 33.5K+ files
100,00059 hours 58 minAkubrapairtree02/02/20 (cd7ece7)Stopped and restarted;  time excludes read time for datastream index on startup (29 minutes)
All 390,000X hoursAkubrapairtree02/02/20 (cd7ece7)
  • No labels