...
- Storage environment: for the purposes of this test (and for our real migration), we are migrating from one CIFS-mounted remote filesystem to another CIFS-mounted remote filesystem.
- Datastream index: takes about 1h10m minutes to build, and occupies 327MB of disk space.
- Source layout. Akubra hash storage, using the pattern "#/##/##" for both datastreams and objects.
- Average seconds per object is calculated based on the difference between the time the first object is processed (after the datastream index has been generated) and the time the last object is processed.
Issues
Migration Tests
UW Digital Collections Center Production Repository
...
| Code Block | ||||
|---|---|---|---|---|
| ||||
$ java -jar target/migration-utils-4.4.1-SNAPSHOT-driver.jar --migration-type=FEDORA_OCFL --source-type=akubra --datastreams-dir=/fedora3-prod/fedora/datastreams --objects-dir=/fedora3-prod/fedora/objects --target-dir=/fedora-migration-test --index-dir=/var/tmp/datastream-index |
Number | Execution | Average seconds per object | OCFL repository size |
|---|
Layout
Migration | Notes | |
|---|---|---|
| 1000 | Datastream index: 1h17m OCFL repo: 4h36m |
| 2. |
| 9 sec |
| 133GB |
| with param --pid-file=1000pids.txt datastream index cleared after run | |
| 10,000 | Datastream index: 1h5m OCFL repo: 11h48m | 4.3 sec |
| 147GB | (81586bf ) | with param --pid-file=10000pids.txt Most objects are XML docs in this batch. |
| 100,000 | Datastream index: 1h9m OCFL repo: 3d20h16m | 3.3 sec |
| 1.6TB | | with param --pid-file=100000pids.txt datastream index cleared after run |
| All 561,000 |
Datastream index: 1h10m | 3.2 sec | 9TB | | all pids |