Alpha-1 Test Results
The Alpha-1 benchmark test results are from June:
These tests ingested a very small number of objects (10), and files ranging from 1 byte to 100MB.
Alpha-3 Test Parameters
- Number of Objects: In initial testing, all nodes were created as children of a single node. This resulted in degraded ingest performance as the repository size grew, particularly after 20K objects.
- Datastream Size: In my experience, the larger files in those tests (1MB, 8MB, 100MB) are fairly typical for high-res images and audio/video derivatives, and audio/video masters are much larger.
Testing full-scale ingest would take more disk space and time than we have available. So it probably makes more sense to do two separate tests:
- Many Objects: 25K objects with small datastreams (100KB): 2.5GB total data size
- Large Datastreams: 100 objects with larger datastreams (50MB): 5GB total data size
In addition to varying the number of objects and datastream size, it is particularly important to perform some tests on the same hardware in order to have directly-comparable data:
- Thread Test: Run the same test on the same hardware with increasing numbers of client threads
- Fedora 3/4 Comparison: Run the same tests on the same hardware with both Fedora 3 and 4
Ingest Test Matrix (Profile)
Fedora 3/4 Comparison
Objects | Datastream | Fedora 3 Time (ms) | Fedora 4 | Notes |
---|---|---|---|---|
100 | 50MB | 389,034 | 281,301 | Performance evaluation on the SCC Cluster (Tests #A & #C included here) |
100 | 50MB | 881,883 | Performance evaluation on AWS #C | |
100 | 50MB | 123285 | 70,728 | Test - Cluster Setup - Yale #A & #C |
100 | 50MB | 57,814 | 134,471 | Test - Platform Profile: Single VM at UNC Chapel Hill |
100 | 50MB | 231,663 | 131,143 | Platform: Test - Platform Profile: lib-devsandbox1.ucsd.edu, Repository: Test - Repository Profile: Minimal , average of 100 ingest batches |
Using Benchtool
The ingest benchmarking tool is available on github: https://github.com/futures/benchtool
To build and run benchtool against Fedora 4 (creating 1000 objects with 1KB datastreams using 5 client threads):
git clone git@github.com:futures/benchtool.git cd benchtool mvn install java -cp target/bench-tool-0.0.1-SNAPSHOT-jar-with-dependencies.jar org.fcrepo.bench.BenchToolFC4 http://localhost:8080 1000 1024 5 ingest
To run against Fedora 3 (creating 1000 objects with 1KB datastreams):
java -cp target/bench-tool-0.0.1-SNAPSHOT-jar-with-dependencies.jar org.fcrepo.bench.BenchToolFC3 http://localhost:8983/fedora fedoraAdmin fedoraAdmin 1000 1024 ingest
Running benchtool will output a running progress percentage, and then output the time to perform the ingest:
Complete ingest of 1000 files took 23301 ms
Ingest/Read/Update/Delete Test Results
Platform/Repository Profiles | Repository | Objects | Datastream Size | Ingest (ms) | Read (ms) | Update (ms) | Delete (ms) |
---|---|---|---|---|---|---|---|
lib-devsandbox1.ucsd.edu, Repository: Minimal | fcrepo3/hydra-jetty | 1,000 | 50 MB | 470,466 | 2,198,425 | 10,787 | |
lib-devsandbox1.ucsd.edu, Repository: Minimal | fcrepo4 | 1,000 | 50 MB | 527,349 | 1,287,506 | 39,336 | |
lib-devsandbox1.ucsd.edu | fcrepo3/hydra-jetty | 1,000 | 10 MB | 485,700 | 107,649 | 438,001 | 52,611 |
lib-devsandbox1.ucsd.edu | fcrepo4 (2707674c86e5a82df82f937fb87c6a1a259c0bd3) | 1,000 | 10 MB | 334,177 | 144,367 | 324,908 | 45,334 |
lib-devsandbox1.ucsd.edu | barmintor/fcrepo4 (a0a6f1af258f1247c824e30a186f38c2071395a3) (with custom ispn config) | 1,000 | 10 MB | 304,447 | 144,682 | 281,381 | 37,805 |
lib-devsandbox1.ucsd.edu + NAS + file | fcrepo4 (ceae23e7a18b30a9aa283e5a66da290a551fe8ab) | 1,000 | 10 MB | 331,110 | 155,743 | 354,122 | 45,119 |
lib-devsandbox1.ucsd.edu + NAS + leveldb | fcrepo4 (ceae23e7a18b30a9aa283e5a66da290a551fe8ab) | 1,000 | 10 MB | 344,792 | 167,567 | 366,714 | 45,689 |
lib-devsandbox1.ucsd.edu + local disk + file | fcrepo4 (ceae23e7a18b30a9aa283e5a66da290a551fe8ab) | 1,000 | 10 MB | 235,597 | 169,930 | 253,178 | 45,492 |
lib-devsandbox1.ucsd.edu + local disk + leveldb | fcrepo4 (ceae23e7a18b30a9aa283e5a66da290a551fe8ab) | 1,000 | 10 MB | 228,773 | 161,836 | 246,275 | 42,535 |
Ingest/Read/Update/Delete Test Results (Rewritten Benchtool)
A complete rewrite of our benchmarking tool in early December 2013 means that the results are not directly-comparable to earlier results with the previous version.
Platform/Repository Profiles | Repository | Objects | Datastream Size | Ingest (ms) | Read (ms) | Update (ms) | Delete (ms) |
---|---|---|---|---|---|---|---|
lib-devsandbox1.ucsd.edu | fcrepo3/hydra-jetty | 1,000 | 10 MB | 510,534 | 39,862 | 518,293 | 33,446 |
lib-devsandbox1.ucsd.edu Repository: Minimal | fcrepo4 (44edb61e179204e272f0937c17cff08f35d5a471) | 1,000 | 10 MB | 278,235 | 21,242 | 288,054 | 23,744 |
lib-devsandbox1.ucsd.edu Repository: Minimal config: custom | fcrepo4 (44edb61e179204e272f0937c17cff08f35d5a471) | 1,000 | 10 MB | 265,231 | 16,562 | 250,832 | 5,473 |
lib-devsandbox1.ucsd.edu | fcrepo4 (ae92a7eaeee55f55c4452e2dc66cf778f07bb315) | 1,000 | 10 MB | 394,888 | 19,226 | 378,856 | 116,420 |