Table of Contents |
---|
Alpha-1
...
Test Results
The most recent Alpha-1 benchmark test results I see are from June 2013:
These tests ingested a very small number of objects/containers (10), and files ranging from 1 byte to 100MB.
...
Alpha-3 Test
...
- Number of Objects: In some initial testing on my laptop, I noticed degraded ingest performance as the repository size grew, particularly after 20K objects.
- Datastream Size: In my experience, the larger files in those tests (1MB, 8MB, 100MB) are fairly typical for high-res images and audio/video derivatives, and audio/video masters are much larger.
Testing full-scale ingest would take more disk space and time than we have available. So it probably makes more sense to do two separate tests:
- Many Objects: 25K objects with small datastreams (100KB): 2.5GB total data size
- Large Datastreams: 100 objects with larger datastreams (50MB): 5GB total data size
In addition to varying the number of objects and datastream size, it is particularly important to perform some tests on the same hardware in order to have directly-comparable data:
- Thread Test: Run the same test on the same hardware with increasing numbers of client threads
- Fedora 3/4 Comparison: Run the same tests on the same hardware with both Fedora 3 and 4
Test Matrix
Fedora 3/4 Comparison
Objects | Datastream | Fedora 3 Time (ms) | Fedora 4 | Notes |
---|---|---|---|---|
100 | 50MB | 389,034 | 281,301 | Performance evaluation on the SCC Cluster #A & #C |
100 | 50MB | 881,883 | Performance evaluation on AWS #C | |
100 | 50MB | 8,200 | 70,728 | Test - Cluster Setup - Yale #A & #C |
25,000 | 100KB |
Thread Test
Objects | Datastream | 1 Thread | 2 Threads | 3 Threads | 4 Threads |
---|---|---|---|---|---|
100 | 50MB | ||||
25,000 | 100KB |
Using Benchtool
The ingest benchmarking tool is available on github: https://github.com/futures/benchtool
To build and run benchtool (creating 1000 objects with 1KB datastreams using 5 client threads):
Code Block |
---|
git clone git@github.com:futures/benchtool.git
cd benchtool
mvn install
java -cp target/bench-tool-0.0.1-SNAPSHOT-jar-with-dependencies.jar org.fcrepo.bench.BenchToolFC4 http://localhost:8080 1000 1024 5 |
While running benchtool will output a running progress percentage, and then output the time to perform the ingest:
...
Results
The Alpha-3 benchmark test results from December 2013:
Create/Read/Update/Delete Test Results
Testing was performed to create, read, update and delete objects/containers in Fedora 3, Fedora 4-alpha3, and Fedora 4-alpha5 (4-current), using the cURL command-line tool (Fedora 3 test script, Fedora 4 test script). Each test run processed 3,000 objects/containers, and each object had a single 10MB binary. Each repository was tested twice and the average times are summarized below.
Platform | Repository | Objects/Containers | Binary Size | Ingest (sec) | Read (sec) | Update (sec) | Delete (sec) |
---|---|---|---|---|---|---|---|
lib-devsandbox1.ucsd.edu | Fedora 3.7.1 | 3,000 | 10 MB | 1,343 | 813 | 1,195 | 42 |
lib-devsandbox1.ucsd.edu | Fedora 4.0-alpha3 | 3,000 | 10 MB | 1,396 | 798 | 1,202 | 139 |
lib-devsandbox1.ucsd.edu | Fedora 4.0-alpha5 (4-current) | 3,000 | 10 MB | 1,214 | 613 | 1,112 | 121 |