...
Download of large amount of content as single user operation. This may require any number of repository operations to accomplish. Whether content and metadata stays fixed from the beginning to the end of the operation is to be defined. This is needs consideration a whole intellectual entity, graph or DIP is considered the unit. Also we need to consider what this means for continuous access operations.
Tests
Baseline and Simple Concurrency Testing Matrix
The tests in the table will start with a single load injector and worker to use as a baseline. Then each of the tests are executed to test concurrency with increasing numbers of load injectors and workers until performance declines and/or error rates become large.
Category | User Operation | Repository Operation | Test Metric | Test | Priority | Notes |
---|---|---|---|---|---|---|
Simple Ingest | Small Files | Rate | 1 | Increase Load Injectors until max rate is found. Synthetic data is acceptable for this test | ||
Simple Ingest | Medium Files | Rate | 1 | Increase Load Injectors until max rate is found. Synthetic data is acceptable for this test | ||
Simple Ingest | Large Files | Rate | 1 | Increase Load Injectors until max rate is found. Synthetic data is acceptable for this test | ||
Simple Ingest | Media Files | |||||
Simple Ingest | Large File Count | Rate - Normalized Count | 1 | Ingest files to a substantial number to explore maximum file count. Normalized to ignore size of a given file. | ||
Bulk Ingest | Small Files | |||||
Bulk Ingest | Medium Files | |||||
Bulk Ingest | Large Files | |||||
Bulk Ingest | Large File Count | Rate - Normalized Count | Ingest files to a substantial number to explore maximum file count | |||
Simple Access | Small Files | Random Access | 1 | Increase Load Injectors until max rate is found. Site should contain a set of files of uniform size. Tests can vary download mix by URL. It is essential that caching be avoided. Synthetic data is acceptable for this test. | ||
Simple Access | Medium Files | Random Access | 1 | Increase Load Injectors until max rate is found. Site should contain a set of files of uniform size. Tests can vary download mix by URL. It is essential that caching be avoided. Synthetic data is acceptable for this test. | ||
Simple Access | Large Files | Random Access | 1 | Increase Load Injectors until max rate is found. Site should contain a set of files of uniform size. Tests can vary download mix by URL. It is essential that caching be avoided. Synthetic data is acceptable for this test. | ||
Simple Access | Mixed Files | Random Access | 1 | Increase Load Injectors until max rate is found. Site should contain a set of files of all three sizes. Tests can vary download mix by URL. It is essential that caching be avoided. Synthetic data is acceptable for this test. | ||
Simple Access | Large File Count | Random Access | 1 | Increase Load Injectors until max rate is found. Access a large number of files until time is exhausted. | ||
Bulk Ingest | Small Files | |||||
Bulk Ingest | Medium Files | |||||
Bulk Ingest | Large Files | |||||
Bulk Ingest | Mixed Files |
Concurrency Tests
The tests in the table combine two or more different tests from the table above. It starts with concurrent operation of at least two workers, one for each simple test. Then each of the tests are executed with increasing numbers of load injectors and workers until performance declines and/or error rates become large.Concurrency Tests
Category | User Operation | Repository Operation | Test Metric | Test | Priority | Notes |
---|---|---|---|---|---|---|
Concurrent Test #1 | Simple Access Mixed Files Simple Ingest Mixed Files | Rate Random Access | 1 | Number of Load Injectors TBD Number of Load Injectors TBD | ||
Concurrent Test #N | Authoring Simulation | Rate Directed Access |
...
These scenarios expand on the previous single load injector tests to use multiple read, write, and read-write tests via the REST api.
If the group think this is useful then I can break it into a matrix.
Note: all tests need to be taken until:
a steady state is achieved
a declining state is achieved
Fedora 4 no longer respondsexhibits high error rates or stops responding
Load Injectors
1 Load Injector to provide a baseline similar to the previous testing regimen
3 Load Injectors
6 Load Injectors (since this is where Fedora 3 starts to exhibit limits)
12 Load Injectors (since this is where Fedora 3 always exhibits limits)
24 Load Injectors
...
Step up rates X2 until flat line
Then proceed to declining response and failure , high error rates or non-response
Fedora Configurations
- Not Clustered
- Clustered
- Not ReplicatedNot
- Replicated
Resources
- https://github.com/pinterest/bender/blob/master/http/TUTORIAL.md
- https://github.com/fcrepo4-labs/fcrepo-test-grinder
- http://grinder.sourceforge.net/index.html