Performance Testing Scenarios
Performance testing may be simplest if real-world scenarios are used that are drawn from the use patterns for Fedora 4. In other words, performance testing should be informed by the expected way Fedora will be used. No single software product can perform well for every possible use (a.k.a pattern of use). We need to define the expected uses for Fedora, and acknowledge the trade-offs that are being made. While we need to recognize that there will be unanticipated uses, we can only test for the cases that best characterize how we expect Fedora to be used. This sets expectations for those building systems using Fedora and guidance for Fedora committers. When there is a use is identified during system design, developers can decide if Fedora is suitable as a part of their implementation. Fedora committers can decide either Fedora can be extended to support that use or suggest how Fedora can be used in combination with other tools to support that use.
There are four major categories of use that help us construct a realistic performance test suite.
Authoring
Authoring is the activity of creating or assembling new content. This includes both constructing wholly new content and referencing existing content. It is different from ingest in that is characterized by incremental assembly of the content which many rapid write/read cycles. It is often performed as part of some sort of authoring workflow commonly with multiple actors performing both overlapping and different roles. During this phase the content (and metadata) is rapidly changing.
Small Ingest
Small Ingest consist of upload of single or small amounts of content and metadata. It can be accomplished with a single atomic operation or a short series of RESTful operations without resorted to a read prior completion. The duration of the small ingest is expected to be approximately the time it takes to upload the content and metadata starting with the beginning of the connection, where the connection is terminated after the operation. It is expected that the ability to read (access) the uploaded content and metadata should happen fairly soon after the upload is complete.
Bulk Ingest
In bulk ingest, a large quantity of content and metadata is ingested as a logical unit or is continuous. This likely happen over a number of connections or may utilize methods that are optimized for bulk ingest. It is characterized by the expectation that there may be a defined delay between when the ingest is started and part or all of the content and metadata becomes available for read.
Additional Testing Scenarios
These scenarios expand on the previous single stimulus tests to use multiple read, write, and read-write tests via the REST api.
If the group think this is useful then I can break it into a matrix.
Note: all tests need to be taken until:
a steady state is achieved
a declining state is achieved
Fedora 4 no longer responds
Multiple Read
Stimulators
1 stimulator to provide a baseline similar to the previous testing regimen
3 stimulators
6 stimulators (since this is where Fedora 3 starts to exhibit limits)
12 stimulators (since this is where Fedora 3 always exhibits limits)
24 stimulators
Payloads
1K file
1M file
50M file (Avg Video)
2.7G file (DVD)
Rates
Step up rates X2 until flat line
Then proceed to declining response and failure or non-response
Multiple Write
Stimulators
1 stimulator to provide a baseline similar to the previous testing regimen
3 stimulators
6 stimulators (since this is where Fedora 3 starts to exhibit limits)
12 stimulators (since this is where Fedora 3 always exhibits limits)
24 stimulators
Payloads
1K file
1M file
50M file (Avg Video)
2.7G file (DVD)
Rates
Step up rates X2 until flat line
Then procede to declining response and failure or non-response
Read-Write
Stimulators
1 stimulator to provide a baseline similar to the previous testing regimen
3 stimulators
6 stimulators (since this is where Fedora 3 starts to exhibit limits)
12 stimulators (since this is where Fedora 3 always exhibits limits)
24 stimulators
Payloads
This needs to be matrixed. The payloads should be mixed but not randomly to make the tests repeatable.
1K file
1M file
50M file (Avg Video)
2.7G file (DVD)
Rates
Step up rates X2 until flat line
Then proceed to declining performance and failure or non-response
