Performance Testing Scenarios
Performance testing may be simplest if real-world scenarios are used that are drawn from the use patterns for Fedora 4. In other words, performance testing should be informed by the expected way Fedora will be used. No single software product can perform well for every possible use (a.k.a pattern of use). We need to define the expected uses for Fedora, and acknowledge the trade-offs that are being made. While we need to recognize that there will be unanticipated uses, we can only test for the cases that best characterize how we expect Fedora to be used. This sets expectations for those building systems using Fedora and guidance for Fedora committers. When there is a use is identified during system design, developers can decide if Fedora is suitable as a part of their implementation. Fedora committers can decide either Fedora can be extended to support that use or suggest how Fedora can be used in combination with other tools to support that use.
There are four major categories of use that help us construct a realistic performance test suite.
Authoring
Authoring is the activity of creating or assembling new content. This includes both constructing wholly new content and referencing existing content. It is different from ingest in that is characterized by incremental assembly of the content which many rapid write/read cycles. It is often performed as part of some sort of authoring workflow commonly with multiple actors performing both overlapping and different roles. During this phase the content (and metadata) is rapidly changing.
Small Ingest
Small Ingest consist of upload of single or small amounts of content and metadata
Additional Testing Scenarios
These scenarios expand on the previous single stimulus tests to use multiple read, write, and read-write tests via the REST api.
...
Note: all tests need to be taken until:
a steady state is achieved
a declining state is achieved
Fedora 4 no longer responds
Multiple Read
Stimulators
1 stimulator to provide a baseline similar to the previous testing regimen
3 stimulators
6 stimulators (since this is where Fedora 3 starts to exhibit limits)
12 stimulators (since this is where Fedora 3 always exhibits limits)
24 stimulators
Payloads
1K file
1M file
50M file (Avg Video)
2.7G file (DVD)
Rates
Step up rates X2 until flat line
Then proceed to declining response and failure or non-response
Multiple Write
Stimulators
1 stimulator to provide a baseline similar to the previous testing regimen
3 stimulators
6 stimulators (since this is where Fedora 3 starts to exhibit limits)
12 stimulators (since this is where Fedora 3 always exhibits limits)
24 stimulators
Payloads
1K file
1M file
50M file (Avg Video)
2.7G file (DVD)
Rates
Step up rates X2 until flat line
Then procede to declining response and failure or non-response
Read-Write
Stimulators
1 stimulator to provide a baseline similar to the previous testing regimen
3 stimulators
6 stimulators (since this is where Fedora 3 starts to exhibit limits)
12 stimulators (since this is where Fedora 3 always exhibits limits)
24 stimulators
Payloads
This needs to be matrixed. The payloads should be mixed but not randomly to make the tests repeatable.
1K file
1M file
50M file (Avg Video)
2.7G file (DVD)
Rates
Step up rates X2 until flat line
Then proceed to declining performance and failure or non-response