This document details both the categories of tests that help in understanding Fedora's performance and scalability characteristics, as well as the tools used to perform those tests. This documentation is an aggregation of various testing initiatives from within the Fedora community over the last several years. Although the tooling has been updated to work with the most recent version of Fedora (6.0), it is important that the results produced through the current tests be compared to historical results in order to better understand the changes (improvements or degradations) in Fedora's performance characteristics over time.
API specification compliance
What is Fedora's degree of compliance with the Fedora API specification?
What is Fedora's degree of compliance with the OCFL specification? Tests related to this category should be run against both the OCFL produced by Fedora, as well as the OCFL produced by the Fedora 3 to 6 migration tooling.
What are Fedora's ingest performance characteristics? Specifically, how much content can reasonably be stored in Fedora (number of objects, number of petabytes)? How long does it take to ingest a large corpus of data into Fedora?
What are Fedora's read performance characteristics? Specifically, how does the response time of various read requests change as the size of the repository grows?
How do Fedora's ingest and read performance characteristics change with auto-versioning on or disabled?
What are Fedora's query response time characteristics for common and complex queries as the size of the repository grows?
Migration performance and validation
What are the migration times from Fedora 3 to an OCFL representation for a variety of corpus sizes (number of objects, number of petabytes)? What are the rebuild times for Fedora 6 layered over the OCFL representation?
What is the speed of the rebuild with different data profiles, database engines, and instance types?
How is rebuild performance affected by instance size (cpu count), database engine, database instance size?
- Bash scripts used for measuring timing of fixed-scope operations.
- Relevant for: Ingest performance, Read performance, Versioning performance, Search performance
- Detailed usage documentation
- JMeter scripts used for measuring limits of scale and stress. These tests run until the application dies.
- Fedora API Test Suite is used to measure compliance of the Fedora application with the Fedora API specification.
- Python OCFL validator used to measure compliance of an OCFL structure with the OCFL specification.
- Fedora 3 to 6 migration tooling used for migrating F3 objects into OCFL over which F6 can be layered.
Current test results can be found on the Fedora 6 Test Results page.
One of the benefits of running these tests is to measure the change in performance characteristics in relation to previous versions of Fedora. Ideally, one of the products of this testing initiative will be to have a clear depiction, across the test categories, of performance result summaries from the past and present.
Although a consolidated summary of previous tests is not available in a single location, the following references point to the locations where previous test results can be found:
- Results from "bash scripts"
- Results from "JMeter scripts"
- Results from "Fedora API Test Suite"
- Results from very early, ad hoc testing