Page tree
Skip to end of metadata
Go to start of metadata

This document details both the categories of tests that help in understanding Fedora's performance and scalability characteristics, as well as the tools used to perform those tests. This documentation is an aggregation of various testing initiatives from within the Fedora community over the last several years. Although the tooling has been updated to work with the most recent version of Fedora (6.0), it is important that the results produced through the current tests be compared to historical results in order to better understand the changes (improvements or degradations) in Fedora's performance characteristics over time.

Note: This testing approach was initially structured by the Product Technology Subgroup's: Fedora 6 Test Plan

Test Categories

API specification compliance

What is Fedora's degree of compliance with the Fedora API specification?

OCFL compliance

What is Fedora's degree of compliance with the OCFL specification? Tests related to this category should be run against both the OCFL produced by Fedora, as well as the OCFL produced by the Fedora 3 to 6 migration tooling.

Ingest performance

What are Fedora's ingest performance characteristics? Specifically, how much content can reasonably be stored in Fedora (number of objects, number of petabytes)? How long does it take to ingest a large corpus of data into Fedora?

Read performance

What are Fedora's read performance characteristics? Specifically, how does the response time of various read requests change as the size of the repository grows?

Versioning performance

How do Fedora's ingest and read performance characteristics change with auto-versioning on or disabled?

Search performance

What are Fedora's query response time characteristics for common and complex queries as the size of the repository grows?

Migration performance and validation

What are the migration times from Fedora 3 to an OCFL representation for a variety of corpus sizes (number of objects, number of petabytes)? What are the rebuild times for Fedora 6 layered over the OCFL representation?

Rebuild performance

What is the speed of the rebuild with different data profiles, database engines, and instance types?

How is rebuild performance affected by instance size (cpu count), database engine, database instance size?


Test Tooling

  1. Bash scripts used for measuring timing of fixed-scope operations.
    1. https://github.com/fcrepo4-labs/fcrepo-performance-test-scripts
    2. Relevant for: Ingest performance, Read performance, Versioning performance, Search performance
    3. Detailed usage documentation
  2. JMeter scripts used for measuring limits of scale and stress. These tests run until the application dies.
    1. https://github.com/fcrepo4-labs/fcrepo4-jmeter
    2. Relevant for: Ingest performance, Read performance, Versioning performance
    3. Detailed usage documentation
  3. Fedora API Test Suite is used to measure compliance of the Fedora application with the Fedora API specification.
    1. https://github.com/fcrepo/Fedora-API-Test-Suite
    2. Relevant for: API specification compliance
    3. Detailed usage documentation
  4. Python OCFL validator used to measure compliance of an OCFL structure with the OCFL specification.
    1. https://github.com/zimeon/ocfl-py
    2. Relevant for: OCFL compliance, Migration performance and validation
    3. Detailed usage documentation
  5. Fedora 3 to 6 migration tooling used for migrating F3 objects into OCFL over which F6 can be layered.
    1. https://github.com/fcrepo-exts/migration-utils
    2. Relevant for: Migration performance and validation
    3. Detailed usage documentation

Test Results

Current test results can be found on the Fedora 6 Test Results page.

Historical Results

One of the benefits of running these tests is to measure the change in performance characteristics in relation to previous versions of Fedora. Ideally, one of the products of this testing initiative will be to have a clear depiction, across the test categories, of performance result summaries from the past and present.

Although a consolidated summary of previous tests is not available in a single location, the following references point to the locations where previous test results can be found:

  1. Results from "bash scripts"
    1. Many Members Performance Testing
  2. Results from "JMeter scripts"
    1. Test 1 Results Summary
    2. Test 2 Results Summary
    3. Test 3 Results Summary
    4. Test 4 Results Summary
    5. Test 5 Results Summary
  3. Results from "Fedora API Test Suite"
    1. spec-tests
  4. Results from very early, ad hoc testing
    1. 2014-08-20 - Performance Summary



  • No labels