Overview

These tests will be conducted using The Grinder.  It is a load testing framework written in Java, that uses Java, Jython and/or Clojure for writing the tests.  There is always a degree of overlap between functional tests and performance test since we will be using Fedora 4 functions. However, we are only testing performance in this activity.  We won't repeat general documentation on The Grinder here but will specify our configuration for performing tests.  These tests will primarily be performed using the AWS cloud so we will not have full control of our infrastructure and we will not be able to eliminated variable outside of our control.  However, we can run the tests in a consistent configuration and long enough that metrics should provide an average performance.  This is presuming we avoid I/O bottlenecking of the network or storage resources.  It would be best if we monitor CPU usage because we will likely be sharing servers with other virtual machines.

The same load testing infrastructure can be used for bottleneck analysis at a later time.

Basic Configuration

This section describes the logical configuration.  Detailed configurations will vary depending on the load we need to place on Fedora 4 to test its limits.  It is helpful to understand the basic components of the testing framework. 

• Worker processes
  • Interprets test scripts and performs the tests.
  • Each worker process can run many tests in parallel using a number of worker threads.
• Agent processes
  • Long running process that starts and stops worker processes as required.
  • Maintains a local cache of test scripts distributed from the console.
• The Console
  • Coordinates the other processes.
  • Collates and displays statistics.
  • Provides script editing and distribution.

Basic Deployment

Load Injection Machine

• M3 Small - At least 2 of them
• A Small may have adequate I/O because the load injector usually needs less I/O bandwidth than the system under test
• The CPU and I/O bandwidth will determine the number of worker we can run though CPU is rarely an issue
• Since the configuration of the machine including the tests to be executed may be set prior to running a test procedure, a single AMI should be sufficient for all test.

System under Test

• This is a Fedora 4 installation
• It should be as self contained as possible in the initial tests with all the dependencies loaded on the same server
• Ephemeral storage, preferably SSDs should be used to avoid storage I/O bottlenecks to baseline the code performance, then switch to persistent EBS disks for a realistic performance test
• M3 2XL.  We need to eliminate as many variables as possible.  According to AWS, we will be the only virtual machine on that size server.  To manage cost the server need to be operating only during the test.  Later we can try smaller servers to see if we get a predictable delta in performance.
• Likely this can be a single AMI though we need to consider how to change configurations.  This would be a good candidate for managing via Puppet

Console

• The Grinder can be run entirely from the command line but the Console is very helpful during "testing the test" and setting up configurations
• The Console can be set up two ways:
  • Use an AMI that has a UI configured plus VNC or equivalent.  The reduces charged AWS I/O and can support more than one user collaborating.
  • Use a local Workstation.  This reduces AWS server charges but increases AWS I/O charges.  It can also cause problems if the Workstation loses connection during a test.  However, it is extremely convenient for developing tests.