Page History

...

• Created a plug-in for LLStore which stores all object and datastream content to S3
  • Amazon S3 Storage project in the incubator
• Created two private AMIs

1. ami-69ff1b00: A 32-bit image using Fedora Core 6 OS which includes a checkout of the current Fedora trunk and all of the tools necessary to build/install/run a Fedora server.
2. ami-73fa1e1a: Starting from image above, adds scripting to update and build the code, install the server based on an install.properties file included with instance startup, and start the server.

• Data is stored using instance storage, which is ideal for automated testing, but not for running a real repository.
• Performed testing with storing content on EBS, but no significant integration with Fedora.
• Conducted a simple performance measure by running the ConfigB system tests on Fedora servers which were identical except for the storage location, using instance storage, S3, or EBS as the location to write objects and datastreams. Results specify the time required to run the ConfigB test suite.
  • Instance Storage: 7 min 12 sec
  • S3 Storage: 17 min 59 sec
  • EBS Storage: 6 min 45 sec

Scalability Testing

• Created a simple test to perform a continuous ingest of objects into Fedora. The goal of this test is to determine how many objects can be ingested before the system fails, and determine what that failure point is.
• The test was run on a medium-size, high-CPU EC2 instance. The MySQL database was on the 10GB OS partition, Fedora ran on the 350GB /mnt partition, and the results file was written to a mounted EBS volume. The test ended when MySQL ran out of disk space. About 16 million objects were ingested during the test.

Where do we want to go with AWS?

...

• Automated Testing / Nightly Builds
  • Create an AMI which can be started with a particular configuration on which unit and system tests can be run using Bamboo or another test automation tool.
  • Current AMI is a good start toward this. Would need to add capabilities to determine database preference from install.properties and then start and configure that database. May also want to determine a way to choose a JVM version other than 1.5.

• Performance Testing
  • Parameterized EC2 instances provide a good candidate for simplifying the setup and cleanup necessary for running test instances. Tests can also be run in parallel by simply starting more instances.
  • Tests will likely need to be vetted against a standard server to determine if there are any unexpected effects caused by running on EC2.

• FedoraShare
  • Create an AMI which runs a Fedora instance with content/DB/triplestore data saved to an EBS volume. Include an application sitting on top of Fedora which allows people to use Fedoa in interesting ways.

• Mediation (DSpace/Fedora)
  • Mediate between multiple repository systems
  • New repository service which takes ideas from DSpace/Fedora
  • Simple services (example: MrSID image conversion)

• Akubra
  • Store to S3
  • Using EBS if running on EC2

• Simple startup of Fedora server
  • Basic public AMI with Fedora
  • Having pre-configured Fedora servers that can be made available quickly
  • Easy way to get up and running with Fedora
  • How to handle billing issues?
  • Adding value with durable service overlay

• Federation/scaling/high availability experimentation
  • Allow starting new EC2 instances to provide higher availability, better performance, etc.

• Information modeling support

• Local use
  • Website on EC2, data on S3, etc