Feature CommunityFeature Steward: TBD Member List: Dan Davis Support for Hierarchical StorageThere has been a long outstanding need for support of hierarchical storage in the Fedora Repository and related components. Recent events have underscored that this need has again moved to the forefront with increasing use of the Fedora Repository in research applications. Examples include UPEI's Virtual Research Environment (VRE) deployments using Islandora, Max Planck Institutes eSciDoc (Fiz Karlsruhe) and the emerging NSF Cyberinfrastructure program. Other products such as the DICE iRODS and SDSC SRB have provided a means to virtualize storage including support for hierarchical storage. There have also been requests from institutional repository and humanities researchers who have large collections for the addition of this feature. In this space, we will be exploring the requirements related to supporting hierarchical storage and enabling the community in adding support to Fedora Commons components. Characteristics of Hierarchical StorageAlso known as tiered storage, the driving rationale for using hierarchical storage is the notion that costs can be reduced by storing all or part of collection of files (bitstreams) on lower performing, less expensive storage technologies while keeping a copy of some part of the collection on a high performing, more expensive storage technology for immediate use. The hierarchical storage system implements some policy for determining on what tier files are stored to meet system goals. In real world implementations, many other needs may be considered since hierarchical storage can be used to help with other system requirements such as backup, replication, high availability and disaster recovery. However, it is cost that underlies any decision to deploy hierarchical storage. The Fedora architecture presents some unique problems and opportunities in supporting hierarchical storage. Even though storage cost is the driving reason for using hierarchical storage, it can provide a solution for other system requirements. We hope in this forum can list those requirements to inform the design for a hierarchical storage integration which fits well as part of the overall Fedora architecture. In addition to the system requirements listed above, the design may need to consider large datastreams, partial reads, updates/versioning, and other features already listed in the JIRA tracker. |