Since 2007, Penn State University has been researching ways of addressing the challenges associated with storage support for the tremendous growth of information and data within Penn State University Libraries and the university as a whole. Simple processes used in the past, such as copying data sets, finding a specific subset of data, or replacing storage arrays, are becoming extremely complex operations that are too large and time consuming to accomplish within acceptable levels of service. Software solutions built to manage large repositories of data also require application-specific knowledge, long-term migration, and integration challenges that often result in large and unwieldy silos of data.
To begin to address these challenges the Storage Networking Industry Association (SNIA) is endorsing the use of storage tiers and classes, data life-cycle strategies and new standards for managing structured data, such as the eXtensible Access Method (XAM). As DLT began researching the XAM standard, implementation hurdles became apparent; hurdles such as integration with existing applications, working across heterogeneous data silos, and hitherto lack of application of metadata standards in the storage domain. In order to assess XAM's potential, Penn State initiated an Archival Storage Prototype project in collaboration with the Penn State University Libraries, WPSU a unit of Outreach at Penn State, and Penn State ITS Emerging Technologies. The prototype entailed the development of a Storage Services Gateway that resides between applications and storage,that consists of a policy engine and a metadata engine, and serves as a mechanism to access our storage pool via standards-based protocols including AJAX, SOAP, REST, HTTP, and FTP.
Our project team included metadata experts who led the definition of technical and administrative metadata used to search federated content, create retention policies, and route objects to tiered storage. Additionally, we wanted to demonstrate how the use of metadata is instrumental in the lifecycle management of data saved into our storage cloud. Another key component of the Archival Storage Prototype project was to test and demonstrate XAM capabilities for enhanced storage management. The project team focused on our tier 2 (fixed content) storage service using XAM as the framework to access private cloud storage targets to save, manage, and retrieve fixed content objects and associated metadata. Fundamental to the Storage Services Gateway, was the development of a REST API that is capable of performing XAM translations when utilizing our storage cloud. REST is a well-established software architecture for many web developers and is common in software repositories, therefore enhancing adoption of the XAM protocol with little change to existing code or skill sets.
