Table of Contents |
---|
Introduction
The DuraCloud application provides a set of services which can be deployed and used for a variety of purposes, primarily to process the content which has been loaded into DuraCloud storage. The following list of services describes how each service is expected to be used and the options available for tailoring the service to your needs.
Note that all services currently have a "Location" configuration option which is intended to allow for the deployment of services at varying locations. At the moment, however, services can only be deployed on the primary service instance. As this configuration option is consistent across all services it will not be included in the listing for each service.
Also note that the current way all DuraCloud services are configured, they will not auto-restart if they fail during processing. If you notice a failed job state, simply redeploy the service. Automatic service recovery is on the roadmap for DuraCloud development in the near future and will be made available as soon as possible.
Duplicate on Upload
Description:
The Duplicate on Upload service provides a way to ensure that the content added to DuraCloud is stored with at least two storage providers. The Duplicate on Upload service performs on-ingest duplication of content. This means that once the Duplicate on Upload service is deployed, it watches for all content that is added to your DuraCloud account, determines if it should be copied to another DuraCloud store, and if so, performs the copy. All content that is copied will be placed in an identically named space in the secondary storage location.
Configuration Options:
- Duplicate from this store: The primary storage location which DuraCloud will monitor for file additions. When files are added to this store, they will be copied to the secondary store.
- Duplication to this store: The secondary store where content will be copied after it has been added to the primary store.
Duplicate on Demand
Description:
The Duplicate On Demand service provides a simple way to duplicate content from one space to another. This service is primarily focused on allowing the duplication of content from the primary storage provider to a secondary provider. To begin, a source space is chosen, along with a store and space to which content will be duplicated. The service then performs a copy of all content and metadata in the source space to the duplication space, creating the space if necessary. When the service has completed its work, a results file will be stored in the chosen space and a set of files (primarily logs) created as part of the process will be stored in the work space.
Configuration Options:
- Source Space: DuraCloud space where source files can be found
- Replicate to this store: DuraCloud store to which content will be copied
- Replicate to this space: DuraCloud space where content will be copied
- Store results file in this space: DuraCloud space (on the primary store) where results file will be placed
- Standard vs. Advanced Bulk configuration
- Standard allows the user to choose "Optimize for cost" or "Optimize for speed"
- Advanced allows the user to configure the number and type of servers to use
- Number of Server Instances: The number of servers to use to perform the replication task.
- Type of Server: The type (size) of server used as perform the task. The larger the server, the faster the processing will occur. Larger servers also cost more than smaller servers to run. For more information, see the Amazon EC2 documentation.
Image Server
Description:
The Image Server provides a viewer for image files through use of the Djatoka image server. While this service is geared towards serving JPEG 2000 images, it supports multiple image file types by converting them to JPEG 2000 format on the fly.
Note that the current implementation of this service requires that spaces be set to OPEN in order to use the viewer to view image files.
Configuration Options:
None
Media Streamer
Description:
The Media Streamer provides streaming capabilities for video and audio files. The service Media Streaming capabilities provided by DuraCloud allow video and audio files to be streamed over RTMP. This feature in DuraCloud takes advantage of Amazon Cloudfront streaming, so files to be streamed must be within a space spaces on an Amazon provider. Also, all media to be streamed by this service needs to be within a single space. Amazon Cloudfront streaming uses the Flash Media Server to host streaming files over RTMP. File formats supported include MP3, MP4 and FLV among others. For a full listing of supported file types see the Flash Media Server documentation.
Configuration Options:
- Source Media Space: The DuraCloud space where the source video and audio files to be streamed are stored. The Media Streamer service attempts to stream all files in this space.
- Viewer Space: A DuraCloud space where example viewer files will be stored. After the service has started, this space will include a playlist including all items in the source media space as well as example html and javascript files which can be used to display a viewer.
Bit Integrity Checker
Description:
The Bit Integrity Checker provides the ability to verify that the content held within DuraCloud has maintained its bit integrity. There are five modes of operation.
Modes:
- Verify the bit integrity of a list of items
- Verify the bit integrity of an entire space
- Generate bit integrity information for a list of items
- Generate bit integrity information for an entire space
- Compare two different bit integrity reports
Configuration Options:
- Mode 1
- "Get integrity information from..." : defines the source of the MD5: pre-stored metadata, regenerated, regenerated with salt
- "Salt" : any character string that will be appended to the item stream during the MD5 regeneration
- "Space with input listing" : space holding the list of items over which to run the service
- "Input listing name" : item name of list of items over which to run the service
- "Output space" : destination space of service outputs
- "Output listing name" : destination item name of MD5s listing
- "Output report name" : destination item name of fixity report
- "Store" : underlying storage provider over which service will run
- Mode 2
- "Get integrity information from..." : defines the source of the MD5: pre-stored metadata, regenerated, regenerated with salt
- "Salt" : any character string that will be appended to the item stream during the MD5 regeneration
- "Space with input listing" : space holding the list of items over which to run the service
- "Space containing content items" : source space of items over which to run the service
- "Input listing name" : item name of list of items over which to run the service
- "Output space" : destination space of service outputs
- "Output listing name" : destination item name of MD5s listing
- "Output report name" : destination item name of fixity report
- "Store" : underlying storage provider over which service will run
- Mode 3
- "Get integrity information from..." : defines the source of the MD5: pre-stored metadata, regenerated, regenerated with salt
- "Salt" : any character string that will be appended to the item stream during the MD5 regeneration
- "Space with input listing" : space holding the list of items over which to run the service
- "Input listing name" : item name of list of items over which to run the service
- "Output space" : destination space of service outputs
- "Output listing name" : destination item name of MD5s listing
- "Store" : underlying storage provider over which service will run
- Mode 4
- "Get integrity information from..." : defines the source of the MD5: pre-stored metadata, regenerated, regenerated with salt
- "Salt" : any character string that will be appended to the item stream during the MD5 regeneration
- "Space containing content items" : source space of items over which to run the service
- "Output space" : destination space of service outputs
- "Output listing name" : destination item name of MD5s listing
- "Store" : underlying storage provider over which service will run
- Mode 5
- "Space with input listing" : space holding first list of MD5s
- "Space with second input listing" : space holding second list of MD5s
- "Input listing name" : item name of first list of MD5s
- "Second input listing name" : item name of second list of MD5s
- "Output space" : destination space of service outputs
- "Output report name" : destination item name of fixity report
- "Store" : underlying storage provider over which service will run
Bulk Bit Integrity Checker
Description:
The Bulk Bit Integrity Checker provides a simple way to determine checksums (MD5s) for all content items in any particular space by leveraging an Amazon Hadoop cluster. This service is designed for large datasets (+10GB).
Configuration Options:
- Source Space: DuraCloud space where source files are stored
- Destination Space: DuraCloud space where report file will be placed
- Standard vs. Advanced Bulk configuration
- Standard allows the user to choose "Optimize for cost" or "Optimize for speed"
- Advanced allows the user to configure the number and type of servers to use
- Number of Server Instances: The number of servers to use to perform the MD5 generation task.
- Type of Server: The type (size) of server used as perform the task. The larger the server, the faster the processing will occur. Larger servers also cost more than smaller servers to run. For more information, see the Amazon EC2 documentation.
Image Transformer
Description:
The Image Transformer provides a simple way to transform relatively small numbers of image files from one format to another.
Note that the ImageMagick service must be deployed prior to using the Image Transformer
Configuration Options:
- Source Space: DuraCloud space where source image files are stored
- Destination Space: DuraCloud space where transformed image files will be placed, along with a file which details the results of the conversion process
- Destination Format: The image format to which the source files will be transformed
- Destination Color Space: The colorspace of the transformed files, either "Source Image Color Space", meaning that the colorspace of the original image will be used, or sRGB, meaning that the colorspace will be transformed to sRGB.
- Source file name prefix: Only files beginning with the value provided here will be transformed. For example, if you enter ABC, only files whose names begin with the string ABC will be processed. This field is optional.
- Source file name suffix: Only files ending with the value provided here will be transformed. For example, you enter .jpg, only files whose names ends with the string .jpg will be processed. This field is optional.
Bulk Image Transformer
Description:
The Bulk Image Transformer provides a simple way to transform image files from one format to another in bulk. This service uses Amazon's Elastic Map Reduce capability to run the image transformation task within a Hadoop cluster.
Configuration Options:
- Source Space: DuraCloud space where source image files are stored
- Destination Space: DuraCloud space where transformed image files will be placed, along with a file which details the results of the transformation process
- Destination Format: The image format to which the source files will be transformed
- Destination Color Space: The colorspace of the transformed files, either "Source Image Color Space", meaning that the colorspace of the original image will be used, or sRGB, meaning that the colorspace will be transformed to sRGB.
- Source file name prefix: Only files beginning with the value provided here will be transformed. For example, if you enter ABC, only files whose names begin with the string ABC will be processed. This field is optional.
- Source file name suffix: Only files ending with the value provided here will be transformed. For example, you enter .jpg, only files whose names ends with the string .jpg will be processed. This field is optional.
- Standard vs. Advanced Bulk configuration
- Standard allows the user to choose "Optimize for cost" or "Optimize for speed"
- Advanced allows the user to configure the number and type of servers to use
- Number of Server Instances: The number of servers to use to perform the image transformation task.
- Type of Server: The type (size) of server used as perform the task. The larger the server, the faster the processing will occur. Larger servers also cost more than smaller servers to run. For more information, see the Amazon EC2 documentation.
Note that there have been issues discovered during testing of the Bulk Image Transformer. If you choose to run this service, it is recommended that the size of images being used be kept under 100MB. The likelihood of success appears to increase with server size, and number of servers being set to 3 or more is recommended. If you do run this service, please note the data set and configuration and make us aware of the outcome.
System Transformer Utility
Description:
The System Transformer Utility deploys the ImageMagick application on a DuraCloud service instance, which allows other services to take advantage of its features. The Image Transformer requires that this service be deployed in order to operate correctly.
Configuration Options:
None
System WebApp Utility
Description:
The System WebApp Utility coordinates the installation, de-installation, startup and shutdown of Apache Tomcat servers on a DuraCloud service instance. These Tomcat servers are created to allow other DuraCloud services to deploy web applications. The Image Server requires that this service be deployed in order to operate correctly.
Configuration Options:
Open and Secure Streaming
DuraCloud supports two different types of streaming, open and secure. Open streaming allows anyone with access to the URL for a streamed content item to stream the content. This works well for open access content which is intended to be shared and accessed widely. Secure streaming requires that a request be made to DuraCloud to retrieve a signed URL. That signed URL can then be used to stream the file. The request to retrieve a signed URL can specify how long the content will be available to stream (the default is 8 hours) as well as the IP address or IP address range where the streaming is allowed to take place. The purpose of secure streaming is to restrict the use of the stream. This is ideal for scenarios where streamed content is not free to use or must only be provided to a limited audience. Note that both types of streaming, open and secure, use the RTMP protocol, which protects the source file from being downloaded. The RTMP protocol requires that a flash-based streaming media player be used to play the streamed content.
Using Media Streaming
Follow these steps to stream media files with DuraCloud
- Create a space in DuraCloud which you will use to host streamed files
- Transfer media files to the space. Be sure that the file are using supported formats (see the link above).
- Enable streaming
- To enable open streaming: Select the space in the DuraCloud interface and click the "ON" button next to "Streaming:" in the top row of buttons.
- To enable secure streaming: Perform a POST HTTP call to the URL https://{institution}.duracloud.org/durastore/task/enable-streaming. The body of the POST request should include this JSON document: {"spaceId" : "","secure" : "true"}. Fill in the ID of the space to stream. Note that this call is using the DuraCloud REST API.
- Wait up to 15 minutes. If this is the first time the space has been streamed, it can take up to 15 minutes for the files to be available on the Amazon edge servers.
- Stream a file
- When using open streaming:
- Select a media file in the space. A video player will appear in the Content Detail pane. Verify that you are able to play the streamed file.
- Look in the space properties for the RTMP streaming address. This is the path you will use for streaming files. Alternatively, you can perform a get-url task call through the DuraCloud REST API to retrieve the streaming URL for each content item to be streamed. These URLs are predictable and do not expire.
- When using secure streaming:
- Spaces using secure streaming do not provide playback via the DuraCloud UI. You will need to perform a "get-signed-url" call to retrieve a signed URL for each content item to be streamed, and stream the file through an RTMP compatible player. More details about the get-signed-url call can be found in the Amazon S3 Storage Provider tasks section of the DuraCloud REST API.
- When using open streaming:
- Set up your website or application to provide access to the streamed files. Some example files to get you started are listed below.
Warning |
---|
The Flash Media Server used by Amazon Cloudfront and media players like JWPlayer and Flowplayer require certain specific conventions for requesting streamed files. There are two primary variables, one being a prefix which may need to preceed the file name (example prefix values are "mp3:" and "mp4:"). The other variable is whether a file extension is allowed on the file name. Getting these combinations right is particularly important when using secure streaming, as the player cannot request the file with alternative file names to match its preferences. Not all file types use the same combination of prefix and file extension settings. For example, it is common for MP4 files to require a prefix and extension (example file name: "mp4:videofile.mp4") while MP3 files require a prefix but no extension (example file name: "mp3:audiofile"). The prefix value, when needed, should be added to the stream path by using the "resourcePrefix" parameter on the get-url or get-signed-url call made through the DuraCloud REST API. In most cases, the file extension will need to be part of the stored file name. Even if files are named with a file extension (which is typically the case), calls to retrieve a streaming URL can specify the file name with no extension. |
Integration Files
The following files are available as a bundle here.
They are intended as a starting point for integrating streaming media into your own website.
- player.swf - The flash-based video player JWPlayer
- playlist.xml - An example playlist which would include a list of items in your Source Media Space
- playlistplayer.html - An HTML file which uses JWPlayer to display the items in the playlist
- singleplayer.html - An HTML file which uses JWPlayer to display a single media file
- stylish.swf - A supplementary flash file used to style the JWPlayer
- swfobject.js - A javascript file (available from here) used to embed the JWPlayer on a web page
- viewer.js - A javascript file used to simplify the loading of JWPlayer
All of the above files are intended as examples only. Their purpose is give developers a starting point for embedding video streamed by DuraCloud on their own web pages.
Info |
---|
If you add files to a space with streaming turned on, those files will automatically be made available for streaming as well. |
...