About This Page
The purpose of this page is to inform the design and development of Akubra by providing an analysis of existing Blob Storage APIs and implementations.
Here we define Pluggable Blob Storage API as a programming interface with the following characteristics:
A Pluggable Blob Storage API MUST:
- Be primarily concerned with providing direct read/write access to named files.
- Be
The APIs listed on this page have the following attributes in common:
- They each provide a means to read and write persistent files with unique ids.
- They were designed with back-end pluggability in mind.
- They are not concerned with directory/folder semantics.
A Pluggable Blob Storage API MAY:
- Provide read/write access to metadata about stored files.
- Provide a way to list or query stored files.
- Provide transactional capabilities.
A Pluggable Blob Storage API MUST NOT:
- Be concerned with container (directory/folder/collection) semantics.
- Be concerned with versioning semantics (e.g. revision history)They are not concerned with versioning semantics, but may allow files to be overwritten.
APIs and Implementations
API | Language | Target Application | Known Implementations |
|---|---|---|---|
Java | General-Purpose | AKU-FS: FSBlobStore | |
Java | Alfresco CMS | ALF-FS: FileContentStore | |
Java | DSpace 2 (Unreleased) | BIT-FED: FedoraBitStore | |
Python | Any Using Django | ||
Java | Daisy CMS | ||
COM | MS SQL Server | ||
Java | Apache JackRabbit | ||
Java | Apache JackRabbit | ||
Java | Fedora Repository (2.x to 3.1) | LLS-FS: GenericFileSystem | |
Java | Topaz OTM | ||
Java / C | General-Purpose |
Capability Matrix
API/Impl | ID Type | ID Gen | Write? | Rewrite? | Read-Only MD | Writable MD | Listing Options | Transactions | |||
|---|---|---|---|---|---|---|---|---|---|---|---|
AKU-API | URI | User Internal or Store External | Yes | Yes | Size | ? | All | Optional, JTA | |||
AKU-FS | file:/// URI | Store Internal | Yes | Yes | Size | None | All | No | |||
API/Impl | ID Type | ID Gen | Write? | Rewrite? | Read-Only MD | Writable MD | Listing Options | Transactions | |||
ALF-API | URL | User Internal or Store External | Yes | Yes | Size | None | All | Optional, JTA | |||
ALF-FS | store:// URL | User Internal or Store External | Yes | Yes | Size | None | All | Optional, JTA | |||
ALF-HTT | http:// URL | N/A | No | No | Size | None | None | Optional, JTA | |||
ALF-REP | URL | User Internal or Store External | Yes | Yes | Size | None | All | Optional, JTA | |||
ALF-ROU | URL | User Internal or Store External | Yes | Yes | Size | None | All | Optional, JTA | |||
API/Impl | ID Type | ID Gen | Write? | Rewrite? | Read-Only MD | Writable MD | Listing Options | Transactions | |||
BIT-API | Unicode String | User or Store Internal | Yes | Yes | Any (Map) | None | None | No | |||
BIT-FED | (NCNameChar)* | User or Store Internal | Yes | No | Size | None | None | No | |||
<ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="e97086d2-18dd-46e6-99ea-5ef29a603c0a"><ac:plain-text-body><![CDATA[ | BIT-FS | ([DEV:0-9]){38,40} | User or Store Internal | Yes | Yes | Size | None | None | No | ||
BIT-HON | ? | ? | ? | ([DEV:0-9]){38,40} | Internal | Yes | Yes ? | ? | None | None | No <ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="5e02c294-b2d4-4d0c-9dfc-525190c21c65"><ac:plain-text-body><![CDATA[ |
BIT-S3 | ([DEV:0-9]){38,40} | User or Store Internal | Yes | Yes | Size | None | None | No | <ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="cb685afa-eb0c-408d-8cbd-e866a1591f4d"><ac:plain-text-body><![CDATA[ | ||
BIT-SRB | ([DEV:0-9]){38,40} | User or Store Internal | Yes | Yes | Size | None | None | No | |||
API/Impl | ID Type | ID Gen | Write? | Rewrite? | Read-Only MD | Writable MD | Listing Options | Transactions | |||
DJA-API | Unicode String | User External | Yes | Yes | Size | None | None | No | |||
DJA-FS | Unicode String | User External | Yes | Yes | Size | None | None | No | |||
DJA-S3 | Unicode String | User External | Yes | Yes | Size | None | None | No | |||
API/Impl | ID Type | ID Gen | Write? | Rewrite? | Read-Only MD | Writable MD | Listing Options | Transactions | |||
DSY-API | Unicode String | Store Internal | Yes | No | None | None | None | No | |||
DSY-FS | Unicode String | Store Internal | Yes | No | None | None | None | No | |||
API/Impl | ID Type | ID Gen | Write? | Rewrite? | Read-Only MD | Writable MD | Listing Options | Transactions | <ac:structured-macro ac:name="unmigrated-wiki-markup" ac:schema-version="1" ac:macro-id="dfb8c09c-0b8e-4ab9-af41-86c2743ae1a8"><ac:plain-text-body><![CDATA[|||
EBS-API | byte[] | Store Internal | Yes | No | Size | None | None | No | |||
EBS-FS | GUID or String | Store Internal | Yes | No | Size | None | None | No | |||
API/Impl | ID Type | ID Gen | Write? | Rewrite? | Read-Only MD | Writable MD | Listing Options | Transactions | |||
JKB-API | Unicode String | Store Internal | Yes | Yes | None | None | None | No | |||
JKB-DB | propertyId + index | Store Internal | Yes | Yes | None | None | None | No | |||
JKB-FS | Filesystem Path | Store Internal | Yes | Yes | None | None | None | No | |||
API/Impl | ID Type | ID Gen | Write? | Rewrite? | Read-Only MD | Writable MD | Listing Options | Transactions | |||
JKD-API | Hex String | Store Internal | Yes | No | Size | None | All | No | |||
JKD-DB | SHA-1 Digest of Content | Store Internal | Yes | No | Size | None | All | No | |||
JKD-FS | SHA-1 Digest of Content | Store Internal | Yes | No | Size | None | All | No | |||
API/Impl | ID Type | ID Gen | Write? | Rewrite? | Read-Only MD | Writable MD | Listing Options | Transactions | |||
LLS-API | pid or pid + dsId | User External | Yes | Yes | None | None | None | No | |||
LLS-FS | pid or pid + dsId | User External | Yes | Yes | None | None | None | No | |||
LLS-HON | pid or pid + dsId | User External | Yes | Yes | None | None | None | No | |||
LLS-IRD | pid or pid + dsId | User External | Yes | Yes | None | None | None | No | |||
LLS-S3 | pid or pid + dsId | User External | Yes | Yes | None | None | None | No | |||
LLS-SRB1 | pid or pid + dsId | User External | Yes | Yes | None | None | None | No | |||
LLS-SRB2 | pid or pid + dsId | User External | Yes | Yes | None | None | None | No | |||
API/Impl | ID Type | ID Gen | Write? | Rewrite? | Read-Only MD | Writable MD | Listing Options | Transactions | |||
OTM-API | Unicode String | User External | Yes | No | None | None | None | Optional, JTA | |||
OTM-FED | Unicode String | User External | Yes | No | None | None | None | Best-effort, JTA | |||
OTM-FS | Unicode String | User External | Yes | No | None | None | None | Yes, JTA | |||
API/Impl | ID Type | ID Gen | Write? | Rewrite? | Read-Only MD | Writable MD | Listing Options | Transactions | |||
XAM-API | XUID | Store Internal | Yes | No * 1 | Last-Modified | Several Predefined | All | No | |||
XAM-CEN | XUID | Store Internal | Yes | No * | Last-Modified | Several Predefined | All | No | |||
XAM-HON | XUID | Store Internal | Yes | No * | Last-Modified | Several Predefined | All | No | |||
XAM-IAP | XUID | Store Internal | Yes | No * | Last-Modified | Several Predefined | All | No |
* 1 XAM is designed with a fixed-content assumption – - data cannot be modified without changing its XUID. All
2 In XAM, all metadata fields are either binding or non-binding. Binding metadata cannot change while keeping the same XUID for an XSet, while non-binding metadata can.
Other APIs/
...
Protocols/Impls of Interest
- Apache Sling - REST interface on JCR, need to look at this more...
- AtomPub - Collection semantics
- Azure Blob Storage API - Collection semantics, REST-based
- CMIS - Draft; has versioning and container semantics.
- DSpace 1 BitstreamStorageManager - Single impl, supports either local filesystem or SRB
- Eclipse - BlobStore- Single impl, maps UUIDs to files on disk
- Google Datastore API - Google's "cloud" underneath, REST-ish, container semantics, transactions!...but quite limited today (1MB limit on content, 1000 limit on local files...etc)
- JAF DataSource - Not a Store API; just models a file with a MIME type
- JSR170/283 - JCR; has versioning and container semantics.
- Jakarta Slide ContentStore - Defunct; has versioning semantics
- Jargon API for SRB and iRODS - Models files and directories, with metadata
- JSPWiki FileSystemProvider - Fedora Repository Impl available; API has versioning semantics.
- MediaWiki FileStore - PHP source
- MogileFS - Obtaining
- GlusterFS- a clustered file-system capable of scaling to several peta-bytes
- NFS - Directory semantics.
- OKI Repository OSID - Has container semantics (Assets may contain other assets)
- Plone Large File Management - Several options, but no common interfacealternatives to ZODB. Appear to be explicitly configured as part of a "content type". For example, see FileSystemStorage
- Sakai Content Hosting Service - Collection semantics. JCR impl
- Skwish - Transactions and reads/writes over a collection of "segments" on a filesystem. See Motivation
- SMB/CIFS - Directory semantics.
- Storage Resource Manager - Used by LHC; does not provide direct data access, only references.MediaWiki FileStore - PHP source