About This Page
The purpose of this page is to inform the design and development of Akubra by providing an analysis of existing Blob Storage APIs and implementations.
Here we define Pluggable Blob Storage API as a programming interface with the following characteristics:
A Pluggable Blob Storage API MUST:
- Be primarily concerned with providing direct read/write access to named files.
- Be
The APIs listed on this page have the following attributes in common:
- They each provide a means to read and write persistent files with unique ids.
- They were designed with back-end pluggability in mind.
- They are not concerned with directory/folder semantics.
A Pluggable Blob Storage API MAY:
- Provide read/write access to metadata about stored files.
- Provide a way to list or query stored files.
- Provide transactional capabilities.
A Pluggable Blob Storage API MUST NOT:
- Be concerned with container (directory/folder/collection) semantics.
- Be concerned with versioning semantics (e.g. revision history)They are not concerned with versioning semantics, but may allow files to be overwritten.
APIs and Implementations
API | Language | Target Application | Known Implementations |
---|---|---|---|
Java | General-Purpose | AKU-FS: FSBlobStore | |
Java | Alfresco CMS | ALF-FS: FileContentStore | |
Java | DSpace 2 (Unreleased) | BIT-FED: FedoraBitStore | |
DJA-API: Django FileStorageStorage | Python General-Purpose | Any Using Django | |
Java | Daisy CMS | ||
COM | MS SQL Server | ||
Java | Apache JackRabbit | ||
Java | Apache JackRabbit | ||
Java | Fedora Repository (2.x to 3.1) | LLS-FS: GenericFileSystem | |
Java | Topaz OTM | ||
Java / C | General-Purpose | XAM-CNTCEN: EMC Centera |
Capability Matrix
API/Impl | IDType | ID Type | ID Gen | Write? IDGen | Rewrite? | Read-Only MD | Writable MD Custom MD | Listing Options | Transactions |
---|---|---|---|---|---|---|---|---|---|
AKU-API | URI | User Internal or Store External | Yes | Yes | size Size | ? | All | Optional, JTA | |
AKU-FS | file:/// URIs URI | Internal | Store Yes | Yes | size | No | No | No |
Capabilities of Interest
- Ids
- Supported ID types?
- Can be store-generated?
- Can be user-provided?
- Content Rewritable?
- Readonly Metadata
- Which are impls expected to provide for reading?
- Settable Metadata
- Which are impls expected to provide for writing?
- Extensible?
- Transactional
- Supported by API?
- Required by impls?
- Two-phase commit?
Other APIs/Implementations of Interest
Size | None | All | No | |||||
API/Impl | ID Type | ID Gen | Write? | Rewrite? | Read-Only MD | Writable MD | Listing Options | Transactions |
---|---|---|---|---|---|---|---|---|
ALF-API | URL | Internal or External | Yes | Yes | Size | None | All | Optional, JTA |
ALF-FS | store:// URL | Internal or External | Yes | Yes | Size | None | All | Optional, JTA |
ALF-HTT | http:// URL | N/A | No | No | Size | None | None | Optional, JTA |
ALF-REP | URL | Internal or External | Yes | Yes | Size | None | All | Optional, JTA |
ALF-ROU | URL | Internal or External | Yes | Yes | Size | None | All | Optional, JTA |
API/Impl | ID Type | ID Gen | Write? | Rewrite? | Read-Only MD | Writable MD | Listing Options | Transactions |
BIT-API | Unicode String | Internal | Yes | Yes | Any (Map) | None | None | No |
BIT-FED | (NCNameChar)* | Internal | Yes | No | Size | None | None | No |
BIT-FS | ([DEV:0-9]){38,40} | Internal | Yes | Yes | Size | None | None | No |
BIT-HON | ([DEV:0-9]){38,40} | Internal | Yes | Yes | ? | None | None | No |
BIT-S3 | ([DEV:0-9]){38,40} | Internal | Yes | Yes | Size | None | None | No |
BIT-SRB | ([DEV:0-9]){38,40} | Internal | Yes | Yes | Size | None | None | No |
API/Impl | ID Type | ID Gen | Write? | Rewrite? | Read-Only MD | Writable MD | Listing Options | Transactions |
DJA-API | Unicode String | External | Yes | Yes | Size | None | None | No |
DJA-FS | Unicode String | External | Yes | Yes | Size | None | None | No |
DJA-S3 | Unicode String | External | Yes | Yes | Size | None | None | No |
API/Impl | ID Type | ID Gen | Write? | Rewrite? | Read-Only MD | Writable MD | Listing Options | Transactions |
DSY-API | Unicode String | Internal | Yes | No | None | None | None | No |
DSY-FS | Unicode String | Internal | Yes | No | None | None | None | No |
API/Impl | ID Type | ID Gen | Write? | Rewrite? | Read-Only MD | Writable MD | Listing Options | Transactions |
EBS-API | byte[] | Internal | Yes | No | Size | None | None | No |
EBS-FS | GUID or String | Internal | Yes | No | Size | None | None | No |
API/Impl | ID Type | ID Gen | Write? | Rewrite? | Read-Only MD | Writable MD | Listing Options | Transactions |
JKB-API | Unicode String | Internal | Yes | Yes | None | None | None | No |
JKB-DB | propertyId + index | Internal | Yes | Yes | None | None | None | No |
JKB-FS | Filesystem Path | Internal | Yes | Yes | None | None | None | No |
API/Impl | ID Type | ID Gen | Write? | Rewrite? | Read-Only MD | Writable MD | Listing Options | Transactions |
JKD-API | Hex String | Internal | Yes | No | Size | None | All | No |
JKD-DB | SHA-1 Digest of Content | Internal | Yes | No | Size | None | All | No |
JKD-FS | SHA-1 Digest of Content | Internal | Yes | No | Size | None | All | No |
API/Impl | ID Type | ID Gen | Write? | Rewrite? | Read-Only MD | Writable MD | Listing Options | Transactions |
LLS-API | pid or pid + dsId | External | Yes | Yes | None | None | None | No |
LLS-FS | pid or pid + dsId | External | Yes | Yes | None | None | None | No |
LLS-HON | pid or pid + dsId | External | Yes | Yes | None | None | None | No |
LLS-IRD | pid or pid + dsId | External | Yes | Yes | None | None | None | No |
LLS-S3 | pid or pid + dsId | External | Yes | Yes | None | None | None | No |
LLS-SRB1 | pid or pid + dsId | External | Yes | Yes | None | None | None | No |
LLS-SRB2 | pid or pid + dsId | External | Yes | Yes | None | None | None | No |
API/Impl | ID Type | ID Gen | Write? | Rewrite? | Read-Only MD | Writable MD | Listing Options | Transactions |
OTM-API | Unicode String | External | Yes | No | None | None | None | Optional, JTA |
OTM-FED | Unicode String | External | Yes | No | None | None | None | Best-effort, JTA |
OTM-FS | Unicode String | External | Yes | No | None | None | None | Yes, JTA |
API/Impl | ID Type | ID Gen | Write? | Rewrite? | Read-Only MD | Writable MD | Listing Options | Transactions |
XAM-API | XUID | Internal | Yes | No 1 | Last-Modified | Several Predefined | All | No |
XAM-CEN | XUID | Internal | Yes | No | Last-Modified | Several Predefined | All | No |
XAM-HON | XUID | Internal | Yes | No | Last-Modified | Several Predefined | All | No |
XAM-IAP | XUID | Internal | Yes | No | Last-Modified | Several Predefined | All | No |
1 XAM is designed with a fixed-content assumption - data cannot be modified without changing its XUID.
2 In XAM, all metadata fields are either binding or non-binding. Binding metadata cannot change while keeping the same XUID for an XSet, while non-binding metadata can.
Other APIs/Protocols/Impls of Interest
- Apache Sling - REST interface on JCR, need to look at this more...
- AtomPub - Collection semantics
- Azure Blob Storage API - Collection semantics, REST-based
- CMIS - Draft; has versioning and container semantics.
- DSpace 1 BitstreamStorageManager - Single impl, supports either local filesystem or SRB
- Eclipse - BlobStore- Single impl, maps UUIDs to files on disk
- Google Datastore API - Google's "cloud" underneath, REST-ish, container semantics, transactions!...but quite limited today (1MB limit on content, 1000 limit on local files...etc)
- JAF DataSource - Not a Store API; just models a file with a MIME type
- JSR170/283 - JCR; has versioning and container semantics.
- Jakarta Slide ContentStore - Defunct; has versioning semantics
- Jargon API for SRB and iRODS - Models files and directories, with metadata
- JSPWiki FileSystemProvider - Fedora Repository Impl available; API has versioning semantics.
- MediaWiki FileStore - PHP source
- MogileFS - Obtaining
- GlusterFS- a clustered file-system capable of scaling to several peta-bytes
- NFS - Directory semantics.
- OKI Repository OSID - Has container semantics (Assets may contain other assets)
- Plone Large File Management - Several options, but no common interfacealternatives to ZODB. Appear to be explicitly configured as part of a "content type". For example, see FileSystemStorage
- Sakai Content Hosting Service - Collection semantics. JCR impl
- Skwish - Transactions and reads/writes over a collection of "segments" on a filesystem. See Motivation
- SMB/CIFS - Directory semantics.
- Storage Resource Manager - Used by LHC; does not provide direct data access, only references.MediaWiki FileStore - PHP source