Developers' Blog

Fedora's SOAP API has historically been split into two: API-A and API-M. The REST API, being resource-oriented, naturally isn't. And the control over which endpoints can be set to require authentication is inconsistent across the two. The old "Authenticate for API-A? Authenticate for API-M?" options no longer make sense when you look at Fedora's APIs as a whole.

Here's one way the situation could be improved.

1. For the SOAP API (all read-oriented and write-oriented methods), always require authentication.
2. For the REST API, on a per-verb basis (POST/PUT/DELETE/GET), offer the following options at install time:
   1. Proactive Challenge: Always require authentication.
   2. Reactive Challenge: Only require authentication if an un-authenticated request failed due to AuthZ rules.

NOTE: This is being tracked as FCREPO-668

Paul was describing how Mulgara Resolverswork today and Aaron pointed out that it may be possible to plug MPTStorein as a resolver.  This is interesting because it would allow one to use Mulgara's higher-level query capabilities (e.g. SPARQL) on top of MPTStore.  Additionally, for Fedora this would allow us to continue offering triplestore engine choice while working directly against the Mulgara interfaces vs going through Trippi.

Note however, that when using XA1.x (Mulgara's current storage engine), getting good throughput for lots of small writes would still require some sort of buffering layer.  Long-term, we believe XA2 will remove the requirement to do buffering "above" Mulgara for good write throughput.  XA2 is designed to make newly-committed changesets immediately visible, then merging them in the background.

Key Mulgara Interfaces

• org.mulgara.resolver.spi.Resolver
• Transactions:
  • org.mulgara.resolver.spi.EnlistableResource
• Query Results:
  • org.mulgara.resolver.spi.Resolution
  • org.mulgara.store.tuples.Tuples
• Queries:
  • org.mulgara.query.Constraint
  • org.mulgara.query.ConstraintExpression

Key MPTStore Interfaces

• org.nsdl.mptstore.core.DatabaseAdaptor
• Query Results:
  • org.nsdl.mptstore.core.QueryResults

Questions

• If the MPTStore resolver provides an XAResource by implementing the EnlistableResource interface, does this make Mulgara transactions automatically work with the resolver?
• What are the implications of this issue?

I recently posted some screenshots of the new web-based Fedora administrative GUI. Take a look here: Web Admin Screenshots

I've recently made a few significant changes to the FCREPO Tracker and thought it would be good to share them here. Some of this info will eventually work it's way into the Developer's Wiki, but for now it's a blog post.

Improvements

• Wider Access: The FCREPO Tracker now allows issues to be submitted from anyone in the community.  Formerly, we experimented with having a dedicated tracker for community-submitted issues, but decided it'd be best to track everything in one place and improve the issue workflow instead (see below).
• Wider Scope: Fedora, GSearch, OAIProvider, and DirIngest are now listed as "Components" within the FCREPO tracker.  These are all releasable products within the Fedora Repository Project.  As part of this change, I made the JIRA "Versions" for the project product-specific (e.g. "GSearch 2.2").  This makes sense because each product has it's own release schedule.
• Improved Workflow: Issues now start in the new Received state, rather than the Open state.  The distinction is that Open issues have been validated and determined by the committers to be in-scope.  More specifics below.

Issue Lifecycle

Received

All new issues begin in this state.  If it's clearly in scope for the project, it will be moved to "Open" by one of the committers.  If it's not in scope or is a duplicate, etc, it will closed.  If an issue remains in the recieved state for a longer than a couple weeks, it means we think it deserves more discussion before making a determination.

Open

Issues in this state are in scope for a future release.  The target release may not have been determined yet.  Generally higher-priority items will worked on first.  The committers will take our best guess on the initial priority of issues, but the priority may be increased or decreased based on input we receive (votes, comments) from the community.

Issues that are not yet assigned can be worked on by anyone in the community.  Attaching a good quality patch to an open issue is the best way to vote

In Progress

This state means a developer is currently working on the issue.  Small issues will usually bypass the review step and be Closed (and resolved as "Fixed") once the changes are committed to trunk and pass automated tests.  Larger issues will be moved to the "In Review" state at the discretion of the developer.

In Review

The assignee has asked someone to take a look at the solution before closing the issue.

Reopened

The issue was thought to be resolved, but isn't.

Closed

The issue has been resolved.  Possible resolutions are:

• Fixed
• Won't Fix (out of scope)
• Incomplete (not completely described)
• Cannot Reproduce (for Bugs)

I just posted a mockup of how I'm thinking the new Administrator GUI may look/work. Take a look, and add a comment letting me know what you think.

New Fedora Administrator UI Mockup

At the on-site last week, we had some additional discussion of the "Service Ladder" idea.  Notes and images from the board can be found at the On-Site Nov 3-7 Meeting Notes page.

Fedora 3.1 has been released!
Fedora 2.2.4 has been released!
WooHoo!

In other news, I'm working up a plan for an application called FedoraShare. This will be a front-end application for Fedora which integrates with content that is stored on other sites around the web and allows users to tag, comment on, and create relationships between any content, whether it be stored in Fedora or not. For more info, take a look at my write up and screenshot mockups.

Other than various activities surrounding the 3.1 release coming up next week, my main work of interest this week was doing a review of Open Laszlo and writing up my thoughts on Flex and Open Laszlo so far. The end goal of this investigation is to get started on a new web-based Administrator GUI for Fedora.

Trippi has been updated to include the latest Mulgara release (2.0.6). Notably, this latest release of Mulgara includes updates to the storage layer (XA 1.1) which improves performance while using less space and fewer files. See the Mulgara release notes for more details.

I've been a bit remiss about actually creating file releases for Trippi on Sourceforge when new versions have been tagged, but this version has had a proper release.

Fedora trunk and the maintenance branch have also been updated, so 3.1 and 2.2.4 (due later next week) will both include Trippi 1.4.1 (and thereby Mulgara 2.0.6).

The first half of this week I spent with Chris at MIT sitting in on the DSpace 2.0 architecture design meeting. In this session, they were focused primarily on determining the high-level services necessary to support the next version of DSpace.

• One discussion which we returned to several times was the idea that there is fundamental set of services needed by any repository which could be constructed in a layered structure. Chris posted an initial write-up of these ideas.

• Another discussion that Chris and I had on the side with Richard Rodgers was that the notification capabilities of both Fedora and DSpace present a way to coordinate the content available in each system. An example of this could be to allow both Fedora and DSpace to store content in the same location (such as on Amazon S3), then use update notifications to create/update/delete the object model. This gets around the need to have a shared data model in order to share content.

The last half of the week was spent finishing up tasks prior to the 3.1 release.

• Completed and merged FCREPO-245, which allows GSearch (and any other user of the MessagingClient) to start up prior to the messaging provider being available. This was primarily an issue when Fedora and GSearch were running in the same server and GSearch was started first.

• Completed and started review of FCREPO-241, which provides authentication filtering for the REST API relative to the authentication settings of API-A and API-M.

I've released version 0.1 of the Fedora unAPI HTTP Service to Sourceforge.

No major changes since the original prototype, except for the addition of a FedoraPmhResolver. In contrast to the FedoraResolver, the FedoraPmhResolver relies on Fedora's OAI service to describe the available formats for Fedora objects. Depending on the desired use, one may be more suitable than another. The FedoraResolver is more flexible and can support arbitrary formats, but it requires Fedora 3.x. The FedoraPmhResolver should work with any version of Fedora that supports OAI-PMH (although I've only tested it with 3.0).

Similarly, I expect the DSpacePmhResolver should work with any version of DSpace that support OAI-PMH, but I've only tested it against 1.5.1.

Bill and I are leaving Boston today after a good set of meetings with the DSpace folks.  I'll be hitting the road shortly, but I wanted to get some thoughts out on what we're calling "The Ladder" for lack of a better term.

The concept of the ladder is this: Assuming we have Akubra, what are the next "rungs" of agreement in a shared persistence model?  We played with the idea of different levels, starting with the purely structural, and moving carefully up into semantics.

At Level 0, you have Blobs with Ids. This is Akubra.
At Level 1, Relationships
At Level 2, Aggregation
At Level 3, Metadata/Semantics

I should point out that this ordering was a result of brainstorming, and we did not attempt to flesh out the details for each level.  But it gave us a sort of strawman to frame the discussion.

Here's my first attempt at refinement after sleeping on it.  Two key goals:

1. It is possible to persist everything in Level N + 1 into N.  If this is done, rebuilding everything from level 0 is possible.
2. Though RDF might not be presumed, it is possible to represent each level in RDF (minus blob content)

Level 0

Blobs with Ids and readonly properties (size, minimal others).

Level 1

Entities with readonly + writable relationships and properties.

Entities optionally have content (represent bitstreams aka Blobs).

Level 2

Entities with everthing in level 1, plus key inferred (read-only) relationships.  A big requirement here is to support two-way links for whole-part relations.  Might also include transitive relationships.

It's possible to infer such relationships from what's expressed in level 1 and a set of inference rules.

Level 3

Higher-order repository domain objects.  The concept of a metadata entity relating to content entity would be represented here.

• We decided during our status meeting that we'd shoot for Oct 15th-17th timeframe for the 2.2.4/3.1 releases; Bill and I will be in Cambridge meeting with DSpace folks much of this week.
• I gave an update on Akubra Analysis of Existing Approaches at the architecture meeting.
• I moved the Akubra project infrastructure (wiki, svn, tracking, mailing lists) to Fedora Commons. This is the first project that will use our locally-managed Subversion installation. It's also the first to use Google Groups for the mailing lists (rather than Sourceforge.net).
• Unfortunately I didn't get a chance to review outstanding branches by Bill and Eddie...hopefully I'll be able to do that on and off while in Cambridge this week.

Just wanted to share this paper, in which the term "spanning layer" was coined.

Interoperation, Open Interfaces, and Protocol Architecture - David D. Clark

Interesting stuff.