Page History

Table of Contents

The Fedora Repository makes it possible to design custom event-driven application workflows. For instance, a common use case is sending content to an external search engine or triplestore. Other repositories may wish to generate derivative content such as creating a set of smaller images from a high-resolution master.

...

The properties field will list the RDF properties that changed with that event. NODE_REMOVED events contain no properties. The fcrepo component for Camel is configured to recognize these headers and act appropriately.

Examples

LDPath Transformations

If an fcr:transform program has been installed as mytransform, you can generate a JSON representation of a container and send it to a low-latency, highly available document store, such as Riak. The following route determines if a container has been removed or simply added/updated. It then routes the message appropriately to a load-balancer sitting in front of the Riak HTTP endpoint.

Code Block

language	scala
title	Camel route to populate a Riak store, using the Scala DSL
linenumbers	true

val riakKeyProcessor = (exchange: Exchange) => {
	exchange.getIn.setHeader(
		Exchange.HTTP_PATH,
		"/buckets/fcrepo/keys/" + URLEncoder.encode(exchange.getIn.getHeader("org.fcrepo.jms.identifier", classOf[String]))
	)
}
 
"activemq:topic:fedora" ==> {
	choice() {
		when(_.in("org.fcrepo.jms.eventType") == "http://fedora.info/definitions/v4/repository#NODE_REMOVED") {
			setHeader(Exchange.HTTP_METHOD, constant("DELETE"))
			process(riakKeyProcessor)
			to("http4:localhost:8098")
		}
		otherwise() {
		    to("fcrepo:localhost:8080/fedora/rest")
			filter(xpathFilter) {
                to("fcrepo:localhost:8080/fedora/rest?transform=mytransform")
	      	    setHeader(Exchange.HTTP_METHOD, constant("PUT"))
		        process(riakKeyProcessor)
                to("http4:localhost:8098")
			}
		}
	}
}

External Triplestore

Some additional processing must be done to transform an application/n-triples response into a valid application/sparql-update payload before sending to an external triplestore such as Fuseki or Sesame. The fcrepo component contains some processors in org.fcrepo.camel.processor to handle this case. The examples below assume that messages have already been routed based on eventType (see below) and passed to the appropriate queue.

Code Block

language	java
title	Populate an external triplestore
linenumbers	true

from("direct:delete")
  .process(new SparqlDescribeProcessor())
  .to("http4:localhost:3030/db/query")
  .process(new SparqlDeleteProcessor())
  .to("http4:localhost:3030/db/update");
 
from("direct:new")
 .to("fcrepo:localhost:8080/rest")
 .process(new SparqlInsertProcessor())
 .to("http4:localhost:3030/db/update");
 
from("direct:update")
  .to("fcrepo:localhost:8080/rest")
  .process(new SparqlUpdateProcessor())
  .to("http4:localhost:3030/db/update");

When using these Sparql* processor classes, it is also possible to apply these operations to named graphs. While Fedora does not support named graphs, it is possible to assign nodes to certain named graphs in an external triplestore. The CamelFcrepoNamedGraph header can be used to apply a sparql-update operation to a particular named graph. For instance, to route all operations to a named graph (in this case the graph URI is defined dynamically as a property placeholder:

Code Block

language	java
title	Named Graph
linenumbers	true

from("direct:update")
  .to("fcrepo:localhost:8080/rest")
  .setHeader(FcrepoHeaders.FCREPO_NAMED_GRAPH)
    .simple("{{named.graph}}")
  .process(new SparqlUpdateProcessor())
  .to("http4:localhost:3030/ds/update");

Or, to partition the graph based on an existing RDF property:

Code Block

language	java
title	Multiple Named Graphs
linenumbers	true

from("direct:update")
  .to("fcrepo:localhost:8080/rest")
  .setHeader(FcrepoHeaders.FCREPO_NAMED_GRAPH)
    .xpath("/rdf:RDF/rdf:Description/ex:namedGraph/text()", String.class, ns)
  .process(new SparqlUpdateProcessor())
  .to("http4:localhost:3030/ds/update");

Or, you may want (possibly overlapping) "public" and "private" named graphs, defined as constants:

Code Block

language	java
title	Constant Named Graphs
linenumbers	true

from("direct:update")
  .to("fcrepo:localhost:8080/rest")
  .multicast("direct:public", "direct:private");

from("direct:public")
  .filter(publicPredicate)
    .setHeader(FcrepoHeaders.FCREPO_NAMED_GRAPH)
      .constant("http://site/graph/public")
    .process(new SparqlUpdateProcessor())
    .to("http4:localhost:3030/ds/update");

from("direct:private")
  .filter(privatePredicate)
    .setHeader(FcrepoHeaders.FCREPO_NAMED_GRAPH)
      .constant("http://site/graph/private")
    .process(new SparqlUpdateProcessor())
    .to("http4:localhost:3030/ds/update");

In each case the Sparql update operation will include a GRAPH <uri> { ... } clause, where uri is the value of the FCREPO_NAMED_GRAPH header (in the Spring DSL, this header can be accessed as CamelFcrepoNamedGraph). It is important that the value of the named graph header is a properly formed URI.

Event-based Routing

It is often helpful to route messages to different queues based on the eventType value. This example splits messages on eventType values and routes the messages to appropriate queues. Following this example, it would be prudent to aggregate the messages based on org.fcrepo.jms.identifier value after retrieving the messages from the downstream queues.

Code Block

language	xml
title	Content-based Routing
linenumbers	true

<route id="fcrepo-event-splitter">
    <description>
        Retrieve messages from the fedora topic. Event types are comma-delimited, so split them into separate messages before routing them.
    </description>
    <from uri="activemq:topic:fedora"/>
    <setBody>
        <simple>${header.org.fcrepo.jms.eventType}</simple>
    </setBody>
    <split>
        <tokenize token=","/>
        <setHeader headerName="org.fcrepo.jms.eventType">
            <simple>${body}</simple>
        </setHeader>
        <setBody>
            <simple>null</simple>
        </setBody>
        <to uri="seda:fcrepo-event-router"/>
    </split>
</route>
 
<route id="fcrepo-event-router">
    <description>
        Route messages based on the eventType.
    </description>
    <from uri="seda:fcrepo-event-router"/>
    <choice>
        <when>
            <simple>${header.org.fcrepo.jms.eventType} == "http://fedora.info/definitions/v4/repository#NODE_REMOVED"</simple>
            <to uri="activemq:queue:fcrepo.delete"/>
        </when>
        <when>
            <simple>${header.org.fcrepo.jms.eventType} == "http://fedora.info/definitions/v4/repository#NODE_ADDED"</simple>
            <to uri="activemq:queue:fcrepo.add"/>
        </when>
        <when>
            <simple>${header.org.fcrepo.jms.eventType} == "http://fedora.info/definitions/v4/repository#PROPERTY_ADDED"</simple>
            <to uri="activemq:queue:fcrepo.update"/>
        </when>
        <when>
            <simple>${header.org.fcrepo.jms.eventType} == "http://fedora.info/definitions/v4/repository#PROPERTY_CHANGED"</simple>
            <to uri="activemq:queue:fcrepo.update"/>
        </when>
        <when>
            <simple>${header.org.fcrepo.jms.eventType} == "http://fedora.info/definitions/v4/repository#PROPERTY_REMOVED"</simple>
            <to uri="activemq:queue:fcrepo.update"/>
        </when>
        <otherwise>
            <log message="No router for ${header.org.fcrepo.jms.eventType}"/>
        </otherwise>
    </choice>
</route>

Supporting Queues

The default configuration is fine for locally-deployed listeners, but it can be problematic in a distributed context. For instance, if the listener is restarted while a message is sent to the topic, that message may be missed. Furthermore, if there is a networking hiccup between Fedora's local broker and the remote listener, that too can result in lost messages. Instead, in this case, a queue may be better suited.

ActiveMQ supports “virtual destinations”, allowing your broker to automatically forward messages from one location to another. If Fedora4 is deployed in Tomcat, the ActiveMQ configuration will be located in WEB-INF/classes/config/activemq.xml. That file can be edited to include the following block:

Supporting Queues

The default configuration is fine for locally-deployed listeners, but it can be problematic in a distributed context. For instance, if the listener is restarted while a message is sent to the topic, that message may be missed. Furthermore, if there is a networking hiccup between Fedora's local broker and the remote listener, that too can result in lost messages. Instead, in this case, a queue may be better suited.

ActiveMQ supports “virtual destinations”, allowing your broker to automatically forward messages from one location to another. If Fedora4 is deployed in Tomcat, the ActiveMQ configuration will be located in WEB-INF/classes/config/activemq.xml. That file can be edited to include the following block:

Code Block

language	xml
title	activemq.xml customization: supporting a queue/fedora endpoint
linenumbers	true

<destinationInterceptors>
  <virtualDestinationInterceptor>
    <virtualDestinations>
      <compositeTopic name="fedora">
        <forwardTo>
          <queue physicalName="fedora"/>
        </forwardTo>
      </compositeTopic>
    </virtualDestinations>
  </virtualDestinationInterceptor>
</destinationInterceptors>

Now a consumer can pull messages from a queue without risk of losing messages.

This configuration, however, will not allow any other applications to read from the original topic. If it is necessary to have /topic/fedora available to consumers, this configuration will be useful:

Code Block

language	xml
title	activemq.xml customization: supporting topic and queue endpoints
linenumbers	true

<destinationInterceptors>
  <virtualDestinationInterceptor>
    <virtualDestinations>
      <compositeTopic name="fedora" forwardOnly="false">
        <forwardTo>
          <queue physicalName="fedora"/>
        </forwardTo>
      </compositeTopic>
    </virtualDestinations>
  </virtualDestinationInterceptor>
</destinationInterceptors>

Now, both /topic/fedora and /queue/fedora will be available to consumers.

Distributed Brokers

The above example will allow you to distribute the message consumers across multiple machines without missing messages, but it can also be useful to distribute the message broker across multiple machines. This can be especially useful if you want to further decouple the message producers and consumers. It can also be useful for high-availability and failover support.

ActiveMQ supports a variety of distributed broker topologies. To push messages from both the message queue and topic to a remote broker, this configuration can be used:

Code Block

language	xml
title	activemq.xml customization: supporting a queue/fedora endpointdistributed brokers
linenumbers	true

<destinationInterceptors>
  <virtualDestinationInterceptor><networkConnectors>
  <networkConnector name="fedora_bridge" dynamicOnly="true" uri="static:(tcp://remote-host:61616)">
    <virtualDestinations><dynamicallyIncludedDestinations>
      <compositeTopic<topic namephysicalName="fedora">
        <forwardTo>
    />
      <queue physicalName="fedora"/>
        </forwardTo>dynamicallyIncludedDestinations>
      </compositeTopic>networkConnector>
    </virtualDestinations>
  </virtualDestinationInterceptor>
</destinationInterceptors>

Now a consumer can pull messages from a queue without risk of losing messages.

This configuration, however, will not allow any other applications to read from the original topic. If it is necessary to have /topic/fedora available to consumers, this configuration will be useful

</networkConnectors>

Protocol Support

ActiveMQ brokers support a wide variety of protocols. If Fedora's internal broker is bridged to an external broker, please remember to enable the proper protocols on the remote broker. This can be done like so:

Code Block

language	xml
title	activemq.xml customization:

...

protocol support
linenumbers	true

...

<transportConnectors>
  <transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>

...

<transportConnector name="stomp" uri="stomp://0.0.0.0:61613"/>
</transportConnectors>

Each transportConnector supports many additional options that can be added to this configuration.

Deployment

Camel routes can be deployed in any JVM container. In order to deploy to Jetty or Tomcat, the route must be built as a WAR file. This command will get you started:

Code Block

language	bash

$> mvn archetype:generate \
  -DarchetypeGroupId=org.apache.camel.archetypes \
  -DarchetypeArtifactId=camel-archetype-war \
  -DarchetypeVersion=2.14.0 \
  -DgroupId=org.example.camel \
  -DartifactId=my-camel-route \
  -Dversion=1.0.0-SNAPSHOT \
  -Dpackage=org.example.camel

After the project has been built (mvn install), you will find the WAR file in ./target. That file can simply be copied to the webapps directory of your Jetty/Tomcat server.

Another popular deployment option is Karaf, which is a light-weight OSGi-based JVM container. Karaf has the advantage of supporting hot code swapping, which allows you to make sure that your routes are always running. It also allows you to deploy XML-based routes (Spring or Blueprint) by simply copying the files into a $KARAF_HOME/deploy directory. If deploying camel routes to Karaf, Blueprint-based routes have some advantages over the Spring-based DSL, particularly in terms of being able to use property placeholders within your routes.

Karaf can be set up by:

downloading Karaf 4.x or later from an apache.org mirror
running ./bin/karaf to enter the shell

installing required bundles:

Code Block

language	bash
title	Karaf console

$> feature:repo-add camel 2.16.2
$> feature:repo-add activemq 5.11.1
$> feature:install camel
$> feature:install activemq-camel

# display available camel features
$> feature:list | grep camel

# install camel features, as needed
$> feature:install camel-http4
 
# install fcrepo-camel (as of v4.4.0)
$> feature:repo-add mvn:org.fcrepo.camel/fcrepo-camel/4.4.0/xml/features
$> feature:install fcrepo-camel

setting up a service wrapper (so that karaf runs as a system-level service)
Code Block
language bash
title Karaf console
$> feature:install wrapper $> wrapper:install
following the directions provided by this command

Now, routes can be deployed (and re-deployed) by simply copying JAR files or XML documents to $KARAF_HOME/deploy.

Fedora Camel Toolbox

The Fedora project distributes camel routes for several common repository tasks as part of the fcrepo-camel-toolbox project, for use with Karaf version 4.x. Additional information is available on the Integration Services page. Detailed installation instructions are available as part of the project README and follow this pattern

Now, both /topic/fedora and /queue/fedora will be available to consumers.

Distributed Brokers

The above example will allow you to distribute the message consumers across multiple machines without missing messages, but it can also be useful to distribute the message broker across multiple machines. This can be especially useful if you want to further decouple the message producers and consumers. It can also be useful for high-availability and failover support.

ActiveMQ supports a variety of distributed broker topologies. To push messages from both the message queue and topic to a remote broker, this configuration can be used:

Code Block

language	xml
title	activemq.xml customization: distributed brokers
linenumbers	true

<networkConnectors>
  <networkConnector name="fedora_bridge" dynamicOnly="true" uri="static:(tcp://remote-host:61616)">
    <dynamicallyIncludedDestinations>
      <topic physicalName="fedora"/>
      <queue physicalName="fedora"/>
    </dynamicallyIncludedDestinations>
  </networkConnector>
</networkConnectors>

Protocol Support

ActiveMQ brokers support a wide variety of protocols. If Fedora's internal broker is bridged to an external broker, please remember to enable the proper protocols on the remote broker. This can be done like so:

Code Block

language	xml
title	activemq.xml customization: protocol support
linenumbers	true

<transportConnectors>
  <transportConnector name="openwire" uri="tcp://0.0.0.0:61616"/>
  <transportConnector name="stomp" uri="stomp://0.0.0.0:61613"/>
</transportConnectors>

...

Deployment

Camel routes can be deployed in any JVM container. In order to deploy to Jetty or Tomcat, the route must be built as a WAR file. This command will get you started:

Code Block

language	bash

$> mvn archetype:generate \
  -DarchetypeGroupId=org.apache.camel.archetypes \
  -DarchetypeArtifactId=camel-archetype-war \
  -DarchetypeVersion=2.14.0 \
  -DgroupId=org.example.camel \
  -DartifactId=my-camel-route \
  -Dversion=1.0.0-SNAPSHOT \
  -Dpackage=org.example.camel

After the project has been built (mvn install), you will find the WAR file in ./target. That file can simply be copied to the webapps directory of your Jetty/Tomcat server.

Another popular deployment option is Karaf, which is a light-weight OSGi-based JVM container. Karaf has the advantage of supporting hot code swapping, which allows you to make sure that your routes are always running. It also allows you to deploy XML-based routes (Spring or Blueprint) by simply copying the files into a $KARAF_HOME/deploy directory. If deploying camel routes to Karaf, Blueprint-based routes have some advantages over the Spring-based DSL, particularly in terms of being able to use property placeholders within your routes.

Karaf can be set up by:

...

:

Code Block

language	bash
title	Karaf

...

Shell

# install fcrepo-camel-toolbox (as of v4.1.0)
$> feature:

...

repo-add mvn:org.fcrepo.camel/fcrepo-camel-toolbox/4.1.0/xml/features
 
# install fcrepo-camel-toolbox (as of v4.

...

5.0)
$> feature:repo-add mvn:org.fcrepo.camel/

...

toolbox-

...

features/4.

...

5.0/xml/features

...

language	bash
title	Karaf console

# display available features
$> feature:list | grep fcrepo
 
# install

...

feature
$>

...

feature:install

...

 fcrepo-indexing-triplestore

Now, routes can be deployed (and re-deployed) by simply copying JAR files or XML documents to $KARAF_HOME/deploy.

Monitoring Your Camel Routes

...

Page tree

Versions Compared

Old Version 6

New Version Current

Key

Examples

LDPath Transformations

External Triplestore

Event-based Routing

Supporting Queues

Supporting Queues

Distributed Brokers

Protocol Support

Deployment

Fedora Camel Toolbox

Distributed Brokers

Protocol Support

Deployment

Monitoring Your Camel Routes