Building DuraCloud Software from Source

Introduction

DuraCloud application software is composed of many parts. A breakdown of the primary pieces is as follows:

• DuraStore - this web application provides the access to and management of storage resources, which includes handling the storage portion of the DuraCloud REST API
• StorageProviders - this set is made up of the StorageProvider interfaces and the implementations which connect to distinct cloud stores (currently Amazon S3, Rackspace CloudFiles, and Windows Azure)
• DuraService - this web application handles the deployment and management of services within DuraCloud, which includes handling the services portion of the DuraCloud REST API
• DuraBoss - this web application includes the reporter, executor, auditor, and manifest projects, each of which expose their own distinct set of calls via the DuraCloud REST API
• DurAdmin - this web application is the UI front end of DuraCloud, it provides users with a view into the information available from the other applications. DurAdmin uses the REST APIs of the other applications to communicate with them.
• Services - the set of all deployable services, as well as the support projects that allow the DuraCloud services infrastructure to function
• Security - handles security for the DuraCloud applications
• Common - a set of projects which provide utilities for other portions of the codebase to reuse

The DuraCloud software, by its very nature, is designed to be integrated with underlying cloud storage providers. As may be expected, these integrations are necessary for the system to be properly exercised. In order for DuraCloud to connect to these underlying providers, appropriate credentials must be provided as part of the application initialization step. It is recommended that you acquire the necessary storage provider credentials prior to attempting to set up DuraCloud. Only one storage provider is required to run DuraCloud.

The storage providers which are currently supported are listed here.

This guide lays out the steps necessary to begin using DuraCloud:

1. Build and deploy the DuraCloud web applications
2. Set up the OSGi services container
3. Initialize the DuraCloud applications
4. Test your installation

Although this document is written from a Linux environment perspective, analogous builds/installations have been tested in Windows (but may have limitations, as noted below). Any comments or feedback are welcomed.

Prerequisites

Software that must be installed on your system prior to building/using DuraCloud

1. Java 7
   
   1. The Oracle JDK is recommended for building DuraCloud, as this is the JDK used for DuraCloud testing and the djatoka service (used as a basis for one of the DuraCloud services) has known compatibility issues with open-jdk.
2. Maven 2.2.1
3. Tomcat 6.x
4. ActiveMQ 5.7 or above
5. Subversion

Setting up DuraCloud

Build and deploy the DuraCloud web applications

1. Check out latest stable release from Subversion repository

   svn co https://svn.duraspace.org/duracloud/tags/duracloud-2.3.1
   

2. Set environment variables

   export JAVA_OPTS="-XX:MaxPermSize=256m"
   export MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=1024m"
   

3. Configure Tomcat
   

   1. Add to $CATALINA_HOME/conf/tomcat-users.xml

      <tomcat-users>
        <role rolename="manager"/>
        <role rolename="admin"/>
        <user username="[ANY-USERNAME]" password="[ANY-PASSWORD]" roles="admin,manager"/>
      </tomcat-users>
      

   2. Add to  $CATALINA_HOME/conf/server.xml

      Add the config attribute "URIEncoding" with value "UTF-8" to your Tomcat Connector.  Your connector may look like the following:

      <Connector port="8080" protocol="HTTP/1.1" 
                     connectionTimeout="20000" 
                     redirectPort="8443"
                     URIEncoding="UTF-8" />

4. Start Tomcat

   $CATALINA_HOME/bin/startup.sh
   

5. Configure Maven2
   

   1. Add tomcat user to $M2_HOME/conf/settings.xml

      <servers>
        <server>
          <id>tomcat-server</id>
          <username>[ANY-USERNAME]</username>
          <password>[ANY-PASSWORD]</password>
        </server>
      </servers>
      

6. Modify the default ActiveMQ configuration file (there a many configuration options but this is quick approach)

   cd [ACTIVEMQHOME]/conf
   cp activemq.xml activemq.xml.orig
   [yourfaveditor] activemq.xml

   DuraCloud expects the broker to run on localhost:61617. We will make it the default for simplicity by finding the "transport" stanza and modifying it.

   <transportConnectors>
     <!-- DOS protection, limit concurrent connections to 1000 and frame size to 100MB -->
     <transportConnector name="openwire" uri="tcp://0.0.0.0:61617?maximumConnections=1000&amp;wireformat.maxFrameSize=104857600"/>
     <!-- <transportConnector name="openwire" uri="tcp://0.0.0.0:61616?maximumConnections=1000&amp;wireformat.maxFrameSize=104857600"/> -->
     <transportConnector name="amqp" uri="amqp://0.0.0.0:5672?maximumConnections=1000&amp;wireformat.maxFrameSize=104857600"/>
   </transportConnectors>
   

7. Start ActiveMQ

   cd [ACTIVEMQHOME]/bin
   ./activemq start

8. Build
   

   1. From top of source tree

      mvn clean install
      

Set up the OSGi services container

This step assumes the successful completion of the previous build instructions

Once the OSGi services container is running, check to ensure that it was created properly

1. In the console where the "run" script was executed, an "osgi" prompt should be available. If it is not available, hitting enter should bring it up.
2. Type "ss" and hit enter. This should list all of the available bundles. This list should include 50 items, all of which are either in the ACTIVE or RESOLVED state.

Initialize the DuraCloud applications

1. Use the application initialization (app-config) utiltiy to configure the deployed DuraCloud applications

   1. Build app-config utility, from within the //app-config module

      mvn assembly:assembly
      

   2. Run the app-config utility

      java -jar target/app-config-[VERSION]-driver.jar <init.props>
      

      1. The init.props file is a configuration file which specifies all of the information necessary for the DuraCloud applications to run. An example of this file can be found at //app-config/src/main/resources/init.props. This file will need to be updated to match your environment.
   3. When the app-config utility completes successfully, the last line of output printed to the console will be the word "success". If this is not the case, check that your configuration file includes the correct information.

Test your installation

1. Once all of the above steps have been completed, your DuraCloud should be ready to test.
   
   1. Go to http://localhost:8080/duradmin (change host or port as necessary).
   2. Log in using the credentials provided in your configuration file
   3. You should be able to view, add, update, and delete spaces and content in Spaces tab
   4. You should be able to deploy services in the Services tab
2. Congratulations! You now have a functional DuraCloud installation.

Optional items

SyncTool installers

1. The installers for the Sync Tool (Windows, Mac, Linux) are built using BitRock InstallBuilder Enterprise, so this tool must be installed locally.
   
   1. Download and install BitRock InstallBuilder Enterprise version 8.5.2 or above
   2. Place the license file for InstallBuilder in the InstallBuilder installation directory
2. Update the maven settngs.xml file ($M2_HOME/conf/settings.xml) to include the path to the installbuilder executable

   1. Add a duracloud profile to include the InstallBuilder path, and include the duracloud profile in the set of active profiles

      <profiles>
        <profile>
          <id>duracloud</id>
          <properties>
            <installbuilder.executable.path>[FULL-PATH-TO-BITROCK-INSTALLBUILDER-EXECUTABLE]</installbuilder.executable.path>
          </properties>
        </profile>
      </profiles>
      
      ...
      
      <activeProfiles>
        <activeProfile>duracloud</activeProfile>
      </activeProfiles>

   2. From the synctoolui directory in the DuraCloud source tree, execute

      mvn clean install -Pinstallers

   3. The installers can be found under the synctoolui/target directory

Code coverage

1. If you plan on using Clover, the following element needs to be added to your maven 'settings.xml'

   <profiles>
     <profile>
       <id>profile-clover</id>
       <activation>
         <property>
           <name>profile</name>
           <value>clover</value>
         </property>
       </activation>
       <properties>
         <cloverLicense>[LOCATION-OF-clover.license-FILE]</cloverLicense>
       </properties>
     </profile>
   </profiles>
   

2. To run clover

   mvn clover2:instrument clover2:aggregate clover2:clover -Pprofile-clover
   

3. A report will be generated in the following directory:
   //target/site/clover/

Service tests within OSGi (Linux only)

1. Assuming that the OSGi services container is set up and running (as described above), tests that deploy services into the OSGi environment may be run

   1. From inside the //integration-test module

      mvn install -PrunServicesAdminTests
      

Logging

1. DuraCloud uses the SLF4j logging framework backed by the LogBack implementation
2. By adding either a logback.xml or logback-test.xml file on the classpath, logging configuration can be customized

DuraCloud internal tools

ServicesAdmin CLI

1. This tool provides a command line interface for interacting with the 'services-admin' installed in a running OSGi container (see notes above for starting the container)

2. To build and run the CLI, from within the //servicesadminclient module

   mvn assembly:assembly
   java -cp target/servicesadminclient-[VERSION]-cli.jar
   

Building Java client packages

1. To create a distributable zip of the storeclient, serviceclient, or reportclient which includes their dependencies, from within the project directory (//storeclient, //serviceclient, //reportclient) run

   mvn install -Ppackage-client
   

2. The packaged zip will be found under the project's target directory

Misc configuration/discussion

Services on Windows

The following services do not function in a Windows deployment environment

• WebAppUtilService
• HelloWebappWrapper
• J2KService
• ImageMagickService

If you would like to run the ImageConversionService, you must install ImageMagick and have its /bin directory in your PATH, which is essentially what the ImageMagickService does in a Linux environment.