Page History

Introduction

DuraCloud application software is composed of many parts. A breakdown of the primary pieces is as follows:

• DuraStore - this web application provides the access to and management of storage resources, which includes handling the storage portion of the DuraCloud REST API
• StorageProviders - this set is made up of the StorageProvider interfaces and the implementations which connect to distinct cloud stores (currently Amazon S3, Rackspace CloudFiles, and EMC Atmos); the storage providers which are currently supported are listed here.
• DurAdmin DuraService - this web application handles the deployment and management of services within DuraCloud, which includes handling the services portion of the DuraCloud REST API
• Services - the set of all deployable services, as well as the support projects that allow the DuraCloud services infrastructure to function
• ComputeProviders - this set is made up of the ComputeProvider interfaces and the implementation which connect to distinct cloud compute services (currently Amazon EC2, using the typica library)is the UI front end of DuraCloud, it provides users with a view into the information available from the other applications. DurAdmin uses the REST APIs of the other applications to communicate with them.
• Security - handles security for the DuraCloud applications
• Common - a set of projects which provide utilities for other portions of the codebase to reuse

The DuraCloud software, by its very nature, is designed to be integrated with underlying cloud storage and compute providers.
As  As may be expected, these integrations are exercised through many "integration tests"these integrations are necessary for the system to be properly exercised. In order for DuraCloud to connect to these underlying providers, appropriate credentials must first be established and available to the build framework.
However, in order to get up and running quickly with the software, the baseline can be initially built with just its unit tests.
Once the developer has independently acquired credentials for one or more of the supported underlying storage providers (below) builds may then be run with integration tests.

• Amazon S3
• Rackspace CloudFiles
• Windows Azure

This guide lays out the three tiers of building/testing the baseline:

be provided as part of the application configuration step. It is recommended that you acquire the necessary storage provider credentials prior to attempting to set up DuraCloud. Only one storage provider is required to run DuraCloud.

This guide lays out the steps necessary to begin using DuraCloud:

1. Configure DuraCloud database
2. Build and deploy the DuraCloud web applications
3. Test your installation
4. build with unit tests
5. build with storage and services integration tests
6. build with OSGi services container integration tests

Although this document is written from a Linux environment perspective, analogous builds/installations have been tested in Windows (but may have limitations, as noted below). Any comments or feedback are welcomed.

Prerequisites

Software that must be installed on your system prior to building/using DuraCloud

1. Maven 2.2.1 or above
2. Tomcat 6.x or above
3. Java 6 (note: the djatoka service has compatibility issues with open-jdk)
4. Subversion

Building DuraCloud

Any portions of the configuration below for which you need to include a replacement value will be written in all capital letters and included in brackets: \[LIKE-THIS\]

1. Java: Version 8 required
   
   1. The Oracle JDK is recommended for building DuraCloud, as this is the JDK used for DuraCloud testing.
2. Maven 3.x
3. Tomcat 8.x
4. Git
5. DuraCloud Management Console
6. DuraCloud Mill

Setting up DuraCloud

Build and deploy the DuraCloud web applications

...

1. Check out latest stable release from

   Subversion

   GitHub repository

   Code Block
   svn cogit clone --branch duracloud-4.0.0 https://svngithub.duraspace.orgcom/duracloud/tags/duracloud-0.9.0

2. Set environment variables

   Code Block
   export JAVA_OPTS="-XX:MaxPermSize=256m" export MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=1024m"

3. Configure Tomcat

   Code Block $CATALINA

   1. Add to $CATALINA_HOME/conf/tomcat-users.xml

      No Format
      <tomcat-users> <role rolename="manager-gui"/> <role rolename="manager-script"/> <role rolename="admin"/> <user username="[ANY-USERNAME]" password="[ANY-PASSWORD]" roles="admin,manager-gui,manager-script"/> </tomcat-users>

   Start tomcat

   1. Add to  $CATALINA_HOME/

   bin

   1. conf/

   startup.sh
4. Configure Maven2
   1. Add tomcat user to $M2_HOME/conf/settings.xml

      No Format
      <servers> <server> <id>tomcat-server</id> <username>[ANY-USERNAME]</username> <password>[ANY-PASSWORD]</password> </server> </servers>

5. Build with only unit tests
   1. From top of source tree

      Code Block
      mvn clean install -PskipIntTests

      Note: Each of the maven2 project modules are configured to halt the build if there is a test failure. The only exception to this is the integration-test module which is configured to run through all of its tests regardless of failures. So the above build command can optionally exclude the '-PskipIntTests' flag if desired.

Info

Building in Windows: Currently the service projects which deploy as WAR files (beginning with WebAppUtilService) do not pass unit tests on a Windows environment due to limitations regarding the deployment of Tomcat instances. You may choose to either perform a build skipping tests "mvn clean install -DskipTests" or to allow the tests to run but not stop the build "mvn clean install -Dmaven.test.failure.ignore=true"

Build with integration tests

This step assumes the successful completion of the previous build instructions.
As mentioned above, the integration tests require credentials in order to connect to the underlying storage providers. The tests themselves are configured to acquire these credentials from a locally created, encrypted database.

1. Create unit-test-db
   • From inside the //unit-test-db module, run:

     Code Block
     mvn assembly:assembly java -jar target/unit-test-db-[VERSION]-db-util.jar

2. Running the above db util jar will provide a commandline interface for adding credentials
3. Add credentials for s3, emc, rackspace, root-user
   1. Set the root-user to username: 'root', password: 'rpw' to just use the application default.
   2. See root user discussion below for ways of changing the default.
4. Add the connection details for the unit-test-db to $M2_HOME/conf/settings.xml

   No Format

   <profile> <id>always</id> <properties> <duracloud.home>[LOCATION-WHERE-APPLICATION-HAS-WRITE-ACCESS]</duracloud.home> <unit.database.home.default>[LOCATION-WHERE-UNIT-TEST-DB-WAS-CREATED]</unit.database.home.default> <unit.database.password.default>[UNIT-TEST-DB-BOOT-PASSWORD]</unit.database.password.default> </properties> </profile> <activeProfiles> <activeProfile>always</activeProfile> </activeProfiles>

   • where duracloud.home is the directory under which the logs and osgi-container will be placed
   • where unit.database.home.default is the location of the unit-test-database created above
   • where unit.database.password.default is the boot password used during creation of the unit-test-database above
5. The integration tests also expect a registry (read: 'space') of services to be available within the primary storage provider of service storage host
   1. The naming of the space follows the convention "duracloud-<version>-service-repo"
   2. The services to be loaded into the registry can be found in the target directories of each of the service projects
   3. See discussion below about application configuration, including the service storage host
6. Build with unit and integration tests
   1. From top of source tree

      Code Block
      mvn clean install

7. As mentioned before, if there are any failures in the //integration-test module, the build will still complete, but the failures will be listed

Build with OSGi services container integration tests

This step assumes the successful completion of the previous build instructions

Optional items

Code coverage

1. 1. server.xml

      Add the config attribute "URIEncoding" with value "UTF-8" to your Tomcat Connector.  Your connector may look like the following:

      No Format
      <Connector port="8080" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443" URIEncoding="UTF-8" />

   2. Create a duracloud properties file (/any/path/to/duracloud.properties) that contains the following keys: 

      No Format
      mill.db.name=<mill db name> mill.db.port=<mysql port> mill.db.host=<mysql host> mill.db.user=<username> mill.db.pass=<password> db.name=<ama db name> db.host=<mysql ama host> db.port=<mysql ama port> db.user=<username> db.pass=<password>

   3. Add -Dduracloud.config.file=/any/path/to/duracloud.properties to your JAVA_OPTS or CATALINA_OPTS environment variables. You can also set these values in $CATALINA_HOME/bin/setenv.sh (linux,osx) or $CATALINA_HOME/bin/setenv.bat (windows).  The duracloud.config.file system property can also refer to an Amazon S3 address using the s3://<bucket>/<path to file> syntax provided your tomcat instance is running on an instance with the appropriate AWS credentials.  More information on AWS credentials management.

2. Start Tomcat

   Code Block
   $CATALINA_HOME/bin/startup.sh

3. Configure Maven3
   

   1. Add tomcat user to $M2_HOME/conf/settings.xml. Make sure that the username and password used here match those included in the tomcat-users.xml file.

      No Format
      <servers> <server> <id>tomcat-server</id> <username>[ANY-USERNAME]</username> <password>[ANY-PASSWORD]</password> </server> </servers>

4. Build
   

   1. From top of the source tree, execute (note that this build runs unit tests, but not integration tests):

      Code Block
      mvn clean install -DskipIntTests

5. Bonus, running Integration Tests
   1. Grab a copy of the file in the codebase under common-json/src/main/resources/test-config.json and place it somewhere on your system
   2. You'll need an account for each of the providers to be tested. Update the JSON file to include your credentials. These tests actually communicate with each storage provider and verify that the calls being made work properly
   3. Add an environment variable called DURACLOUD-TEST-CONFIG with the value being the full path to your updated credentials JSON file.  A sample credential configuration file can be found in the baseline at  <project root>/integration/src/test/resources/test-config.json

   4. From the top of the source tree, execute 

      Code Block
      mvn clean install -pl integration

Test your installation

1. Once all of the above steps have been completed, your DuraCloud should be ready to test.
   
   1. Go to http://localhost:8080/duradmin (change host or port as necessary).
   2. Log in using valid DuraCloud Management Console credentials.
   3. You should be able to view, add, update, and delete spaces and content in Spaces tab
2. Congratulations! You now have a functional DuraCloud installation.

Optional items

SyncTool installers

1. The installers for the Sync Tool (Windows, Mac, Linux) are built using BitRock InstallBuilder Enterprise, so this tool must be installed locally.
   
   1. Download and install BitRock InstallBuilder Enterprise version 15.1.0 or above
   2. Place the license file for InstallBuilder in the InstallBuilder installation directory
2. Update the maven settngs.xml file ($M2_HOME/conf/settings.xml) to include the path to the installbuilder executableIf you plan on using Clover, the following element needs to be added to your maven 'settings.xml' No Format

   1. Add a duracloud profile to include the InstallBuilder path, and include the duracloud profile in the set of active profiles

      Code Block

   1. <profiles> <profile>

   <id>profile-clover<

   1. <id>duracloud</id>

   <activation>
   <property> <name>profile</name> <value>clover</value> </property> </activation>

   1. <properties>

   <cloverLicense>[LOCATION-OF-clover.license-FILE]</cloverLicense>

   1. <installbuilder.executable.path>[FULL-PATH-TO-BITROCK-INSTALLBUILDER-EXECUTABLE]</installbuilder.executable.path> </properties> </profile> </profiles>

   1. ... <activeProfiles> <activeProfile>duracloud</activeProfile> </activeProfiles>

   2. From the //synctoolui directory in the DuraCloud source tree, execute

   To run clover

   1. Code Block

   1. mvn

   clover2:instrument clover2:aggregate clover2:clover -Pprofile-clover

   1. clean install -Pinstallers

   2. The installers can be found under the synctoolui/target directory
   A report will be generated in the following directory:
   //target/site/clover/

Logging

1. DuraCloud uses the SLF4j SLF4J logging framework backed by the LogBack implementation
2. By adding either a logback.xml or logback-test.xml file on the classpath, logging configuration can be customized

DuraCloud internal tools

ServicesAdmin CLI

1. This tool provides a commandline interface for interacting with the 'services-admin' installed in a running OSGi container (see notes above for starting the container)
2. To build and run the CLI, from within the //servicesadminclient module

   Code Block
   mvn assembly:assembly java -cp target/servicesadminclient-[VERSION]-cli.jar

Application initialization utility

1. This utility takes a config file (example at //app-config/src/main/resources/init.props) and initializes an instance of duracloud
2. Until the applications durastore and duraservice are initialized, they are non-functional
3. To build and run the app-config utility, from within the //app-config module

   Code Block
   mvn assembly:assembly java -jar target/app-config-1.0.0-driver.jar

...

Building Java client packages

1. To create a distributable zip of the storeclient

   and its

   or reportclient which includes their dependencies, from within the project directory (//storeclient, //reportclient) run

   Code Block
   mvn install -Ppackage-client

2. The packaged zip will be found at /storeclient/target/store-client.zip

Misc configuration/discussion

Services on Windows

The following services do not function in a Windows deployment environment

• WebAppUtilService
• HelloWebappWrapper
• J2KService
• ImageMagickService

If you would like to run the ImageConversionService, you must install ImageMagick and have its /bin directory in your PATH, which is essentially what the ImageMagickService does in a linux environment.

root user

application config

...

1. under the project's target directory