Introduction
DuraCloud application software is composed of many parts. A breakdown of the primary pieces is as follows:
- DuraStore - this web application provides the access to and management of storage resources, which includes handling the storage portion of the DuraCloud REST API
- StorageProviders - this set is made up of the StorageProvider interfaces and the implementations which connect to distinct cloud stores (currently Amazon S3, Rackspace CloudFiles, and EMC Atmos)
- DuraService - this web application handles the deployment and management of services within DuraCloud, which includes handling the services portion of the DuraCloud REST API
- Services - the set of all deployable services, as well as the support projects that allow the DuraCloud services infrastructure to function
- ComputeProviders - this set is made up of the ComputeProvider interfaces and the implementation which connect to distinct cloud compute services (currently Amazon EC2, using the typica library)
- Security - handles security for the DuraCloud applications
- Common - a set of projects which provide utilities for other portions of the codebase to reuse
The DuraCloud software, by its very nature, is designed to be integrated with underlying cloud storage and compute providers.
As may be expected, these integrations are exercised through many "integration tests". In order for DuraCloud to connect to these underlying providers, appropriate credentials must first be established and available to the build framework.
However, in order to get up and running quickly with the software, the baseline can be initially built with just its unit tests.
Once the developer has independently acquired credentials for one or more of the supported underlying storage providers (below) builds may then be run with integration tests.
This guide lays out the three tiers of building/testing the baseline:
- build with unit tests
- build with storage and services integration tests
- build with OSGi services container integration tests
Although this document is written from a Linux environment perspective, analogous builds/installations have been tested in Windows (but may have limitations, as noted below). Any comments or feedback are welcomed.
Prerequisites
Software that must be installed on your system prior to building/using DuraCloud
- Maven 2.2.1 or above
- Tomcat 6.x or above
- Java 6 (note: the djatoka service has compatibility issues with open-jdk)
- Subversion
Building DuraCloud
Any portions of the configuration below for which you need to include a replacement value will be written in all capital letters and included in brackets: [LIKE-THIS]
Build with unit tests
- Check out latest stable release from Subversion repository
svn co https://svn.duraspace.org/duracloud/tags/duracloud-0.9.0
- Set environment variables
export JAVA_OPTS="-XX:MaxPermSize=256m" export MAVEN_OPTS="-Xmx1024m -XX:MaxPermSize=1024m"
- Configure Tomcat
- Add to $CATALINA_HOME/conf/tomcat-users.xml
<tomcat-users> <role rolename="manager"/> <role rolename="admin"/> <user username="[ANY-USERNAME]" password="[ANY-PASSWORD]" roles="admin,manager"/> </tomcat-users>
- Add to $CATALINA_HOME/conf/tomcat-users.xml
- Start tomcat
$CATALINA_HOME/bin/startup.sh
- Configure Maven2
- Add tomcat user to $M2_HOME/conf/settings.xml
<servers> <server> <id>tomcat-server</id> <username>[ANY-USERNAME]</username> <password>[ANY-PASSWORD]</password> </server> </servers>
- Add tomcat user to $M2_HOME/conf/settings.xml
- Build with only unit tests
- From top of source tree
Note: Each of the maven2 project modules are configured to halt the build if there is a test failure. The only exception to this is the integration-test module which is configured to run through all of its tests regardless of failures. So the above build command can optionally exclude the '-PskipIntTests' flag if desired.
mvn clean install -PskipIntTests
- From top of source tree
Building in Windows: Currently the service projects which deploy as WAR files (beginning with WebAppUtilService) do not pass unit tests on a Windows environment due to limitations regarding the deployment of Tomcat instances. You may choose to either perform a build skipping tests "mvn clean install -DskipTests" or to allow the tests to run but not stop the build "mvn clean install -Dmaven.test.failure.ignore=true"
Build with integration tests
This step assumes the successful completion of the previous build instructions.
As mentioned above, the integration tests require credentials in order to connect to the underlying storage providers. The tests themselves are configured to acquire these credentials from a locally created, encrypted database.
- Create unit-test-db
- From inside the //unit-test-db module, run:
mvn assembly:assembly java -jar target/unit-test-db-[VERSION]-db-util.jar
- From inside the //unit-test-db module, run:
- Running the above db util jar will provide a commandline interface for adding credentials
- Add credentials for s3, emc, rackspace, root-user
- Set the root-user to username: 'root', password: 'rpw' to just use the application default.
- See root user discussion below for ways of changing the default.
- Add the connection details for the unit-test-db to $M2_HOME/conf/settings.xml
<profile> <id>always</id> <properties> <duracloud.home>[LOCATION-WHERE-APPLICATION-HAS-WRITE-ACCESS]</duracloud.home> <unit.database.home.default>[LOCATION-WHERE-UNIT-TEST-DB-WAS-CREATED]</unit.database.home.default> <unit.database.password.default>[UNIT-TEST-DB-BOOT-PASSWORD]</unit.database.password.default> </properties> </profile> <activeProfiles> <activeProfile>always</activeProfile> </activeProfiles>
- where duracloud.home is the directory under which the logs and osgi-container will be placed
- where unit.database.home.default is the location of the unit-test-database created above
- where unit.database.password.default is the boot password used during creation of the unit-test-database above
- The integration tests also expect a registry (read: 'space') of services to be available within the primary storage provider of service storage host
- The naming of the space follows the convention "duracloud-<version>-service-repo"
- The services to be loaded into the registry can be found in the target directories of each of the service projects
- See discussion below about application configuration, including the service storage host
- Build with unit and integration tests
- From top of source tree
mvn clean install
- From top of source tree
- As mentioned before, if there are any failures in the //integration-test module, the build will still complete, but the failures will be listed
Build with OSGi services container integration tests
This step assumes the successful completion of the previous build instructions
- Start OSGi service container
cd //services/servicesadmin mvn clean -f pom-run.xml pax:provision cd runner chmod +x run.sh export BUNDLE_HOME=[DURACLOUD_HOME]/osgi-container ./run.sh
Where [DURACLOUD_HOME] is a directory where the application has write access (can be same as <duracloud.home> set in Maven settings.xml above)
- The run.sh script will start an OSGi container and commandline interface to it
- The container starts with required bundles including the 'services-admin' installed
- See discussion below on OSGi container for more details
- Once the 'services-admin' is running, tests that deploy services into the OSGi environment may be run
- From inside the //integration-test module
mvn install -PrunServicesAdminTests
- Set up OSGi service container
cd services/servicesadmin mvn clean -f pom-run.xml pax:provision cd runner
- (Optional) Set the OSGi bundle storage location
set BUNDLE_HOME=[BUNDLE_HOME]
Where [BUNDLE_HOME] is the full path to an empty directory where the osgi container content will be stored
- Open the run.bat file in the runner directory in a text editor and replace all instances of "$BUNDLE_HOME" with "%BUNDLE_HOME%"
- Note: A directory called "$BUNDLE_HOME" under the runner directory will be used as the default bundle home if one is not specified.
- (Optional) Set up logging
- Download the logback.xml file found here into your bundle home directory.
- Open the logback.xml file in a text editor and edit the LOG_FILENAME property to point to a full file path (including file name) for a log file.
- Note: One benefit to performing this step will be faster start time for your OSGi container.
- Start OSGI service container
run.bat
- The run.bat script will start an OSGi container and commandline interface to it
- The container starts with required bundles including the 'services-admin' installed
- See discussion below on OSGi container for more details
- Once the 'services-admin' is running, check to ensure that it was created properly
- In the console where run.bat was executed, an "osgi" prompt should be available. If it is not available, hitting enter should bring it up.
- Type "ss" and hit enter. This should list all of the available bundles. This list should include 50 items, all of which are either in the ACTIVE or RESOLVED state.
Optional items
Code coverage
- If you plan on using Clover, the following element needs to be added to your maven 'settings.xml'
<profiles> <profile> <id>profile-clover</id> <activation> <property> <name>profile</name> <value>clover</value> </property> </activation> <properties> <cloverLicense>[LOCATION-OF-clover.license-FILE]</cloverLicense> </properties> </profile> </profiles>
- To run clover
mvn clover2:instrument clover2:aggregate clover2:clover -Pprofile-clover
- A report will be generated in the following directory:
//target/site/clover/
Logging
- DuraCloud uses the SLF4j logging framework backed by the LogBack implementation
- By adding either a logback.xml or logback-test.xml file on the classpath, logging configuration can be customized
DuraCloud internal tools
ServicesAdmin CLI
- This tool provides a commandline interface for interacting with the 'services-admin' installed in a running OSGi container (see notes above for starting the container)
- To build and run the CLI, from within the //servicesadminclient module
mvn assembly:assembly java -cp target/servicesadminclient-[VERSION]-cli.jar
Application initialization utility
- This utility takes a config file (example at //app-config/src/main/resources/init.props) and initializes an instance of duracloud
- Until the applications durastore and duraservice are initialized, they are non-functional
- To build and run the app-config utility, from within the //app-config module
mvn assembly:assembly java -jar target/app-config-1.0.0-driver.jar
StoreClient package
- To create a distributable zip of the storeclient and its dependencies, from within //storeclient run
mvn install -Ppackage-client
- The zip will be found at /storeclient/target/store-client.zip
Misc configuration/discussion
Services on Windows
The following services do not function in a Windows deployment environment
- WebAppUtilService
- HelloWebappWrapper
- J2KService
- ImageMagickService
If you would like to run the ImageConversionService, you must install ImageMagick and have its /bin directory in your PATH, which is essentially what the ImageMagickService does in a linux environment.