The replication service talks to the ingest-server and will pull data when it is ready to the local preservation area. It then registers the token store and collection with ACE, completing ingestion of content into chronopolis.
Installation
Prereqs
- ACE - The replication shell will need to talk to ACE in order to register collections
- Preservation Area - The replication shell pulls from the ingest server directly to your preservation area on a posix file system. In the future we may expand this, for now it's local disk.
RPM
The replication shell is packaged in an rpm for ease of install
- Download the rpm from jenkins
- Use yum to install the rpm: `yum install replication-shell-$version.rpm`
Installed files are as follows:
- /usr/lib/chronopolis/replication-shell.jar
- /etc/chronopolis/application.properties
- /etc/init.d/replication
A 'chronopolis' user is also created as part of the install process, which can write to /var/log/chronopolis.
Configuration and Running
The replication shell reads the configuration file in /etc/chronopolis/application.properties.
# Sample application.properties
## The node name used when sending email notifications after replications
chron.node=node-name
## The rate at which to poll the ingest api, in cron formatting
replication.cron=0 0 * * * *
## The active profile. Should probably just remain as production
## Other profiles: development, amqp
spring.profiles.active=production
# The path on disk to replicate content to
chron.storage.preservation=/tmp/chronopolis-preservation
# ACE AM configuration
ace.am.host=localhost
ace.am.port=8080
ace.am.path=ace-am
ace.am.user=ace-user
ace.am.password=secret-password
ace.am.validate=false
# Ingest API to poll from
## The endpoints can be a comma separated list, ex:
## https://ingest.chronopolis.org/ingest,https://dev.chronopolis.org/ingest-dev
ingest.api.endpoints=http://ingest-server.chronopolis.org/
ingest.api.username=node-name
ingest.api.password=secret-password
# Logging
logging.path=/var/log/chronopolis/
logging.file=/var/log/chronopolis/replication.log
logging.level.org.springframework=ERROR
logging.level.org.hibernate=ERROR
logging.level.org.chronopolis=DEBUG
# SMTP Configuration# smtp.host=localhost.localdomain
# smtp.to=chron-support@sdsc.edu
# smtp.from=localhost
# smtp.send=false
# smtp.send-on-success=true
Notes on Configuration
- The replication.cron timer sets how often the replication-shell queries the ingest-server for active replications. It uses a cron style formatting:
- See http://www.quartz-scheduler.org/documentation/quartz-1.x/tutorials/crontrigger for an overview of the formatting used
- 0 0 * * * *: The default timer, at the top of every hour
- 0 */1 * * * *: A faster timer, once per minute
- The development profile can be used for testing configuration options. It remains in the foreground and has a limited set of commands.
- If your ACE settings are not configured correctly, the replication-shell will fail to start as it needs to register collections to ACE as part of the replication process
- Currently multiple ingest servers will not be queried, only the first one on the list.
- The mail configuration (smtp) is set to send by default.
- If you don't want to send mail, or have a server which does not have smtp capabilities you can turn it off by uncommenting the smtp.send line
Running
To run the shell, use the init script:
`service replication start`
Errors
This will be filled out as we experience problems. Check /var/log/chronopolis/replication.log to see if there are any stack traces.
Current Implementation
- Query Ingest RESTful API for available transfers
- Transfer data (bag + tokens) to local preservation area
- Register to ACE
- Load tokens for collection in ACE
- Issue audit for collection in ACE
- Update Ingest API with the status of the transfer
