Replication

The replication service talks to the ingest-server and will pull data when it is ready to the local preservation area. It then registers the token store and collection with ACE, completing ingestion of content into chronopolis.

Installation

Prereqs

ACE - The replication shell will need to talk to ACE in order to register collections
Preservation Area - The replication shell pulls from the ingest server directly to your preservation area on a posix file system. In the future we may expand this, for now it's local disk.

RPM

The replication shell is packaged in an rpm for ease of install

Download the rpm from our build server
Use yum to install the rpm: `yum install replication-shell-$version.rpm`

Installed files are as follows:

/usr/lib/chronopolis/replication-shell.jar
/etc/chronopolis/application.properties
/etc/init.d/replication

A 'chronopolis' user is also created as part of the install process, which can write to /var/log/chronopolis.

Configuration and Running

The replication shell reads the configuration file in /etc/chronopolis/application.properties.

application.properties

# Sample application.properties
## The node name used when sending email notifications after replications
chron.node=node-name

## The rate at which to poll the ingest api, in cron formatting
replication.cron=0 0 * * * *
 
## The active profile. Should probably just remain as production
## Other profiles: development, amqp
spring.profiles.active=production
 
# The path on disk to replicate content to
chron.storage.preservation=/tmp/chronopolis-preservation
 
# ACE AM configuration
ace.am.host=localhost
ace.am.port=8080
ace.am.path=ace-am
ace.am.user=ace-user
ace.am.password=secret-password
ace.am.validate=false
 
# Ingest API to poll from
## The endpoints can be a comma separated list, ex:
## https://ingest.chronopolis.org/ingest,https://dev.chronopolis.org/ingest-dev 
ingest.api.endpoints=http://ingest-server.chronopolis.org/
ingest.api.username=node-name
ingest.api.password=secret-password
 
# Logging
logging.path=/var/log/chronopolis/
logging.file=/var/log/chronopolis/replication.log
logging.level.org.springframework=ERROR
logging.level.org.hibernate=ERROR
logging.level.org.chronopolis=DEBUG
 
# SMTP Configuration
# smtp.host=localhost.localdomain
# smtp.to=chron-support@sdsc.edu
# smtp.from=localhost
# smtp.send=false
# smtp.send-on-success=true

As of version 2.0, we'll be moving to a yaml based configuration. This looks similar to the above, with a few changes being propagated through the various services to get all the properties to be the same.

application.yml

# Replication Configuration Properties

# Replication Service Configuration
# node: the name to use when sending notification messages
# send-on-success: flag to enable sending notification on successful replications
chron:
  node: chron
  smtp.send-on-success: true

# ACE-AM Configuration
# am: the endpoint of the Audit Manager application
# username: the username to connect to the Audit Manager with
# password: the password to connect to the Audit Manager with
ace:
  am: http://localhost:8080/ace-am/
  username: user
  password: change-me

# Ingest API Configuration
# endpoint: the endpoint of the Ingest Server
# username: the username to connect to the Ingest Server with
# password: the password tot connect to the Ingest Server with
ingest.api:
  endpoint: https://localhost:8080/ingest/
  username: ingest-user
  password: change-me

# Preservation Storage Configuration: Only posix supported at this time
# posix: a list of Storage Filesystems available 
#   id: the id of the Storage Filesystem (optional for replication - Storage doesn't need to be registered with the Ingest Server)
#   path: the path on disk to the Storage FS
storage.preservation:
  posix:
    - id: 1
      path: /export/bags/
    - id: 2
      path: /export/more-bags/

# Replication Cron Job Configuration
# The rate at which to poll the ingest server for replications
replication.cron: 0 0 * * * * 

# Various Configuration Properties
# timeout: the timeout in Minutes for HTTP communication with the Audit Manager
ace.timeout: 5

# SMTP Configuration
smtp:
  send: true
  to: chron-support-l@mailman.ucsd.edu
  from: localhost
  host: localhost.localdomain

# Specify the active profile for loading various services, normally production
spring.profiles.active: production

# Logging properties
logging.file: replication.log
logging.level:
  org.springframework: ERROR
  org.hibernate: ERROR
  org.chronopolis: DEBUG

Notes on Configuration

The replication.cron timer sets how often the replication-shell queries the ingest-server for active replications. It uses a cron style formatting:
- See http://www.quartz-scheduler.org/documentation/quartz-1.x/tutorials/crontrigger for an overview of the formatting used
- 0 0 * * * *: The default timer, at the top of every hour
- 0 */1 * * * *: A faster timer, once per minute
The development profile can be used for testing configuration options. It remains in the foreground and has a limited set of commands.
If your ACE settings are not configured correctly, the replication-shell will fail to start as it needs to register collections to ACE as part of the replication process
Currently multiple ingest servers will not be queried, only the first one on the list.
The mail configuration (smtp) is set to send by default.
- If you don't want to send mail, or have a server which does not have smtp capabilities you can turn it off by uncommenting the smtp.send line

Running

To run the shell, use the init.d script `service replication start`

Errors

This will be filled out as we experience problems. Check /var/log/chronopolis/replication.log to see if there are any stack traces.

Current Implementation

Query Ingest RESTful API for available transfers
Transfer data (bag + tokens) to local preservation area
Register to ACE
Load tokens for collection in ACE
Issue audit for collection in ACE
Update Ingest API with the status of the transfer

Space shortcuts

Page tree