Page History

...

• ACE - The replication shell will need to talk to ACE in order to register collections
• SSH + Rsync - Replication is done with rsync over ssh, so keys must me exchanged to any nodes planning on distributing content so that file transfers can occur.
• Preservation Area - The replication shell pulls from the ingest server directly to your preservation area on a posix file system. In the future we may expand this, for now it's local disk.

...

Installed files are as follows:

• /usr/lib/chronopolis/replication-shell.jar
• /etc/chronopolis/application.properties
• /etc/init.d/replication

A 'chronopolis' user is also created as part of the install process, which can write to /var/log/chronopolis.

Configuration and Running

The replication shell reads the configuration file in /etc/chronopolis/application.properties.

# Sample application.properties
## The node name used when sending email notifications after replications
chron.node=node-name

## The rate at which to poll the ingest api, in cron formatting
replication.cron=0 0 * * * *
 
## The active profile. Should probably just remain as production
## Other profiles: development, amqp
spring.profiles.active=production
 
# The path on disk to replicate content to
chron.storage.preservation=/tmp/chronopolis-preservation
 
# ACE AM configuration
ace.am.host=localhost
ace.am.port=8080
ace.am.path=ace-am
ace.am.user=ace-user
ace.am.password=secret-password
ace.am.validate=false
 
# Ingest API to poll from
## The endpoints can be a comma separated list, ex:
## https://ingest.chronopolis.org/ingest,https://dev.chronopolis.org/ingest-dev 
ingest.api.endpoints=http://ingest-server.chronopolis.org/
ingest.api.username=node-name
ingest.api.password=secret-password
 
# Logging
logging.path=/var/log/chronopolis/
logging.file=/var/log/chronopolis/replication.log
logging.level.org.springframework=ERROR
logging.level.org.hibernate=ERROR
logging.level.org.chronopolis=DEBUG
 
# SMTP Configuration
# smtp.host=localhost.localdomain
# smtp.to=chron-support@sdsc.edu
# smtp.from=localhost
# smtp.send=false
# smtp.send-on-success=true

[noarch] $ rpm -qlp replicationd-2.0.0-20171027.el6.noarch.rpm 
/etc/init.d/replicationd
/usr/local/chronopolis/replication
/usr/local/chronopolis/replication/application.yml
/usr/local/chronopolis/replication/replicationd.jar
/var/log/chronopolis

[noarch] $ rpm -qlp replicationd-2.0.0-20171027.el7.noarch.rpm 
/usr/lib/systemd/system/replicationd.service
/usr/local/chronopolis/replication
/usr/local/chronopolis/replication/application.yml
/usr/local/chronopolis/replication/replicationd.jar
/var/log/chronopolis

User Creation

A 'chronopolis' user is also needed as part of the install process who can write to /var/log/chronopolis and into the preservation storage defined in the configuration. This is no longer handled by the rpm installation process but must be done manually.

Configuration and Running

The replication shell reads the configuration file in /usr/local/chronopolis/replication/applicayion.ymlAs of version 2.0, we'll be moving to a yaml based configuration. This looks similar to the above, with a few changes being propagated through the various services to get all the properties to be the same.

# Replication Configuration Properties

# Replication Service Configuration
# node: the name to use when sending notification messages
# send-on-success: flag to enable sending notification on successful replications
chron:
  node: chron
  smtp.send-on-success: true

# ACE-AM Configuration
# am: the endpoint of the Audit Manager application
# username: the username to connect to the Audit Manager with
# password: the password to connect to the Audit Manager with
ace:
  am: http://localhost:8080/ace-am/
  username: user
  password: change-me

# Ingest API Configuration
# endpoint: the endpoint of the Ingest Server
# username: the username to connect to the Ingest Server with
# password: the password tot connect to the Ingest Server with
ingest.api:
  endpoint: https://localhost:8080/ingest/
  username: ingest-user
  password: change-me

# Preservation Storage Configuration: Only posix supported at this time
# posix: a list of Storage Filesystems available 
#   id: the id of the Storage Filesystem (optional for replication - Storage doesn't need to be registered with the Ingest Server)
#   path: the path on disk to the Storage FS
storage.preservation:
  posix:
    - id: 1
      path: /export/bags/
    - id: 2
      path: /export/more-bags/

# Replication Cron Job Configuration
# The rate at which to poll the ingest server for replications
replication.cron: 0 0 * * * * 

# Various Configuration Properties
# timeout: the timeout in Minutes for HTTP communication with the Audit Manager
ace.timeout: 5

# SMTP Configuration
smtp:
  send: true
  to: chron-support-l@mailman.ucsd.edu
  from: localhost
  host: localhost.localdomain

# Specify the active profile for loading various services, normally production
spring.profiles.active: production
spring.pid.file: /var/run/replicationd.pid

# Logging properties
logging.file: replication.log
logging.level:
  org.springframework: ERROR
  org.hibernate: ERROR
  org.chronopolis: DEBUG 

...

• The replication.cron timer sets how often the replication-shell queries the ingest-server for active replications. It uses a cron style formatting:
  • See http://www.quartz-scheduler.org/documentation/quartz-1.x/tutorials/crontrigger for an overview of the formatting used
  • 0 0 * * * *: The default timer, at the top of every hour
  • 0 */1 * * * *: A faster timer, once per minute
• The development profile can be used for testing configuration options. It remains in the foreground and has a limited set of commands.If your ACE settings are not configured correctly, the replication-shell will fail to start as it needs to register collections to ACE as part of the replication process
• Currently multiple ingest servers will not be queried, only the first one on the list.
• The mail configuration (smtp) is set to send by default.
  
  • If you don't want to send mail, or have a server which does not have smtp capabilities you can turn it off by uncommenting the setting smtp.send line

Running

• • : false
• If the storage is not set up correctly, the replicationd process will not run. This includes existence, and r/w/e permissions
• The pid file likely should not be changed. If you do update it make sure to update the init scripts which rely on it as well.

Running

If using EL6To run the shell, use the init.d script `service replication start`replicationd start`

If using EL7, use systemd `systemd start replicationd`

Errors

This will be filled out as we experience problems. Check /var/log/chronopolis/replication.log to see if there are any stack traces.

...