Versions Compared

Old Version 16

changes.mady.by.user Michael Ritter

Saved on

compared with

New Version 17

changes.mady.by.user Michael Ritter

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

The Bridge Intake Service provides data pulled down from the Duracloud Bridge App a path into Chronopolis. It currently handles bagging of snapshots, which includes validating files from the manifest given by Duracloud.

Links

Installation

Prereqs

  • Running Duracloud Bridge App to connect to
  • Staging area for creating bags

Install

  1. Get the latest rpm from http://adaptci01.umiacs.umd.edu/resource/bridge-intake/master/
  2. yum install

RPM

Installation can be done through the provided rpm. This offers builds for both RHEL6 and RHEL7.

Installation Notes

Installed Files For RHEL6 

/etc/init.d/bridge-intake
/usr/local/chronopolis/intake/dc
/usr/local/chronopolis/intake/dc/application.yml
/usr/local/chronopolis/intake/dc/bridge-intake.jar

Installed Files For RHEL7Installed files are as follows:

/usr/lib/systemd/

...

system/bridge-intake.

...

service
/usr/local/chronopolis/intake/dc
/usr/local/chronopolis/intake/dc/application.yml

...


/usr/local/chronopolis/intake/dc/bridge-intake.jar
/usr/local/chronopolis/intake/dc/bridge-

...

prepare

When running, the service will check for the following directories and create/apply permissions if they do not match:

  • System Logging: /var/log/chronopolis/
  • logging Journal data: /var/lib/chronopolis/data/journal data for tokenization

Configuration

...

Depending on where the data is bound, there are several configuration options available

Chronopolis

...

Code Block
languageyml
titleapplication.yml
collapsetrue
# Cron timer for how often the bridge is polled
bridge:
  poll: 0 0 0 * * *

# Bagging Configuration
## bag.unit: The unit for the maximum size of a Bag
## bag.max-size: The maximum size a Bag is allowed to be
## bag.dpn.node-address: The 'DPN-Node-Address' information field for dpn-info.txt | DEPRECATED
## bag.dpn.node-contact: The 'DPN-Node-Contact' information field for dpn-info.txt | DEPRECATED
## bag.dpn.node-email: The 'DPN-Node-Email' information field for dpn-info.txt     | DEPRECATED
bag:
  max-size: 2
  unit: GIGABYTE
  dpn:
    node-address: University of California, San Diego, 9500 Gilman Dr, La Jolla, CA 92093
    node-contact: Sibyl Schaefer
    node-email:
      - sschaefer@ucsd.edu
      - chronopolis-support-l@mailman.ucsd.edu

# General chron configuration
## chron.node: obsolete;
## chron.workDirectory: directory for storing information used for ongoing operations, e.g. file csv
## chron.stage.bags.posix.id: the id of the bag staging area for this service
## chron.stage.bags.posix.path: the path on disk to the bag staging area for this service 
chron:
  node: chron
  workDirectory: /tmp/chronopolis
  stage.bags:
    posix.id: 4
    posix.path: /scratch0/bags
 
# Chronopolis Ingest API Configuration
## ingest.api.endpoint: the http endpoint of the Ingest Server
## ingest.api.username: the username to connect to the Ingest Server with
## ingest.api.password: the password to connect to the Ingest Server with
ingest.api:
  endpoint: http://localhost:8000/
  username: admin
  password: admin
 
# ACE IMS Configuration
## ace.ims.endpoint: the fqdn of the ims 
## ace.ims.port: the port to connect to the ims on; one of: 80, 443
##
ace.ims:
  endpoint: localhost
  port: 80
  waitTime: 6000

# Constraint information for DPN Replications | DEPRECATED
## constraints.nodes.name: the identifier of the DPN Node
## constraints.nodes.size-limit: limit based on the size-limit of a snapshot
## constraints.nodes.size-limit.unit: the unit to measure the snapshot in
## constraints.nodes.size-limit.size: the size of the snapshot
## constraints.nodes.members: limit based on the depositing member of a snapshot
constraints:
  nodes:
    - name: aptrust
      size-limit: 
        unit: TERABYTE
        size: 10
      members:
        - member-uuid-0
        - member-uuid-1
        - member-uuid-2


# DPN Configuration - the server to use when creating replications
#                     and connection information for the registry | DEPRECATED
dpnReplicationServer: dpn-staging.ucsd.edu
dpn:
  endpoint: http://localhost:3000/
  username: chron
  api-key: replace-me

# Duracloud Bridge Configuration
# The storage areas the bridge app writes in to
# Connection information to query the bridge
## duracloud.bridge.name: an identifier for the bridge
## duracloud.bridge.snapshots: the path on disk to the snapshots directory
## duracloud.bridge.restores: the path on disk to the restores directory
## duracloud.bridge.username: the username to use when querying the bridge
## duracloud.bridge.password: the password to use when querying the bridge
## duracloud.bridge.password: the http endpoint of the bridge 
duracloud:
  bridge:
    - name: bridge-0
      snapshots: /bridge-0/snapshots/
      restores: /bridge-0/restore/
      username: replace-me
      password: replace-me
      endpoint: http://localhost:8080/
    - name: bridge-1
      snapshots: /bridge-1/snapshots/
      restores: /bridge-1/restore/
      username: replace-me
      password: replace-me
      endpoint: http://localhost:8081/

# Push settings to decide what networks to push the snapshots into
pushDPN: false | DEPRECATED
pushChronopolis: false

# Logging configuration
logging:
  file: /var/log/bridgeintake/intake.log
  level:
    org.springframework: ERROR
    org.hibernate: ERROR
    org.chronopolis: debug
    org.chronopolis.intake.duracloud.config: trace
 
# Extraneous settings
#   disable SNI on https connections - false is the default and recommended value
disableSNI: false

Multi-Bridge Support

As of Release 2.4.0, configuration for querying multiple bridges is supported. This is done in the application.yml under the

Highlight
colorcyan
Duracloud

  properties. e.g.

Code Block
languageyml
duracloud:
  bridge:
    - name: Dev Duracloud Bridge
      snapshots: /duracloud-bridge/snapshots/
      restores: /duracloud-bridge/restore/
      username: dura-api-username
      password: dura-api-password
      endpoint: https://duracloud-bridge-dev.ucsd.edu/
    - name: Dev Chronopolis Bridge
      snapshots: /chrono-bridge/snapshots/
      restores: /chrono-bridge/restore/
      username: chrono-api-username
      password: chrono-api-password
      endpoint: http://tdl-bridge-dev.ucsd.edu/

...

Release Notes

Release 3.0.1

09 April, 2019

  • Bug Fix: Resolve error with startup failing to configure our beans

Release 3.0.0

21 March, 2019

  • Remove deprecated DPN workflow
  • Update .gitlab-ci.yml to cache maven artifacts

Release 2.4.3

21 March, 2019

  • Update Chronopolis and chron-test dependencies to latest

Release 2.4.2

23 January, 2019

  • Bump Chronopolis dependency version
  • Bug Fix: Fix csv upload to create the MultiPart data correctly

Release 2.4.1

22 January, 2019

  • Bump Chronopolis dependency version

Release 2.4.0

11 January, 2019

...