The Bridge Intake Service provides data pulled down from the Duracloud Bridge App a path into Chronopolis. It currently handles bagging of snapshots, which includes validating files from the manifest given by Duracloud.

Links

• Gitlab: Bridge-Intake
• Github:Bridge-Restore-2

Installation

Prereqs

• Running Duracloud Bridge App to connect to
• Staging area for creating bags

Install

1. Get the latest rpm from http://adaptci01.umiacs.umd.edu/resource/bridge-intake/master/
2. yum install

RPM

Installation can be done through the provided rpm. This offers builds for both RHEL6 and RHEL7.

Installation Notes

Installed Files For RHEL6 

/etc/init.d/bridge-intake
/usr/local/chronopolis/intake/dc
/usr/local/chronopolis/intake/dc/application.yml
/usr/local/chronopolis/intake/dc/bridge-intake.jar

Installed Files For RHEL7Installed files are as follows:

/usr/lib/

...

systemd/system/bridge-intake.

...

service
/usr/local/chronopolis/intake/dc
/usr/local/chronopolis/intake/dc/application.yml

...

/usr/local/chronopolis/intake/dc/bridge-intake.jar
/usr/local/chronopolis/intake/dc/bridge-

...

Configuration

Depending on where the data is bound, there are several configuration options available

Chronopolis

Prefix

Specifying Replicating Nodes

DPN

Sample application.yml with all properties

prepare

When running, the service will check for the following directories and create/apply permissions if they do not match:

• System Logging: /var/log/chronopolis/
• Journal data: /var/lib/chronopolis/data/

Configuration

# Cron timer for how often the bridge is polled
bridge:
  poll: 0 0 0 * * *
 
# General chron configuration
## chron.node: obsolete;
## chron.workDirectory: directory for storing information used for ongoing operations, e.g. file  node: chron
  bags: /export/bags
  tokens: /export/tokens
  preservation: /data/preservation
  restoration: /export/restore
  prefix: bridge-
  ingest:
    endpoint: http://localhost:8081/
    username: ingest-admin
    password: replace-me
  replicatingTo:
    - ucsd
    - umiacs

# DPN Configuration - the server to use when creating replications
#                     and connection information for the registry
dpnReplicationServer: dpn-staging.ucsd.edu
dpncsv
## chron.stage.bags.posix.id: the id of the bag staging area for this service
## chron.stage.bags.posix.path: the path on disk to the bag staging area for this service
chron:
  node: chron
  workDirectory: /tmp/chronopolis
  stage.bags:
    posix.id: 4
    posix.path: /scratch0/bags
 
# Chronopolis Ingest API Configuration
## ingest.api.endpoint: the http endpoint of the Ingest Server
## ingest.api.username: the username to connect to the Ingest Server with
## ingest.api.password: the password to connect to the Ingest Server with
ingest.api:
  endpoint: http://localhost:8000/
  username: admin
  password: admin
 
# ACE IMS Configuration
## ace.ims.endpoint: the fqdn of the ims
## ace.ims.port: the port to connect to the ims on; one of: 80, 443
##
ace.ims:
  endpoint: http://localhost:3000/
  usernameport: chron80
  api-keywaitTime: replace-me
6000
 
 
# Duracloud Bridge Configuration
# The storage areas the bridge app writes in to
# Connection information to query the bridge
## duracloud.bridge.name: an identifier for the bridge
## duracloud.bridge.snapshots: the path on disk to the snapshots directory
## duracloud.bridge.restores: the path on disk to the restores directory
## duracloud.bridge.username: the username to use when querying the bridge
## duracloud.bridge.password: the password to use when querying the bridge
## duracloud.bridge.password: the http endpoint of the bridge
duracloud:
  bridge:
    - name: bridge-0
      snapshots: /exportbridge-0/snapshots/
      restores: /exportbridge-0/restore/
  bridge: 
    username: bridgereplace-me
      password: replace-me
      endpoint: http://localhost:8080/

# Push settings to decide what networks to push the snapshots into
pushDPN: false
pushChronopolis: false    - name: bridge-1
      snapshots: /bridge-1/snapshots/
      restores: /bridge-1/restore/
      username: replace-me
      password: replace-me
      endpoint: http://localhost:8081/

# Logging configuration
logging:
  file: /var/log/bridgeintake/intake.log
  level:
    org.springframework: ERROR
    org.hibernate: ERROR
    org.chronopolis: debug
    org.chronopolis.intake.duracloud.config: trace
 
# Extraneous settings
#   automatic cleaning of staging areas (not well tested)
#   only perform dry-runs when the Cleaner runs
#   disable SNI on https connections - false is the default and recommended value
cleanerEnabled: false
cleanDryRun: false
disableSNI: false

...

disableSNI: false

Release Notes

Release 3.0.1

09 April, 2019

• Bug Fix: Resolve error with startup failing to configure our beans

Release 3.0.0

21 March, 2019

• Remove deprecated DPN workflow
• Update .gitlab-ci.yml to cache maven artifacts

Release 2.4.3

21 March, 2019

• Update Chronopolis and chron-test dependencies to latest

Release 2.4.2

23 January, 2019

• Bump Chronopolis dependency version
• Bug Fix: Fix csv upload to create the MultiPart data correctly

Release 2.4.1

22 January, 2019

• Bump Chronopolis dependency version

Release 2.4.0

11 January, 2019

• Bridge consolidation work to bring multiple bridge querying to a single service
• Update notification of missing Depositors to fire only once for the lifetime of the bridge
• Fix Bug resulting in successive snapshot runs not being processed if a check against the Depositor fails
• Update ChronopolisIngest for 3.0.0 workflow

Release 2.3.2

09 August, 2018

• Fixed bug which was preventing deserialization of certain classes
• Temporarily disabled HttpFilter because of bug in Ingest Server

Release 2.3.1

07 August, 2018

• Updated logic for appending a prefix to a depositor so that it is used in every operation

Release 2.3.0

10 July, 2018

• Code licensed under BSD-3
• Use separate thread pools when executing Ingestion tasks
  • Long IO operations - bagging, cleaning, etc
  • Short IO operations - HTTP requests
• Bring tokenization process up to date with Chronopolis Core code
• Test that a depositor exists before attempting to push a Bag to Chronopolis or DPN

Release 2.2.1

18 May, 2018

• Fix bug where constraint for bag size was checked incorrectly
• Fix bug where tokens were not being created because a dependency was missing

Release 2.2.0

14 May, 2018

• Fix bug where we could potentially create the wrong number of replications
• Add constraint satisfaction when creating DPN Replications
• Refactor code for Chronopolis and DPN Ingest to better facilitate working on snapshots concurrently
  • Break up DPN Ingestion steps into multiple classes
  • Use standard lib interfaces instead of spring-batch for tasks

Release 2.1.0

8 February, 2018

• Actively clean storage upon completion of data replicating
  • For chronopolis: Ingest reports PRESERVED
  • For DPN: All replications completed
• Mark staging as inactive in Ingest for completed replications in Chronopolis

Release 2.0.3

29 November, 2017

• Query on Bags we created when running Tokenization

Release 2.0.2

9 November, 2017

• Remove creation of /var/log/chronopolis from rpm and instead create it on application startup

Release 2.0.1

8 November, 2017

• Use correct properties when creating Ingest Requests to Chronopolis

Release 2.0.0

30 October, 2017

...