Page tree
Skip to end of metadata
Go to start of metadata

The Bridge Intake Service provides data pulled down from the Duracloud Bridge App a path into Chronopolis. It currently handles bagging of snapshots, which includes validating files from the manifest given by Duracloud.

Links

Installation

Prereqs

  • Running Duracloud Bridge App to connect to
  • Staging area for creating bags

RPM

Installation can be done through the provided rpm. This offers builds for both RHEL6 and RHEL7.

Installation Notes

Installed Files For RHEL6 

/etc/init.d/bridge-intake
/usr/local/chronopolis/intake/dc
/usr/local/chronopolis/intake/dc/application.yml
/usr/local/chronopolis/intake/dc/bridge-intake.jar

Installed Files For RHEL7

/usr/lib/systemd/system/bridge-intake.service
/usr/local/chronopolis/intake/dc
/usr/local/chronopolis/intake/dc/application.yml
/usr/local/chronopolis/intake/dc/bridge-intake.jar
/usr/local/chronopolis/intake/dc/bridge-prepare

When running, the service will check for the following directories and create/apply permissions if they do not match:

  • System Logging: /var/log/chronopolis/
  • Journal data: /var/lib/chronopolis/data/

Configuration

application.yml
# Cron timer for how often the bridge is polled
bridge:
  poll: 0 0 0 * * *
 
# General chron configuration
## chron.node: obsolete;
## chron.workDirectory: directory for storing information used for ongoing operations, e.g. file csv
## chron.stage.bags.posix.id: the id of the bag staging area for this service
## chron.stage.bags.posix.path: the path on disk to the bag staging area for this service
chron:
  node: chron
  workDirectory: /tmp/chronopolis
  stage.bags:
    posix.id: 4
    posix.path: /scratch0/bags
 
# Chronopolis Ingest API Configuration
## ingest.api.endpoint: the http endpoint of the Ingest Server
## ingest.api.username: the username to connect to the Ingest Server with
## ingest.api.password: the password to connect to the Ingest Server with
ingest.api:
  endpoint: http://localhost:8000/
  username: admin
  password: admin
 
# ACE IMS Configuration
## ace.ims.endpoint: the fqdn of the ims
## ace.ims.port: the port to connect to the ims on; one of: 80, 443
##
ace.ims:
  endpoint: localhost
  port: 80
  waitTime: 6000
 
 
# Duracloud Bridge Configuration
# The storage areas the bridge app writes in to
# Connection information to query the bridge
## duracloud.bridge.name: an identifier for the bridge
## duracloud.bridge.snapshots: the path on disk to the snapshots directory
## duracloud.bridge.restores: the path on disk to the restores directory
## duracloud.bridge.username: the username to use when querying the bridge
## duracloud.bridge.password: the password to use when querying the bridge
## duracloud.bridge.password: the http endpoint of the bridge
duracloud:
  bridge:
    - name: bridge-0
      snapshots: /bridge-0/snapshots/
      restores: /bridge-0/restore/
      username: replace-me
      password: replace-me
      endpoint: http://localhost:8080/
    - name: bridge-1
      snapshots: /bridge-1/snapshots/
      restores: /bridge-1/restore/
      username: replace-me
      password: replace-me
      endpoint: http://localhost:8081/

# Logging configuration
logging:
  file: /var/log/bridgeintake/intake.log
  level:
    org.springframework: ERROR
    org.hibernate: ERROR
    org.chronopolis: debug
    org.chronopolis.intake.duracloud.config: trace
 
# Extraneous settings
#   disable SNI on https connections - false is the default and recommended value
disableSNI: false

Release Notes

Release 3.0.1

09 April, 2019

  • Bug Fix: Resolve error with startup failing to configure our beans

Release 3.0.0

21 March, 2019

  • Remove deprecated DPN workflow
  • Update .gitlab-ci.yml to cache maven artifacts

Release 2.4.3

21 March, 2019

  • Update Chronopolis and chron-test dependencies to latest

Release 2.4.2

23 January, 2019

  • Bump Chronopolis dependency version
  • Bug Fix: Fix csv upload to create the MultiPart data correctly

Release 2.4.1

22 January, 2019

  • Bump Chronopolis dependency version

Release 2.4.0

11 January, 2019

  • Bridge consolidation work to bring multiple bridge querying to a single service
  • Update notification of missing Depositors to fire only once for the lifetime of the bridge
  • Fix Bug resulting in successive snapshot runs not being processed if a check against the Depositor fails
  • Update ChronopolisIngest for 3.0.0 workflow

Release 2.3.2

09 August, 2018

  • Fixed bug which was preventing deserialization of certain classes
  • Temporarily disabled HttpFilter because of bug in Ingest Server

Release 2.3.1

07 August, 2018

  • Updated logic for appending a prefix to a depositor so that it is used in every operation

Release 2.3.0

10 July, 2018

  • Code licensed under BSD-3
  • Use separate thread pools when executing Ingestion tasks
    • Long IO operations - bagging, cleaning, etc
    • Short IO operations - HTTP requests
  • Bring tokenization process up to date with Chronopolis Core code
  • Test that a depositor exists before attempting to push a Bag to Chronopolis or DPN

Release 2.2.1

18 May, 2018

  • Fix bug where constraint for bag size was checked incorrectly
  • Fix bug where tokens were not being created because a dependency was missing

Release 2.2.0

14 May, 2018

  • Fix bug where we could potentially create the wrong number of replications
  • Add constraint satisfaction when creating DPN Replications
  • Refactor code for Chronopolis and DPN Ingest to better facilitate working on snapshots concurrently
    • Break up DPN Ingestion steps into multiple classes
    • Use standard lib interfaces instead of spring-batch for tasks

Release 2.1.0

8 February, 2018

  • Actively clean storage upon completion of data replicating
    • For chronopolis: Ingest reports PRESERVED
    • For DPN: All replications completed
  • Mark staging as inactive in Ingest for completed replications in Chronopolis

Release 2.0.3

29 November, 2017

  • Query on Bags we created when running Tokenization

Release 2.0.2

9 November, 2017

  • Remove creation of /var/log/chronopolis from rpm and instead create it on application startup

Release 2.0.1

8 November, 2017

  • Use correct properties when creating Ingest Requests to Chronopolis

Release 2.0.0

30 October, 2017

  • Integrate Chronopolis 2.0.0 Changes
    • Import tokenization module in order to create ACE Tokens for incoming data
    • Use new properties classes for configuration of storage and api communication
    • Capture basic metrics when creating bags
    • Update ingest flow to create bag and fixity
  • Updates to build and init scripts
    • Support EL6 and EL7
    • Quality of life changes to the sysv init script based on rpmlint

Release 1.5.1

30 June, 2017

  • Fix bug that caused replications to be created for each node in DPN

Release 1.5.0

26 June, 2017

  • Add a prefix for chronopolis bound collections
  • When creating replications for dpn, choose replicating nodes in a deterministic manner per snapshot so that all bags in a snapshot end up at the same DPN nodes.


  • No labels