The Bridge Intake Service provides data pulled down from the Duracloud Bridge App a path into Chronopolis. It currently handles bagging of snapshots, which includes validating files from the manifest given by Duracloud.
Installation
Prereqs
- Running Duracloud Bridge App to connect to
- Staging area for creating bags
Install
- Get the latest rpm from http://adaptci01.umiacs.umd.edu/resource/bridge-intake/master/
- yum install
Installed files are as follows:
- /usr/lib/chronopolis/bridge-intake.jar
- /etc/chronopolis/application.yml
- /etc/init.d/bridge-intake
When running, the service will check for the following directories and create/apply permissions if they do not match:
- /var/log/chronopolis/
- logging data
- /var/lib/chronopolis/data/
- journal data for tokenization
Configuration
Depending on where the data is bound, there are several configuration options available
Chronopolis
Sample application.yml with all properties
Multi-Bridge Support
As of Release 2.4.0, configuration for querying multiple bridges is supported. This is done in the application.yml under theDuracloud properties. e.g.
duracloud: bridge: - name: Dev Duracloud Bridge snapshots: /duracloud-bridge/snapshots/ restores: /duracloud-bridge/restore/ username: dura-api-username password: dura-api-password endpoint: https://duracloud-bridge-dev.ucsd.edu/ - name: Dev Chronopolis Bridge snapshots: /chrono-bridge/snapshots/ restores: /chrono-bridge/restore/ username: chrono-api-username password: chrono-api-password endpoint: http://tdl-bridge-dev.ucsd.edu/
Release Notes
Release 2.4.0
11 January, 2019
- Bridge consolidation work to bring multiple bridge querying to a single service
- Update notification of missing Depositors to fire only once for the lifetime of the bridge
- Fix Bug resulting in successive snapshot runs not being processed if a check against the Depositor fails
- Update ChronopolisIngest for 3.0.0 workflow
Release 2.3.2
09 August, 2018
- Fixed bug which was preventing deserialization of certain classes
- Temporarily disabled HttpFilter because of bug in Ingest Server
Release 2.3.1
07 August, 2018
- Updated logic for appending a prefix to a depositor so that it is used in every operation
Release 2.3.0
10 July, 2018
- Code licensed under BSD-3
- Use separate thread pools when executing Ingestion tasks
- Long IO operations - bagging, cleaning, etc
- Short IO operations - HTTP requests
- Bring tokenization process up to date with Chronopolis Core code
- Test that a depositor exists before attempting to push a Bag to Chronopolis or DPN
Release 2.2.1
18 May, 2018
- Fix bug where constraint for bag size was checked incorrectly
- Fix bug where tokens were not being created because a dependency was missing
Release 2.2.0
14 May, 2018
- Fix bug where we could potentially create the wrong number of replications
- Add constraint satisfaction when creating DPN Replications
- Refactor code for Chronopolis and DPN Ingest to better facilitate working on snapshots concurrently
- Break up DPN Ingestion steps into multiple classes
- Use standard lib interfaces instead of spring-batch for tasks
Release 2.1.0
8 February, 2018
- Actively clean storage upon completion of data replicating
- For chronopolis: Ingest reports PRESERVED
- For DPN: All replications completed
- Mark staging as inactive in Ingest for completed replications in Chronopolis
Release 2.0.3
29 November, 2017
- Query on Bags we created when running Tokenization
Release 2.0.2
9 November, 2017
- Remove creation of /var/log/chronopolis from rpm and instead create it on application startup
Release 2.0.1
8 November, 2017
- Use correct properties when creating Ingest Requests to Chronopolis
Release 2.0.0
30 October, 2017
- Integrate Chronopolis 2.0.0 Changes
- Import tokenization module in order to create ACE Tokens for incoming data
- Use new properties classes for configuration of storage and api communication
- Capture basic metrics when creating bags
- Update ingest flow to create bag and fixity
- Updates to build and init scripts
- Support EL6 and EL7
- Quality of life changes to the sysv init script based on rpmlint
Release 1.5.1
30 June, 2017
- Fix bug that caused replications to be created for each node in DPN
Release 1.5.0
26 June, 2017
- Add a prefix for chronopolis bound collections
- When creating replications for dpn, choose replicating nodes in a deterministic manner per snapshot so that all bags in a snapshot end up at the same DPN nodes.