The Bridge Intake Service provides data pulled down from the Duracloud Bridge App a path into Chronopolis. It currently handles bagging of snapshots, which includes validating files from the manifest given by Duracloud.
Links
Installation
Prereqs
- Running Duracloud Bridge App to connect to
- Staging area for creating bags
RPM
Installation can be done through the provided rpm. This offers builds for both RHEL6 and RHEL7.
Installation Notes
Installed Files For RHEL6
/etc/init.d/bridge-intake
/usr/local/chronopolis/intake/dc
/usr/local/chronopolis/intake/dc/application.yml
/usr/local/chronopolis/intake/dc/bridge-intake.jar
Installed Files For RHEL7
/usr/lib/systemd/system/bridge-intake.service /usr/local/chronopolis/intake/dc /usr/local/chronopolis/intake/dc/application.yml /usr/local/chronopolis/intake/dc/bridge-intake.jar /usr/local/chronopolis/intake/dc/bridge-prepare
When running, the service will check for the following directories and create/apply permissions if they do not match:
- System Logging: /var/log/chronopolis/
- Journal data: /var/lib/chronopolis/data/
Configuration
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
# Cron timer for how often the bridge is polled bridge: poll: 0 0 0 * * * # General chron configuration ## chron.node: obsolete; ## chron.workDirectory: directory for storing information used for ongoing operations, e.g. file csv ## chron.stage.bags.posix.id: the id of the bag staging area for this service ## chron.stage.bags.posix.path: the path on disk to the bag staging area for this service chron: node: chron workDirectory: /tmp/chronopolis stage.bags: posix.id: 4 posix.path: /scratch0/bags # Chronopolis Ingest API Configuration ## ingest.api.endpoint: the http endpoint of the Ingest Server ## ingest.api.username: the username to connect to the Ingest Server with ## ingest.api.password: the password to connect to the Ingest Server with ingest.api: endpoint: http://localhost:8000/ username: admin password: admin # ACE IMS Configuration ## ace.ims.endpoint: the fqdn of the ims ## ace.ims.port: the port to connect to the ims on; one of: 80, 443 ## ace.ims: endpoint: localhost port: 80 waitTime: 6000 # Duracloud Bridge Configuration # The storage areas the bridge app writes in to # Connection information to query the bridge ## duracloud.bridge.name: an identifier for the bridge ## duracloud.bridge.snapshots: the path on disk to the snapshots directory ## duracloud.bridge.restores: the path on disk to the restores directory ## duracloud.bridge.username: the username to use when querying the bridge ## duracloud.bridge.password: the password to use when querying the bridge ## duracloud.bridge.password: the http endpoint of the bridge duracloud: bridge: - name: bridge-0 snapshots: /bridge-0/snapshots/ restores: /bridge-0/restore/ username: replace-me password: replace-me endpoint: http://localhost:8080/ - name: bridge-1 snapshots: /bridge-1/snapshots/ restores: /bridge-1/restore/ username: replace-me password: replace-me endpoint: http://localhost:8081/ # Logging configuration logging: file: /var/log/bridgeintake/intake.log level: org.springframework: ERROR org.hibernate: ERROR org.chronopolis: debug org.chronopolis.intake.duracloud.config: trace # Extraneous settings # disable SNI on https connections - false is the default and recommended value disableSNI: false |
Release Notes
Release 3.0.1
09 April, 2019
- Bug Fix: Resolve error with startup failing to configure our beans
Release 3.0.0
21 March, 2019
- Remove deprecated DPN workflow
- Update .gitlab-ci.yml to cache maven artifacts
Release 2.4.3
21 March, 2019
- Update Chronopolis and chron-test dependencies to latest
Release 2.4.2
23 January, 2019
- Bump Chronopolis dependency version
- Bug Fix: Fix csv upload to create the MultiPart data correctly
Release 2.4.1
22 January, 2019
- Bump Chronopolis dependency version
Release 2.4.0
11 January, 2019
- Bridge consolidation work to bring multiple bridge querying to a single service
- Update notification of missing Depositors to fire only once for the lifetime of the bridge
- Fix Bug resulting in successive snapshot runs not being processed if a check against the Depositor fails
- Update ChronopolisIngest for 3.0.0 workflow
Release 2.3.2
09 August, 2018
- Fixed bug which was preventing deserialization of certain classes
- Temporarily disabled HttpFilter because of bug in Ingest Server
Release 2.3.1
07 August, 2018
- Updated logic for appending a prefix to a depositor so that it is used in every operation
Release 2.3.0
10 July, 2018
- Code licensed under BSD-3
- Use separate thread pools when executing Ingestion tasks
- Long IO operations - bagging, cleaning, etc
- Short IO operations - HTTP requests
- Bring tokenization process up to date with Chronopolis Core code
- Test that a depositor exists before attempting to push a Bag to Chronopolis or DPN
Release 2.2.1
18 May, 2018
- Fix bug where constraint for bag size was checked incorrectly
- Fix bug where tokens were not being created because a dependency was missing
Release 2.2.0
14 May, 2018
- Fix bug where we could potentially create the wrong number of replications
- Add constraint satisfaction when creating DPN Replications
- Refactor code for Chronopolis and DPN Ingest to better facilitate working on snapshots concurrently
- Break up DPN Ingestion steps into multiple classes
- Use standard lib interfaces instead of spring-batch for tasks
Release 2.1.0
8 February, 2018
- Actively clean storage upon completion of data replicating
- For chronopolis: Ingest reports PRESERVED
- For DPN: All replications completed
- Mark staging as inactive in Ingest for completed replications in Chronopolis
Release 2.0.3
29 November, 2017
- Query on Bags we created when running Tokenization
Release 2.0.2
9 November, 2017
- Remove creation of /var/log/chronopolis from rpm and instead create it on application startup
Release 2.0.1
8 November, 2017
- Use correct properties when creating Ingest Requests to Chronopolis
Release 2.0.0
30 October, 2017
- Integrate Chronopolis 2.0.0 Changes
- Import tokenization module in order to create ACE Tokens for incoming data
- Use new properties classes for configuration of storage and api communication
- Capture basic metrics when creating bags
- Update ingest flow to create bag and fixity
- Updates to build and init scripts
- Support EL6 and EL7
- Quality of life changes to the sysv init script based on rpmlint
Release 1.5.1
30 June, 2017
- Fix bug that caused replications to be created for each node in DPN
Release 1.5.0
26 June, 2017
- Add a prefix for chronopolis bound collections
- When creating replications for dpn, choose replicating nodes in a deterministic manner per snapshot so that all bags in a snapshot end up at the same DPN nodes.
...