The Bridge Intake Service provides data pulled down from the Duracloud Bridge App a path into Chronopolis. It currently handles bagging of snapshots, which includes validating files from the manifest given by Duracloud.
Links
Installation
Prereqs
- Running Duracloud Bridge App to connect to
- Staging area for creating bags
Install
- Get the latest rpm from http://adaptci01.umiacs.umd.edu/resource/bridge-intake/master/
- yum install
RPM
Installation can be done through the provided rpm. This offers builds for both RHEL6 and RHEL7.
Installation Notes
Installed Files For RHEL6
/etc/init.d/bridge-intake
/usr/local/chronopolis/intake/dc
/usr/local/chronopolis/intake/dc/application.yml
/usr/local/chronopolis/intake/dc/bridge-intake.jar
Installed Files For RHEL7Installed files are as follows:
/usr/lib/systemd/
...
system/bridge-intake.
...
service /usr/local/chronopolis/intake/dc /usr/local/chronopolis/intake/dc/application.yml
...
/usr/local/chronopolis/intake/dc/bridge-intake.jar /usr/local/chronopolis/intake/dc/bridge-
...
Configuration
Depending on where the data is bound, there are several configuration options available
Chronopolis
Prefix
Specifying Replicating Nodes
DPN
...
prepare
When running, the service will check for the following directories and create/apply permissions if they do not match:
- System Logging: /var/log/chronopolis/
- Journal data: /var/lib/chronopolis/data/
Configuration
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
# Cron timer for how often the bridge is polled bridge: poll: 0 0 0 * * * # General chron configuration ## chron.node: obsolete; ## node: chron bags: /export/bags tokens: /export/tokens preservation: /data/preservation restoration: /export/restore prefix: bridge- ingest: endpoint: http://localhost:8081/ username: ingest-admin password: replace-me replicatingTo: - ucsd - umiacs # DPN Configuration - the server to use when creating replications # and connection information for the registry dpnReplicationServer: dpn-staging.ucsd.edu dpnchron.workDirectory: directory for storing information used for ongoing operations, e.g. file csv ## chron.stage.bags.posix.id: the id of the bag staging area for this service ## chron.stage.bags.posix.path: the path on disk to the bag staging area for this service chron: node: chron workDirectory: /tmp/chronopolis stage.bags: posix.id: 4 posix.path: /scratch0/bags # Chronopolis Ingest API Configuration ## ingest.api.endpoint: the http endpoint of the Ingest Server ## ingest.api.username: the username to connect to the Ingest Server with ## ingest.api.password: the password to connect to the Ingest Server with ingest.api: endpoint: http://localhost:8000/ username: admin password: admin # ACE IMS Configuration ## ace.ims.endpoint: the fqdn of the ims ## ace.ims.port: the port to connect to the ims on; one of: 80, 443 ## ace.ims: endpoint: http://localhost:3000/ usernameport: chron80 api-keywaitTime: replace-me 6000 # Duracloud Bridge Configuration # The storage areas the bridge app writes in to # Connection information to query the bridge ## duracloud.bridge.name: an identifier for the bridge ## duracloud.bridge.snapshots: the path on disk to the snapshots directory ## duracloud.bridge.restores: the path on disk to the restores directory ## duracloud.bridge.username: the username to use when querying the bridge ## duracloud.bridge.password: the password to use when querying the bridge ## duracloud.bridge.password: the http endpoint of the bridge duracloud: bridge: - name: bridge-0 snapshots: /exportbridge-0/snapshots/ restores: /exportbridge-0/restore/ bridge: username: bridgereplace-me password: replace-me endpoint: http://localhost:8080/ - name: bridge-1 # Push settings to decide what networks to push the snapshots into pushDPN: false pushChronopolis: false snapshots: /bridge-1/snapshots/ restores: /bridge-1/restore/ username: replace-me password: replace-me endpoint: http://localhost:8081/ # Logging configuration logging: file: /var/log/bridgeintake/intake.log level: org.springframework: ERROR org.hibernate: ERROR org.chronopolis: debug org.chronopolis.intake.duracloud.config: trace # Extraneous settings # automatic cleaning of staging areas (not well tested) # only perform dry-runs when the Cleaner runs # disable SNI on https connections - false is the default and recommended value cleanerEnableddisableSNI: false cleanDryRun: false disableSNI: false |
...
Release Notes
Release 3.0.1
09 April, 2019
- Bug Fix: Resolve error with startup failing to configure our beans
Release 3.0.0
21 March, 2019
- Remove deprecated DPN workflow
- Update .gitlab-ci.yml to cache maven artifacts
Release 2.4.3
21 March, 2019
- Update Chronopolis and chron-test dependencies to latest
Release 2.4.2
23 January, 2019
- Bump Chronopolis dependency version
- Bug Fix: Fix csv upload to create the MultiPart data correctly
Release 2.4.1
22 January, 2019
- Bump Chronopolis dependency version
Release 2.4.0
11 January, 2019
- Bridge consolidation work to bring multiple bridge querying to a single service
- Update notification of missing Depositors to fire only once for the lifetime of the bridge
- Fix Bug resulting in successive snapshot runs not being processed if a check against the Depositor fails
- Update ChronopolisIngest for 3.0.0 workflow
Release 2.3.2
09 August, 2018
- Fixed bug which was preventing deserialization of certain classes
- Temporarily disabled HttpFilter because of bug in Ingest Server
Release 2.3.1
07 August, 2018
- Updated logic for appending a prefix to a depositor so that it is used in every operation
Release 2.3.0
10 July, 2018
- Code licensed under BSD-3
- Use separate thread pools when executing Ingestion tasks
- Long IO operations - bagging, cleaning, etc
- Short IO operations - HTTP requests
- Bring tokenization process up to date with Chronopolis Core code
- Test that a depositor exists before attempting to push a Bag to Chronopolis or DPN
Release 2.2.1
18 May, 2018
- Fix bug where constraint for bag size was checked incorrectly
- Fix bug where tokens were not being created because a dependency was missing
Release 2.2.0
14 May, 2018
- Fix bug where we could potentially create the wrong number of replications
- Add constraint satisfaction when creating DPN Replications
- Refactor code for Chronopolis and DPN Ingest to better facilitate working on snapshots concurrently
- Break up DPN Ingestion steps into multiple classes
- Use standard lib interfaces instead of spring-batch for tasks
Release 2.1.0
8 February, 2018
- Actively clean storage upon completion of data replicating
- For chronopolis: Ingest reports PRESERVED
- For DPN: All replications completed
- Mark staging as inactive in Ingest for completed replications in Chronopolis
Release 2.0.3
29 November, 2017
...