The Bridge Intake Service provides data pulled down from the Duracloud Bridge App a path into Chronopolis. It currently handles bagging of snapshots, which includes validating files from the manifest given by Duracloud.
Links
Installation
Prereqs
- Running Duracloud Bridge App to connect to
- Staging area for creating bags
Install
- Get the latest rpm from http://adaptci01.umiacs.umd.edu/resource/bridge-intake/master/
- yum install
RPM
Installation can be done through the provided rpm. This offers builds for both RHEL6 and RHEL7.
Installation Notes
Installed Files For RHEL6
/etc/init.d/bridge-intake
/usr/local/chronopolis/intake/dc
/usr/local/chronopolis/intake/dc/application.yml
/usr/local/chronopolis/intake/dc/bridge-intake.jar
Installed Files For RHEL7Installed files are as follows:
/usr/lib/
...
systemd/system/bridge-intake.
...
service /usr/local/chronopolis/intake/dc /usr/local/chronopolis/intake/dc/application.yml
...
/usr/local/chronopolis/intake/dc/bridge-intake.jar /usr/local/chronopolis/intake/dc/bridge-
...
prepare
When running, the service will check for the following directories and create/apply permissions if they do not match:
- System Logging: /var/log/chronopolis/
- logging Journal data: /var/lib/chronopolis/data/journaled data for tokenization
Configuration
...
Depending on where the data is bound, there are several configuration options available
Chronopolis
Prefix
Specifying Replicating Nodes
DPN
...
Code Block | ||||||
---|---|---|---|---|---|---|
| ||||||
# Cron timer for how often the bridge is polled bridge: poll: 0 0 0 * * * # Bagging Configuration bag: max-size: 2 unit: GIGABYTE dpn: node-address: University of California, San Diego, 9500 Gilman Dr, La Jolla, CA 92093 node-contact: Sibyl Schaefer node-email: - sschaefer@ucsd.edu - chronopolis-support-l@mailman.ucsd.edu # General chron configurationGeneral chron configuration ## chron.node: obsolete; ## chron.workDirectory: directory for storing information used for ongoing operations, e.g. file csv ## chron.stage.bags.posix.id: the id of the bag staging area for this service ## chron.stage.bags.posix.path: the path on disk to the bag staging area for this service chron: node: chron bagsworkDirectory: /exporttmp/bagschronopolis tokensstage.bags: /export/tokens preservationposix.id: /data/preservation 4 restorationposix.path: /exportscratch0/restorebags prefix: bridge- ingest: endpoint: http://localhost:8081/ username: ingest-admin password: replace-me replicatingTo: - ucsd - umiacs # DPN Configuration - the server to use when creating replications # and connection information for the registry dpnReplicationServer: dpn-staging.ucsd.edu dpn # Chronopolis Ingest API Configuration ## ingest.api.endpoint: the http endpoint of the Ingest Server ## ingest.api.username: the username to connect to the Ingest Server with ## ingest.api.password: the password to connect to the Ingest Server with ingest.api: endpoint: http://localhost:8000/ username: admin password: admin # ACE IMS Configuration ## ace.ims.endpoint: the fqdn of the ims ## ace.ims.port: the port to connect to the ims on; one of: 80, 443 ## ace.ims: endpoint: http://localhost:3000/ usernameport: chron80 api-keywaitTime: replace-me 6000 # Duracloud Bridge Configuration # The storage areas the bridge app writes in to # Connection information to query the bridge ## duracloud.bridge.name: an identifier for the bridge ## duracloud.bridge.snapshots: the path on disk to the snapshots directory ## duracloud.bridge.restores: the path on disk to the restores directory ## duracloud.bridge.username: the username to use when querying the bridge ## duracloud.bridge.password: the password to use when querying the bridge ## duracloud.bridge.password: the http endpoint of the bridge duracloud: bridge: - name: bridge-0 snapshots: /exportbridge-0/snapshots/ restores: /exportbridge-0/restore/ bridge: username: bridgereplace-me password: replace-me endpoint: http://localhost:8080/ # Push settings to decide what networks to push the snapshots into pushDPN: false pushChronopolis: false - name: bridge-1 snapshots: /bridge-1/snapshots/ restores: /bridge-1/restore/ username: replace-me password: replace-me endpoint: http://localhost:8081/ # Logging configuration logging: file: /var/log/bridgeintake/intake.log level: org.springframework: ERROR org.hibernate: ERROR org.chronopolis: debug org.chronopolis.intake.duracloud.config: trace # Extraneous settings # automatic cleaning of staging areas (not well tested) # only perform dry-runs when the Cleaner runs # disable SNI on https connections - false is the default and recommended value cleanerEnableddisableSNI: false cleanDryRun: false disableSNI: false |
...
Release Notes
Release 3.0.1
09 April, 2019
- Bug Fix: Resolve error with startup failing to configure our beans
Release 3.0.0
21 March, 2019
- Remove deprecated DPN workflow
- Update .gitlab-ci.yml to cache maven artifacts
Release 2.4.3
21 March, 2019
- Update Chronopolis and chron-test dependencies to latest
Release 2.4.2
23 January, 2019
- Bump Chronopolis dependency version
- Bug Fix: Fix csv upload to create the MultiPart data correctly
Release 2.4.1
22 January, 2019
- Bump Chronopolis dependency version
Release 2.4.0
11 January, 2019
- Bridge consolidation work to bring multiple bridge querying to a single service
- Update notification of missing Depositors to fire only once for the lifetime of the bridge
- Fix Bug resulting in successive snapshot runs not being processed if a check against the Depositor fails
- Update ChronopolisIngest for 3.0.0 workflow
Release 2.3.2
09 August, 2018
- Fixed bug which was preventing deserialization of certain classes
- Temporarily disabled HttpFilter because of bug in Ingest Server
Release 2.3.1
07 August, 2018
- Updated logic for appending a prefix to a depositor so that it is used in every operation
Release 2.3.0
10 July, 2018
- Code licensed under BSD-3
- Use separate thread pools when executing Ingestion tasks
- Long IO operations - bagging, cleaning, etc
- Short IO operations - HTTP requests
- Bring tokenization process up to date with Chronopolis Core code
- Test that a depositor exists before attempting to push a Bag to Chronopolis or DPN
Release 2.2.1
18 May, 2018
- Fix bug where constraint for bag size was checked incorrectly
- Fix bug where tokens were not being created because a dependency was missing
Release 2.2.0
14 May, 2018
- Fix bug where we could potentially create the wrong number of replications
- Add constraint satisfaction when creating DPN Replications
- Refactor code for Chronopolis and DPN Ingest to better facilitate working on snapshots concurrently
- Break up DPN Ingestion steps into multiple classes
- Use standard lib interfaces instead of spring-batch for tasks
Release 2.1.0
8 February, 2018
...