The Ingest service adds a layer between the Intake and Replication services. The main goal is to validate data which comes from bags and create ACE tokens for that data.
Installed files are as follows
[~] $ rpm -qlp ingest-server-2.0.0-20171027.el6.noarch.rpm /etc/init.d/ingest-server /usr/local/chronopolis/ingest /usr/local/chronopolis/ingest/application.yml /usr/local/chronopolis/ingest/ingest-server.jar /var/log/chronopolis |
[~] $ rpm -qlp ingest-server-2.0.0-20171027.el7.noarch.rpm /etc/init.d/ingest-server /usr/local/chronopolis/ingest /usr/local/chronopolis/ingest/application.yml /usr/local/chronopolis/ingest/ingest-server.jar /var/log/chronopolis |
A 'chronopolis' user is also needed which can write to /var/log/chronopolis and perform various read and write tasks as needed from the Token Staging Area. This is no longer installed as part of the rpm, but should be managed separately and configured in the ingest-server startup script.
Download the schema from the CI server
psql (8.4.20) Type "help" for help. postgres=# CREATE USER chron WITH PASSWORD 'secret-password'; CREATE ROLE postgres=# CREATE DATABASE ingest; CREATE DATABASE postgres=# GRANT ALL PRIVILEGES ON DATABASE ingest to chron; GRANT postgres=# \c ingest; psql (8.4.20) You are now connected to database "ingest". ingest=# SET ROLE chron; SET ingest=> \i /tmp/schema_pg.sql |
As of version 1.1.0, the database has a schema_version table for handling schema migrations. This is managed automatically through flyway, so that the server can be upgraded without needing to worry about manually applying patches. Flyway provides a jar file which we can use to prepare the database for migrations, something which can be applied to previous versions as well.
specify the version which you are creating the baseline (using the MAJOR.MINOR number of the ingest server version)
# # Copyright 2010-2015 Axel Fontaine # ... # Jdbc url to use to connect to the database flyway.url=jdbc:postgresql://localhost/ingest # Fully qualified classname of the jdbc driver (autodetected by default based on flyway.url) # flyway.driver= # User to use to connect to the database (default: <<null>>) flyway.user=chron # Password to use to connect to the database (default: <<null>>) flyway.password=my-postgresql-password ... flyway.baselineVersion=1.5 |
Use the flyway bash script to update the database
$ ./flyway baseline |
This information is now outdated; a new method needs to be created for bootstrapping data with encrypted passwords.
There's also a sql script which contains entries for the nodes and users in chronopolis as well as their roles. The script is found on jenkins, and can be loaded with `\i /path/to/data.sql`. If this is not used, users and nodes will need to be created manually - an admin user will be created on startup with a default password of 'admin' if no users are found. Users can then be created through the web ui.
The ingest server reads from the /usr/local/chronopolis/ingest/application.yml configuration file:
# Ingest Configuration Properties # Ingest Cron job properties # tokens: the rate at which to check for bags which have all tokens and can have a Token Store written # request: the rate at which to check for bags which need their initial replications created # tokenize: the rate at which to check for local bags which need tokens created ingest.cron: tokens: 0 0/10 * * * * request: 0 0/10 * * * * tokenize: 0 0 * * * * * # Ingest AJP Settings # enabled: flag to enable an AJP connector # port: the port for the connector to listen on ingest.ajp: enabled: false port: 8009 # The staging area for writing Token Stores. Nonposix support not yet implemented. ## id: The id of the StagingStorage in the Ingest server ## path: The path to the filesystem on disk chron.stage.tokens.posix.id: -1 chron.stage.tokens.posix.path: /dev/null # If Local Tokenization is desired, include staging information for Bags and ACE IMS connection information ## id: The id of the StagingStorage which the Ingest Server will read from ## path: The path to the filesystem on disk chron.stage.bags.posix.id: -1 chron.stage.bags.posix.path: /dev/null ## port: the port to connect to the ims with ## waitTime: the time to wait between token requests ## endpoint: the fqdn of the ims ## queueLength: the maximum number of requests to send at once ace.ims: port: 80 waitTime: 5000 endpoint: ims.umiacs.umd.edu queueLength: 1000 # Database connection # Initialize should be kept false so that the server does not attempt to run a drop/create on the tables spring.datasource: url: jdbc:postgresql://localhost/ingest username: postgres password: dbpass initialize: false # Specify the active profile for configuring services as a comma separated list # production - remove stdout/stderr from printing and run without accepting input # disable-tokenizer - disable local tokenization services from running spring.profiles.active: production spring.pid.file: /var/run/ingest-server.pid # debug: true server.port: 8080 # Logging properties logging.file: ingest.log logging.level: org.springframework: ERROR org.hibernate: ERROR org.chronopolis: DEBUG |
The ingest server runs as an executable jar. Using the init script allows for starting and stopping of the server as root: `service ingest-server start`
As of version 1.4.0, passwords for users are now encoded using bcrypt. In the event a user forgets their password, we will need to reset it for them. As we do not have email notifications or anything of the like setup, for the moment everything will need to be done manually. We will first need to run the password through an encoder, which can be found online. If you aren't sure how many rounds to use, check the database as the information is kept as part of the encoding, i.e. $2a$08 uses 8 rounds; $2a$10 uses 10 rounds.
Then we connect to the database and issue a simple update:
UPDATE users SET password = '$2a$10$hEYYHV/Fri00RRHjWPIAWuH3NxYpPPjbMU5OsJfH1SAenajQqKjhK' WHERE username = 'user_resetting_password'; |
With the release of version 2.0.0, StorageRegions have been introduced in order to facilitate distribution of content from many nodes in Chronopolis. Configuration for them is as follows: