There are a number of situations where you might want your Circulation Manager to send large chunks of data to an external storage service:

• Exporting MARC records from the Circulation Manager for import by another system, such as a library ILS.
• Making local copies of cover images discovered while importing data from an ebook vendor.
• Making local copies of freely-distributable books discovered while importing someone else’s OPDS feed.

The Circulation Manager supports all three of these cases through a flexible system that lets you configure different storage integrations for different purposes. Although you can set up any number of storage integrations, currently the only supported type of storage integration is an Amazon S3 account.

Whats in this Article

Creating a Storage Service

Whether you want to export MARC records or make local copies of cover images, the first step is to create a new storage service. Under the `Systems Configuration` heading navigation, select `Storage` in the left sidebar.

In the Circulation Manager Admin Interface got to System Configuration and select "Storage" from the configuration menu.

Here you’ll have the option to create a new Storage service or edit an existing service. Configuration works basically the same way regardless of what you kind of data you intend to send to a storage service, but it’s helpful to have a human-friendly name so you can remember what a given Storage service is being used for.

EXAMPLE

In the example above, we have four Storage services:

• two are intended for MARC files
• one for books
• one for image covers

Exporting MARC Records

First, use AWS tools to create an S3 bucket to contain the MARC files. (See above note for links to S3 Documentation)

In the Circulation Manager, create a new storage service and configure it using your S3 access key and secret. Put the name of your newly created S3 bucket into the optional field `MARC File Bucket`.

Here’s a configured storage service called “marc storage” which will use the credentials “access_key” and “secret_key” to upload files to the bucket called “marc1bucket”.

Click on the arrow to open the Required Fields form

Next, Click on the arrow to open the Option Fields Form

Next, you’ll need to set up an `External Catalog` service using the protocol “MARC Export”. This is how the Circulation Manager thinks about your ILS: an external service with which it never interacts directly, but for which it’s supposed to regularly generate MARC files.

Here’s a configured External Catalog service:

An External Catalog service has an optional field called “MARC Mirror”. Set this to point to the Storage service you created in the previous step.

Even though we’ve created four Storage services in this example (“marc storage”, “second marc storage”, “cover storage”, and “book storage”), the only options in the screenshot above are the two Storage services with a value for “MARC File Bucket”. You can’t associate a Storage service with a MARC export if there’s nowhere for the MARC files to go.

Configuring individual libraries

An External Catalog service can have any number of libraries associated with it. You’ll need to add each library whose MARC records you want to be uploaded to the Storage service. Otherwise the External Catalog service won’t know which libraries need their MARC records generated, and it won’t do anything.

When you add a library to an External Catalog service, you’ll have the opportunity to set some optional configuration fields. You can configure how often MARC records should be regenerated for that particular library, and customize what data should go into the MARC records.

When does the export happen?

A script that runs once a day at 1 AM local time goes through each configured External Catalog service, generates any MARC files that need to be generated, and uploads them to the configured Storage service for that External Catalog service.

This means that the first night after you configure a new External Catalog service, MARC records for each associated library will be generated and uploaded to your Storage service. MARC files for each library will be regenerated as necessary. If you use the default update frequency of 30 days, the script will still run once a day, but twenty-nine days out of thirty it will do nothing.

If you need to run the script manually, you can find it as `bin/cache_marc_files`. Note that this won’t force MARC records to be regenerated. If you want to force MARC records to be regenerated and re-uploaded, pass the `--force` option in to `bin/cache_marc_files`.

MARC Organization code

If your library has a MARC organization code (https://www.loc.gov/marc/organizations/), you may include it to identify your library as the originator of the MARC records.

Web catalog links

If you want patrons to be able to borrow and download from your ILS, you’ll need to configure a base URL for the web catalog. The web catalog is a separate application [documentation link needed]. Once this URL is configured, each MARC record will have an 856 field with a link to a page where a patron can borrow the book.

Other MARC fields

You can optionally configure the exported records to include summaries in the 520 field. You can also include Library Simplified genres in the 650 field. Note that Library Simplified uses a custom set of genres that are optimized for browsing, and we don’t currently support more standard subject headings like LC or BISAC.

Downloading the files

You can download the most recent MARC files generated by your Circulation Manager by taking the base URL for your library and appending `/marc`. This will show you a list of links to the MARC files as hosted in the S3 bucket you configured.

As an example, see https://circulation.librarysimplified.org/NYNYPL/marc on NYPL’s Circulation Manager. (NYPL doesn’t use the MARC Export feature, so there are no MARC files for download here; this is just to show you what the URL looks like.)

[screenshot needed]

The page has two types of MARC files - full exports and incremental exports. The full exports will contain MARC records for every book the library currently has, but the incremental exports will only contain books that were added or updated since the previous export. Separate files are available for different lanes, or for the full library.

Making copies of book covers

You can use a storage integration to keep your own copies of the book covers you encounter, rather than relying on the image hosting services provided by the ebook vendors or by NYPL.

First, use AWS tools to create an S3 bucket to store the cover images.

In the Circulation Manager, create a new storage service and configure it using your S3 access key and secret. Put the name of your newly created S3 bucket into the optional field `Book Covers Bucket`.

Here’s a configured storage service called “Book Covers” which will use the credentials “abcde” and “12345” to upload files to the bucket called “some-covers”.

If you create a Storage service and leave it at that, nothing will happen. To make sure covers are actually uploaded to the Storage service, you need to edit a Collection, open its `Optional Fields`, and associate the collection’s “Covers Mirror” with the storage service.

As cover images are encountered during Collection import, they will be mirrored to the storage service designated in that Collection’s “Covers Mirror”.

Making copies of free books

This process is exactly the same as for book covers: create a Storage service and associate it with a Collection. There are only two differences:

1.  When configuring the Storage service, you put the name of the S3 bucket in “Open Access Content Bucket” rather than “Book Covers Bucket”
2. When attaching the Storage service to a Collection, you attach it to the  Collection’s “Books Mirror” rather than its “Covers Mirror”.

As free books are encountered during Collection import, they will be mirrored to the storage service designated in that Collection’s “Books Mirror”.