Page History

This article describes the necessary steps for configuring and running DuraCloud Mill.   DuraCloud Mill is a collection of applications that work in concert with DuraCloud to perform a variety of bulk processing tasks including audit log writing,  manifest maintenance, duplication and bit integrity checks.   The system has been designed to run in an auto-scalable environment such as AWS EC2,  but it can also be run on as a suite of stand-alone applications on any machine with sufficient power.  This article only describes the various applications and how to configure them; it does not cover approaches to leveraging AWS autoscaling and supporting dev opts tools such as Puppet. If you are not yet familiar with the DuraCloud Mill please refer to the Duracloud Mill Overview.  

...

#############################
# LOOPING DUP TASK PRODUCER
#############################
# The frequency for a complete run through all store policies. Specify in hours (e.g. 3h), days (e.g. 3d), or months (e.g. 3m). Default is 1m - i.e. one month
looping.dup.frequency=[]
# Indicates how large the task queue should be allowed to grow before the Looping Task Producer exits.
looping.dup.max-task-queue-size=[]

Configuring and Running Bit Integrity 

Bit integrity works similarly to the loopingduptaskproducer.  It has similar settings as those mentioned above as well as two others, looping.bit.inclusion-list-file and looping.bit.exclusion-list-file.    These two config files let you be more surgical in what you decide to include and exclude from your bit integrity run and function similarly to the duplication policies.    The important thing to note here is that if there are not entries in the inclusion list all accounts, stores, and spaces are included.  

#############################
# LOOPING BIT TASK PRODUCER
#############################
# The frequency for a complete run through all store policies. Specify in hours (e.g. 3h), days (e.g. 3d), or months (e.g. 3m). Default is 1m - i.e. one month
looping.bit.frequency=[]
# Indicates how large the task queue should be allowed to grow before the Looping Task Producer quits.
looping.bit.max-task-queue-size=[]
# A file containing inclusions. Expressions will be matched against the following path: /{account}/{storeId}/{spaceId} and should have the same format. You can use an asterix (*) to indicate all.
# For example,  to indicate all spaces name "space-a"  in the "test" account across all providers you would add a line like this: 
#    /test/*/space-a
# You may also comment lines by using a hash (#) at the beginning of the line.
looping.bit.inclusion-list-file=[]
# A file containing exclusions as regular expressions using the same format as specified for inclusions.
looping.bit.exclusion-list-file=[]

Related articles

...