Page tree

Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

The Retrieval Tool is a utility which is used to transfer (or "retrieve") digital content from DuraCloud to your local file system. It uses the command line (also called the terminal) on your local system to access your DuraCloud account and transfer the content you specify to a local file location of your choice. The content you retrieve also remains in DuraCloud; no content is deleted.

Familiarity with the command line will help you in using the retrieval tool:

Download

Download the retrieval tool from the Downloads page.

Quick Start

For the impatient: Here are the commands you would use to complete the following common tasks. See the sections before for more details about how the Retrieval Tool operates and more information about available options.

You will need to replace the sections in { } !! 

Download all files in a space

Code Block
java -jar retrievaltool-{version}-driver.jar -h {your-duracloud-subdomain}.duracloud.org -u {your-username} -p {your-password} -s {name-of-the-space-to-download} -c {name-of-local-directory-to-place-content}

Download a single file

First: Create a file, named content-list.txt, and place it next to the Retrieval Tool jar file. Open this file in a text editor and add to the first line the full content ID of the one file you want to download. You can find the content ID by logging into the DuraCloud UI, selecting the space. Each of the files that are listed as being in the space will be displayed with their full content ID. If your file has been chunked, remove the chunk extension (e.g. ".dura-manifest" or ".dura-chunk-0001") to get the content ID for use with the Retrieval Tool.

Code Block
java -jar retrievaltool-{version}-driver.jar -h {your-duracloud-subdomain}.duracloud.org -u {your-username} -p {your-password} -s {name-of-the-space-where-the-file-is-stored} -c {name-of-local-directory-to-place-content} --list-file content-list.txt

Download a subset of files in a space

This is a two-part action.

First: Retrieve the list of all content items in the space:

Code Block
java -jar retrievaltool-{version}-driver.jar -h {your-duracloud-subdomain}.duracloud.org -u {your-username} -p {your-password} -s {name-of-the-space-to-list} -c {name-of-local-directory-to-place-list} --list-only

Second: Copy the list file created in the first step and paste it next to the retrieval tool, then open it and delete all content IDs in the list that you DO NOT want to download. When you are done you will have a list that contains only the files that you do want downloaded. Then start the download:

Code Block
java -jar retrievaltool-{version}-driver.jar -h {your-duracloud-subdomain}.duracloud.org -u {your-username} -p {your-password} -s {name-of-the-space-where-files-are-stored} -c {name-of-local-directory-to-place-content} --list-file {name-of-the-list-file}


How the Retrieval Tool Works

...

  • To run the Retrieval Tool, open a terminal or command prompt and navigate to the directory where the Retrieval Tool jar file is located
  • To display the help for the Retrieval Tool, run

    Code Block
    java -jar retrievaltool-{version}-driver.jar


  • When running the Retrieval Tool, you will need to use these options:

    Short Option

    Long Option

    Argument Expected

    Required

    Description

    Default Value (if optional)

    -h

    --host

    Yes

    Yes

    The host address of the DuraCloud DuraStore application

     


    -r

    --port

    Yes

    No

    The port of the DuraCloud DuraStore application

    443

    -u

    --username

    Yes

    Yes

    The username necessary to perform writes to DuraStore

     


    -p

    --password

    Yes

    No

    The password necessary to perform writes to DuraStore. If not specified the retrieval tool will first check to see if an environment variable named "DURACLOUD_PASSWORD" exists, if it does exist the retrieval tool will use its value as the password, otherwise you will be prompted to enter the password.

    Not set

    -i

    --store-id

    Yes

    No

    The Store ID for the DuraCloud storage provider

    The default store is used

    -s

    --spaces

    Yes

    No

    The space or spaces from which content will be retrieved. Either this option or -a must be included

     


    -a

    --all-spaces

    No

    No

    Indicates that all spaces should be retrieved; if this option is included the -s option is ignored

    Not set

    -c

    --content-dir

    Yes

    Yes

    Retrieved content is stored in this local directory

     


    -w

    --work-dir

    Yes

    No

    Logs and output files will be stored in the work directory.  If not specified, this value will default to a directory named duracloud-retrieval-work in the user's home directory.

    duracloud-retrieval-work

    -o

    --overwrite

    No

    No

    Indicates that existing local files which differ from files in DuraCloud under the same path and name sould be overwritten rather than copied

    Not set

    -t

    --threads

    Yes

    No

    The number of threads in the pool used to manage file transfers

    3

    -d--disable-timestampsNoNo

    Indicates that timestamp information found as content item properties in DuraCloud should not be applied to local files as they are retrieved.

    Not set
    -l

    --list-only

    NoNo

    Indicates that the retrieval tool should create a file listing the contents of the specified space rather than downloading the actual content files. The list file will be placed in the specified content directory. One list file will be created for each specified space.

    Not set
    -f

    --list-file

    YesNo

    Retrieve specific contents using content IDs in the specified file. The specified file should contain one content ID per line. This option can only operate on one space at a time.

    Not set


Examples of running the retrieval tool:

  • Retrieve all the files stored within the 2 specified spaces and place them in the specified local content directory under sub-directories matching the specified 2 space names.

    Code Block
    java -jar retrievaltool-{version}-driver.jar -c content -h test.duracloud.org -u myname -p mypassword -s space1 space2 -o


  • Retrieve all the files stored within all spaces and place them in the specified local content directory under sub-directories matching the space names.

    Code Block
    java -jar retrievaltool-{version}-driver.jar -c content -h test.duracloud.org -u myname -p mypassword -a


  • Create a file containing the list of content IDs for the specified spaces using hidden password option (-p command line option not specified, will be prompted for password).  This example would not actually retrieve the content files, rather it creates a list of content files in the specified space.  Each specified space will have its own content list file created in the specified local content directory.  The naming convention of each list file created will be: "<space_id>-content-listing-<storage_provider>.txt"

    Code Block
    java -jar retrievaltool-{version}-driver.jar -h <host> -u <user> -c <content_dir> -s <list of space IDs separated by a space> -l


  • Retrieve only the specified contents by using the list-file option (-f).  The -f option can only operate on one space.  This command would result in having all the content files listed in the specified file of content IDs placed in the specified local content directory.

    Code Block
    java -jar retrievaltool-{version}-driver.jar -h <host> -u <user> -c <content_dir> -f <path_to_file_of_specified_content_IDs> -s <single space ID>