Page History

...

Within the standard operating of Chronopolis, it is likely due to the volume of data we ingest that we will at some point need to repair data held at a node. In the event a node cannot repair their own data, a process will be in place so that the data can be repaired through the Chronopolis network. In this document a basic design proposal for a protocol through which we can repair collections in a combination of manual and automated work will be outlined.

...

As this design is still living, there are still open questions as to how everything should be finalized and what impact they will have on the final result.
 

1. What transfer strategy should we use?Transfer Strategies

• Multiple types of transfers are allowed, however each will need to be implemented.
  • Node to Node: Transfer between replicating nodes using rsync + ssh with no intermediary step
  • Node to Ingest: Push content to the Ingest node from which a node can repair from
  • ACE: Use ACE with https as the transfer mechanism for serving files

2. Should we create a new client for handling repair, or should it be merged in with the replication service?

• If it’s a new client, what type of application would be best (is cli good enough? do we want a gui? maybe some integration with the ingest server instead?)
   

3. Should we put a limit on the number of files being repaired in a single request? 

• At the moment this is unbounded, but we may want to look into it in the future

34. Should we include tokens in this process, but leave implementation out for now?

• Initial version will only handle files, tokens can be added on later

Repair Flow

Basic flow: node_i = invalid; node_v = valid1.

1. node_i sees invalid files in ACE_i

...

1. node_i gathers invalid files and issues a repair request to the ingest server
   

...

1. 1. POST /api/repair
   2. Handled manually
   3. Consider

...

1. 1. having multiple requests in the event many files are corrupt

...

1. node_v sees the repair request

   ...

   1. 1. Handled manually, likely from discussion in the chron group
   2. node_v checks ACE_v to see if

   ...

   1. the files are

   ...

   1. valid

   ...

   1. ...

      1. 1. POST /api/repair/<id>/fulfill if valid

      ...

      1. node_v stages content for node_i
         

      ...

      1. 1. P2P: make a link (or links) to the files in a

      ...

      1. 1. directory for node_i
            

      ...

      1. 1. Ingest: rsync the files up to the ingest server

      ...

      1. 1. ACE: create a token for node_i and make that available
      2. node_v notifies ingest server that content is ready for node_i
         1. POST /api/repair/fulfillment/<id>/ready

      ...

      1. node_i replicates staged content
         1. GET /api/repair/fulfillment?to=node_i&status=ready

      ...

      1. node_i validates staged content
      2. node_i copies staged content to preservation storage
      3. node_i issues an audit of the corrupt files

      ...

      1. node_i responds with the result of the audit
         1. if the audit is not successful a new replication request will need to be made, but we might want to do that by hand
         2. POST /api/repair/fulfillment/<id>/complete

      ...

      Turning this into a graph might be useful

      Transfer Strategies

      Node to Node

      ...

      3. nodei downloads from ACEv using the generated API key

      
      

      API Design - Move to Sub Page

      The API can be viewed with additional formatting and examples at

      ...

      PUT /api/repair/fulfillemnts/<id>/complete

      
      

      Models - Move to Sub Page

      A repair request, sent out by a node who notices they have corrupt files in a collection

      ...