...
| Excerpt |
|---|
Large files can be uploaded via the REST API, or projected into the repository using filesystem federation. Transfer times for uploading to the repository via the REST API are about the same as copying using NFS, and moderately faster than using SCP. Uploading via the REST API to a federated filesystem works, but is significantly slower and requires a large temp directory capacity. |
Java command-line options and System properties can be used
...
| Objects | Datastream Size | Projection Directory | Projected Node Request Duration | Download | Download Throughput |
|---|---|---|---|---|---|
| 1 | 1 GB | 417 ms | 35 ms | 17,333 ms | 59.08 MB/sec |
| 1 | 2 GB | 528 ms | 219 ms | 26,902 ms | 76.13 MB/sec |
| 1 | 4 GB | 432 ms | 54 ms | 47,581 ms | 86.08 MB/sec |
| 1 | 8 GB | 583 ms | 90 ms | 90,705 ms | 90.31 MB/sec |
| 1 | 16 GB | 691 ms | 452 ms | 176,508 ms | 92.82 MB/sec |
| 1 | 32 GB | 445 ms | 34 ms | 348,488 ms | 94.03 MB/sec |
| 1 | 64 GB | 750 ms | 460 ms | 699,937 ms | 93.63 MB/sec |
| 1 | 128 GB | 800 ms | 90 ms | 1,412,640 ms | 92.79 MB/sec |
| 1 | 256 GB | 530 ms | 70 ms | 2,768,570 ms | 94.69 MB/sec |
| 1 | 512 GB | 490 ms | 80 ms | 5,893,420 ms | 88.96 MB/sec |
| 1 | 1 TB | 420 ms | 40 ms | 11,322,330 ms | 92.61 MB/sec |
Related articles
...
Direct Comparison of Different Transfer Methods
Based on the tests below, we believe arbitrarily-large files can be uploaded and downloaded via the REST API, using either repository storage or a federated filesystem (tested up to 1TB). The only apparent limitations are disk space available to store the files, temp directory capacity, and a sufficiently large Java heap size.
- Platform: lib-devsandbox1.ucsd.edu (all data on NAS to handle large files)
- Repository Profile: Federation
- Workflow Profile: Repository/Federation/NFS/SCP Comparison
Comparison of Upload and Download Times for Different Transfer Methods
| Transfer Method | File Size | Upload | Download |
|---|---|---|---|
| REST API (Federated) | 1TB | 732 min (84 GB/sec) | 246 min (250 GB/sec) |
| REST API (Repository) | 1TB | 339 min (181 GB/sec) | 250 min (246 GB/sec) |
| SCP | 1TB | 383 min (160 GB/sec) | |
| NFS | 1TB | 336 min (183 GB/sec) |
Copying Files Between Federated Filesystem and Repository Storage
| Source | Destination | File Size | Copy Time |
|---|---|---|---|
| Repository storage | Federated filesystem | 1TB | 402 min (153 GB/sec) |
| Federated filesystem | Repository storage | 1TB | 345 min (178 GB/sec) |
...