...
Under a single node, batches of 500 objects were created, each with a single 10KB datastream. After each batch, the entire directory contents were listed using the REST API ("whole"). The test was then repeated using the limit parameter to list the first 1,000 objects ("paged"). Listing the objects scales roughly linearly in both cases, but much more steeply for listing all the children. Using paging, acceptance is acceptable up to 30,000 children with no sharp increase observed.
5 million files in a 3-level hierarchy
4.85 million files were ingested using a three-level hierarchy (74 top-level nodes, 256 second-level nodes in each, 256 third-level nodes in each, and one 10KB datastream in each), taking 111 hours. After each batch, three REST API operations were timed: listing the top level of the repository ("toplist"), listing a second-level node ("dirlist"), and retrieving a file ("fileget"). Performance retrieving files and listing the second-level nodes did not degrade with larger numbers of objects. However, listing the top-level of the repository degraded roughly linearly as more objects were added, and became increasing erratic.
Federated filesystem
Files in a single directory
...