...
File retrieval performs very well, and seems to be unaffected by the number of files and the hierarchy used. The performance of using the REST API to list the contents of a directory on a federated filesystem depends on the number of children and grandchildren nodes. So a 3-level hierarchy with 100-128 nodes at each level should allow a federated filesystem with 1-2 million files to perform well, but larger sets of files should use a 4-level (or deeper) hierarchy.
Large Number of Objects Loaded into a LevelDB Backend
Introduction
1 million records were loaded through the REST interface into a three level hierarchy (100 top-level nodes, 100 second-level nodes in each, 100 third-level nodes in each, and one 1 MB datastream in each) in a Fedora instance configured to use the LevelDB backend. The test was a follow-up to the Flat Hierarchy Testing page, where tests were run with to load objects into a flat backend structure. The tests, in both cases, were run on an EC2 m3.medium instance using `curl` through a Bash script. The script successfully loaded 962,964 objects with 1 MB datastreams, but failed to reach the 1 million object mark. The failure was a JVM core dump. The test has been restarted.
e
An earlier