...
For a first test, we're going to ingest all the data from the filesystem into a clean fcrepo3 repository, using the filename as the datastream name.
(Fedora 3.7.1, clean install, using these properties:
| Code Block |
|---|
database=mysql database.driver=included database.jdbcDriverClass=com.mysql.jdbc.Driver database.mysql.jdbcDriverClass=com.mysql.jdbc.Driver database.mysql.driver=included database.jdbcURL=jdbc\:mysql\://localhost/fedora?useUnicode\=true database.mysql.jdbcURL=jdbc\:mysql\://localhost/fedora?useUnicode\=true database.username=fedora database.password=redacted install.type=custom deploy.local.services=false install.tomcat=false servlet.engine=existingTomcat fedora.home=/home/lyberadmin/apps/fedora/home fedora.serverHost=sul-fedora-dev-a.stanford.edu fedora.serverContext=fedora tomcat.http.port=8080 tomcat.shutdown.port=8005 ssl.available=true tomcat.ssl.port=8443 tomcat.home=/usr/share/tomcat6 ri.enabled=true messaging.enabled=false messaging.uri= apim.ssl.required=false apia.ssl.required=false apia.auth.required=false fesl.authz.enabled=false fesl.authn.enabled=true xacml.enabled=false keystore.file=included |
)
(single-threaded) Using curlUsing bash:
| Code Block |
|---|
#!/bin/bash
base_url="http://fedoraAdmin:fedoraAdmin@localhost/fedora"
RuntimePrint()
{
duration=$(echo "scale=3;(${m2t}-${m1t})/(1*10^09)"|bc|sed 's/^\./0./')
echo -e "${objectId} ${datastreams} ${size} ${duration}\tsec"
echo -e "${objectId} ${datastreams} ${size} ${duration}" >> /data/fcrepo3-total-create-object-time
}
CreateObject() {
pid="druid:$1"
curl -X POST "$base_url/objects/$pid" &> /dev/null
cd /data-ro/assets/$1
for f in $( ls ); do
datastreams=$[$datastreams+1]
size=$[$size+`stat -c "%s" $f`]
curl -X POST --data-binary @$f "$base_url/objects/$pid/datastreams/$f?controlGroup=M" &> /dev/null
done
cd /data
}
BenchmarkObject() {
objectId=$1
if [ -d /data-ro/assets/$objectId ]; then
m1t=$(date +%s%N); m1l=$LINENO
CreateObject $objectId
m2t=$(date +%s%N); m2l=$LINENO; RuntimePrint
fi
}
export -f BenchmarkObject
export -f CreateObject
export -f RuntimePrint
export base_url
cat - | parallel -P $THREADS --env _ BenchmarkObject |
Test
...
1a: Single-threaded ingest
| Code Block |
|---|
> quantile(data$V2, c(0, .5, .7, .9, .95, .99, 1))
0% 50% 70% 90% 95% 99% 100%
0.32300 1.32700 2.16100 6.30100 11.10600 36.88524 338.28700 |
Test 1b: Single-threaded iteration
Retrieve object profile
| Code Block |
|---|
> quantile(data$V2, c(0, .5, .7, .9, .95, .99, 1))
0% 50% 70% 90% 95% 99% 100%
0.002 0.054 0.062 0.077 0.089 0.123 3.580 |
...
Test 1c: 8-thread test
...