Testing on AWS
2020-12-16
Expand |
---|
title | Test 4 - Number of containers - default - SHA256 |
---|
|
TesterTestResults- Terminated at 32,203,148 empty containers - no space left on 1-TB device
- Duration: 5 days 4 hrs 10 mins
No Format |
---|
summary = 32203148 in 124:10:42 = 72.0/s Avg: 49 Min: 3 Max: 1126 Err: 1 (0.00%)
Tidying up ... @ Wed Dec 16 03:41:11 UTC 2020 (1608090071444) |
- Grafana dashboard
|
2020-12-09
Expand |
---|
title | Test 4 - Number of containers - default - SHA512 |
---|
|
TesterTestResults- Terminated at 31,593,788 empty containers - no space left on 1-TB device
- Duration: 5 days 7 hr 49 mins
No Format |
---|
summary = 31593788 in 127:49:38 = 68.7/s Avg: 52 Min: 3 Max: 60068 Err: 1 (0.00%)
Tidying up ... @ Wed Dec 09 03:19:59 UTC 2020 (1607483999742) |
- Grafana dashboard
Environment |
2020-12-04
Expand |
---|
title | Test 6 - Number of Archival Groups with nested binaries |
---|
|
TesterTest- Test 6 - Number of Archival Groups with nested binaries
Code Block |
---|
nohup jmeter -Dfedora_4_context=/fcrepo/rest -Dcontainer_ag_threads=4 -n -t ~/fcrepo4-jmeter/fedora.jmx > test-out.txt |
Results- Terminated at 1,143,873 archival groups - no space left on 1-TB device
Duration: 6 hrs 3 mins No Format |
---|
summary = 1143873 in 06:03:53 = 52.4/s Avg: 73 Min: 4 Max: 2055 Err: 4 (0.00%)
Tidying up ... @ Fri Dec 04 06:50:21 UTC 2020 (1607064621527)
|
- Grafana dashboard
Environment- Instance type: t3.xlarge
- 4 vCPUs
- 16 GB memory
- Fedora
- Build #1d26867e (2020-12-03)
- PostgreSQL
- v12.3
- Tomcat info (java -cp /usr/share/tomcat9/lib/catalina.jar org.apache.catalina.util.ServerInfo)
No Format |
---|
Server version: Apache Tomcat/9.0.16 (Ubuntu)
Server built: Sep 11 2019 19:47:51 UTC
Server number: 9.0.16.0
OS Name: Linux
OS Version: 5.4.0-1030-aws
Architecture: amd64
JVM Version: 11.0.9.1+1-Ubuntu-0ubuntu1.18.04
JVM Vendor: Ubuntu |
- JVM (java -version)
No Format |
---|
openjdk version "11.0.9.1" 2020-11-04
OpenJDK Runtime Environment (build 11.0.9.1+1-Ubuntu-0ubuntu1.18.04)
OpenJDK 64-Bit Server VM (build 11.0.9.1+1-Ubuntu-0ubuntu1.18.04, mixed mode, sharing) |
- JAVA_OPTS (in /etc/default/tomcat9)
No Format |
---|
JAVA_OPTS="-Djava.awt.headless=true -server \
-Xms1G -Xmx2G -XX:MaxNewSize=1G \
-Dfcrepo.activemq.directory=/fcrepo/activemq \
-Dfcrepo.home=/fcrepo \
-Dfcrepo.db.url=jdbc:postgresql://localhost:5432/fcrepo \
-Dfcrepo.db.user=*** \
-Dfcrepo.db.password=*** \
-Dfcrepo.metrics.enable=true \
-XX:+HeapDumpOnOutOfMemoryError \
-XX:HeapDumpPath=/fcrepo/mem"
|
|
Early tests
The following tests were performed prior to the resolution of ingest memory leaks.
Expand |
---|
TesterTests on t2.large- Instance type: t2.large
- 2 vCPUs
- 8 GB memory
PostgreSQLEnvironment- Fedora deployed into Tomcat 9
- 6.0.0-SNAPSHOT | Build #3e25823d (2020-09-09)
- PostgreSQL running in Docker
Tests- JMeter test 4 - Number of containers - default
Results- Failed at 305,308 resources - no space left on device
Expand |
---|
summary + 121 in 00:00:26 = 4.7/s Avg: 197 Min: 35 Max: 6224 Err: 1 (0.83%) Active: 0 Started: 1 Finished: 1 summary = 305308 in 04:54:47 = 17.3/s Avg: 53 Min: 3 Max: 7444 Err: 1 (0.00%) Tidying up ... @ Thu Sep 10 01:10:25 UTC 2020 (1599700225557)
Sep 10 01:10:23 ip-10-0-0-234 tomcat9[6964]: java.io.UncheckedIOException: java.nio.file.FileSystemException: /opt/f6/data/staging/c6cd095e-7c40-4659-8532-a08d70342d59: No space left on device Sep 10 01:10:23 ip-10-0-0-234 tomcat9[6964]: at org.fcrepo.storage.ocfl.DefaultOcflObjectSession.createStagingPath(DefaultOcflObjectSession.java:490) Sep 10 01:10:23 ip-10-0-0-234 tomcat9[6964]: at org.fcrepo.storage.ocfl.DefaultOcflObjectSession.writeResource(DefaultOcflObjectSession.java:151) Sep 10 01:10:23 ip-10-0-0-234 tomcat9[6964]: at org.fcrepo.persistence.ocfl.impl.FcrepoOcflObjectSessionWrapper.lambda$writeResource$0(FcrepoOcflObjectSessionWrapper.java:64) Sep 10 01:10:23 ip-10-0-0-234 tomcat9[6964]: at org.fcrepo.persistence.ocfl.impl.FcrepoOcflObjectSessionWrapper.exec(FcrepoOcflObjectSessionWrapper.java:164) Sep 10 01:10:23 ip-10-0-0-234 tomcat9[6964]: at org.fcrepo.persistence.ocfl.impl.FcrepoOcflObjectSessionWrapper.writeResource(FcrepoOcflObjectSessionWrapper.java:64) Sep 10 01:10:23 ip-10-0-0-234 tomcat9[6964]: at org.fcrepo.persistence.ocfl.impl.AbstractRdfSourcePersister.writeRdf(AbstractRdfSourcePersister.java:139) Sep 10 01:10:23 ip-10-0-0-234 tomcat9[6964]: at org.fcrepo.persistence.ocfl.impl.AbstractRdfSourcePersister.persistRDF(AbstractRdfSourcePersister.java:76) Sep 10 01:10:23 ip-10-0-0-234 tomcat9[6964]: at org.fcrepo.persistence.ocfl.impl.CreateRdfSourcePersister.persist(CreateRdfSourcePersister.java:75) Sep 10 01:10:23 ip-10-0-0-234 tomcat9[6964]: at org.fcrepo.persistence.ocfl.impl.OcflPersistentStorageSession.persist(OcflPersistentStorageSession.java:160) Sep 10 01:10:23 ip-10-0-0-234 tomcat9[6964]: at org.fcrepo.kernel.impl.services.CreateResourceServiceImpl.perform(CreateResourceServiceImpl.java:182) Sep 10 01:10:23 ip-10-0-0-234 tomcat9[6964]: at org.fcrepo.http.api.FedoraLdp.createObject(FedoraLdp.java:622) |
Tests on t2.medium- Instance type: t2.medium
- 2 vCPUs
- 4 GB memory
MariaDBEnvironment- Fedora deployed into Tomcat 9
- 6.0.0-SNAPSHOT | Build #9ede6c3c (2020-08-07)
- MariaDB running in Docker
Tests- JMeter test 2 - Size of files - small
Results- Instance froze (became unresponsive) before 100,000 resources
Tests on t2.xlarge- Instance type: t2.xlarge
- 4 vCPUs
- 16 GB memory
MariaDBEnvironment- Fedora deployed into Tomcat 9
- 6.0.0-SNAPSHOT | Build #9ede6c3c (2020-08-07)
- MariaDB running in Docker
Tests- JMeter test 2 - Size of files - small
Results- Failed at 12,799 resources
Expand |
---|
summary + 72 in 00:00:10 = 7.0/s Avg: 138 Min: 62 Max: 147 Err: 1 (1.39%) Active: 0 Started: 1 Finished: 1 summary = 12799 in 00:18:29 = 11.5/s Avg: 81 Min: 3 Max: 270 Err: 1 (0.01%) Sep 08 23:48:10 ip-10-0-0-234 tomcat9[3189]: ERROR 23:48:10.279 (TransactionImpl) Failed to rollback storage in transaction c6af5528-8883-42f1-9d5f-67a352a5c1fb Sep 08 23:48:10 ip-10-0-0-234 tomcat9[3189]: org.fcrepo.persistence.api.exceptions.PersistentStorageException: Unable to rollback storage session c6af5528-8883-42f1-9d5f-67a352a5c1fb completely due to the following errors: Sep 08 23:48:10 ip-10-0-0-234 tomcat9[3189]: Cannot rollback object <info:fedora/28cfe2b3-162e-4d7b-ad19-aa36cb4ea95a>. Rollback to previous version not yet implemented. Sep 08 23:48:10 ip-10-0-0-234 tomcat9[3189]: at org.fcrepo.persistence.ocfl.impl.OcflPersistentStorageSession.rollbackCommittedSessions(OcflPersistentStorageSession.java:415) Sep 08 23:48:10 ip-10-0-0-234 tomcat9[3189]: at org.fcrepo.persistence.ocfl.impl.OcflPersistentStorageSession.rollback(OcflPersistentStorageSession.java:360) Sep 08 23:48:10 ip-10-0-0-234 tomcat9[3189]: at org.fcrepo.kernel.impl.TransactionImpl.lambda$rollback$0(TransactionImpl.java:112) Sep 08 23:48:10 ip-10-0-0-234 tomcat9[3189]: at org.fcrepo.kernel.impl.TransactionImpl.execQuietly(TransactionImpl.java:237) Sep 08 23:48:10 ip-10-0-0-234 tomcat9[3189]: at org.fcrepo.kernel.impl.TransactionImpl.rollback(TransactionImpl.java:111) Sep 08 23:48:10 ip-10-0-0-234 tomcat9[3189]: at org.fcrepo.kernel.impl.TransactionImpl.commit(TransactionImpl.java:92) Sep 08 23:48:10 ip-10-0-0-234 tomcat9[3189]: at org.fcrepo.kernel.impl.TransactionImpl.commitIfShortLived(TransactionImpl.java:177) Sep 08 23:48:10 ip-10-0-0-234 tomcat9[3189]: at org.fcrepo.http.api.FedoraLdp.createObject(FedoraLdp.java:626)
|
PostgreSQLEnvironment- Fedora deployed into Tomcat 9
- 6.0.0-SNAPSHOT | Build #9ede6c3c (2020-08-07)
- PostgreSQL running in Docker
Tests- JMeter test 2 - Size of files - small
Results- Failed at 264,792 resources
Expand |
---|
summary + 1249 in 00:00:30 = 41.6/s Avg: 19 Min: 3 Max: 108 Err: 0 (0.00%) Active: 1 Started: 1 Finished: 0 summary = 263967 in 01:43:56 = 42.3/s Avg: 19 Min: 2 Max: 665 Err: 0 (0.00%) summary + 825 in 00:00:20 = 41.1/s Avg: 20 Min: 17 Max: 583 Err: 1 (0.12%) Active: 0 Started: 1 Finished: 1 summary = 264792 in 01:44:16 = 42.3/s Avg: 19 Min: 2 Max: 665 Err: 1 (0.00%) Sep 09 01:45:57 ip-10-0-0-234 tomcat9[4233]: INFO 01:45:57.933 (FedoraLdp) Delete resource 'perf/perf_119336445' Sep 09 01:46:19 ip-10-0-0-234 tomcat9[4233]: INFO 01:46:19.604 (TransactionImpl) Rolling back transaction b5612f9d-57de-47a7-9069-4f579afe62b5 Sep 09 01:46:19 ip-10-0-0-234 tomcat9[4233]: INFO 01:46:19.604 (OcflPersistentStorageSession) Rolling back storage session b5612f9d-57de-47a7-9069-4f579afe62b5 Sep 09 01:46:20 ip-10-0-0-234 tomcat9[4233]: ERROR 01:46:20.054 (WildcardExceptionMapper) Exception intercepted by WildcardExceptionMapper: Failed to commit object <info:fedora/7d4d0ab1-cebc-4822-bf64-3a7f3353a7ce> in session <b5612f9d-57de-47a7-9069-4f579afe62b5>
|
|