- Created by Andrew Woods, last modified by Danny Bernstein on Mar 03, 2021
You are viewing an old version of this page. View the current version.
Compare with Current View Page History
« Previous Version 17 Next »
Testing on AWS
2020-12-16
Tester
Test
- Test 4 - number-of-containers---default
- Same as test below (2020-12-09), with the only difference being that the OCFL default digest algorithm was set to SHA256 instead of the default SHA512
Results
- Terminated at 32,203,148 empty containers - no space left on 1-TB device
- Duration: 5 days 4 hrs 10 mins
summary = 32203148 in 124:10:42 = 72.0/s Avg: 49 Min: 3 Max: 1126 Err: 1 (0.00%) Tidying up ... @ Wed Dec 16 03:41:11 UTC 2020 (1608090071444)
- Grafana dashboard
Environment
Instance Type: t3.xlarge
Database Engine: Postgres 12.0
Database Server: colocated on instance, running in Docker container
- JAVA_OPTS (in /etc/default/tomcat9)
JAVA_OPTS="-Djava.awt.headless=true -server \ -Xms1G -Xmx2G -XX:MaxNewSize=1G \ -Dfcrepo.activemq.directory=/f6/activemq \ -Dfcrepo.home=/f6 \ -Dfcrepo.db.url=jdbc:postgresql://localhost:5432/fcrepo \ -Dfcrepo.db.user=fcrepo-user \ -Dfcrepo.db.password=fcrepo-pw \ -Dfcrepo.persistence.defaultDigestAlgorithm=sha256 \ -Dfcrepo.metrics.enable=true \ -XX:+HeapDumpOnOutOfMemoryError \ -XX:HeapDumpPath=/f6/mem"
2020-12-09
Tester
Test
- Test 4 - number-of-containers---default
nohup jmeter -Dfedora_4_context=/fcrepo/rest -Dcontainer_threads=4 -n -t ~/fcrepo4-jmeter/fedora.jmx > test-out.txt
Results
- Terminated at 31,593,788 empty containers - no space left on 1-TB device
- Duration: 5 days 7 hr 49 mins
summary = 31593788 in 127:49:38 = 68.7/s Avg: 52 Min: 3 Max: 60068 Err: 1 (0.00%) Tidying up ... @ Wed Dec 09 03:19:59 UTC 2020 (1607483999742)
- Grafana dashboard
Environment
- Same as 2020-12-04 environment
2020-12-04
Tester
Test
- Test 6 - Number of Archival Groups with nested binaries
nohup jmeter -Dfedora_4_context=/fcrepo/rest -Dcontainer_ag_threads=4 -n -t ~/fcrepo4-jmeter/fedora.jmx > test-out.txt
Results
- Terminated at 1,143,873 archival groups - no space left on 1-TB device
Duration: 6 hrs 3 mins
summary = 1143873 in 06:03:53 = 52.4/s Avg: 73 Min: 4 Max: 2055 Err: 4 (0.00%) Tidying up ... @ Fri Dec 04 06:50:21 UTC 2020 (1607064621527)
- Grafana dashboard
Environment
- Instance type: t3.xlarge
- 4 vCPUs
- 16 GB memory
- Fedora
- Build #1d26867e (2020-12-03)
- PostgreSQL
- v12.3
- Tomcat info (java -cp /usr/share/tomcat9/lib/catalina.jar org.apache.catalina.util.ServerInfo)
Server version: Apache Tomcat/9.0.16 (Ubuntu) Server built: Sep 11 2019 19:47:51 UTC Server number: 9.0.16.0 OS Name: Linux OS Version: 5.4.0-1030-aws Architecture: amd64 JVM Version: 11.0.9.1+1-Ubuntu-0ubuntu1.18.04 JVM Vendor: Ubuntu
- JVM (java -version)
openjdk version "11.0.9.1" 2020-11-04 OpenJDK Runtime Environment (build 11.0.9.1+1-Ubuntu-0ubuntu1.18.04) OpenJDK 64-Bit Server VM (build 11.0.9.1+1-Ubuntu-0ubuntu1.18.04, mixed mode, sharing)
- JAVA_OPTS (in /etc/default/tomcat9)
JAVA_OPTS="-Djava.awt.headless=true -server \ -Xms1G -Xmx2G -XX:MaxNewSize=1G \ -Dfcrepo.activemq.directory=/fcrepo/activemq \ -Dfcrepo.home=/fcrepo \ -Dfcrepo.db.url=jdbc:postgresql://localhost:5432/fcrepo \ -Dfcrepo.db.user=*** \ -Dfcrepo.db.password=*** \ -Dfcrepo.metrics.enable=true \ -XX:+HeapDumpOnOutOfMemoryError \ -XX:HeapDumpPath=/fcrepo/mem"
Early tests
The following tests were performed prior to the resolution of ingest memory leaks.
Tester
Tests on t2.large
- Instance type: t2.large
- 2 vCPUs
- 8 GB memory
PostgreSQL
Environment
- Fedora deployed into Tomcat 9
- 6.0.0-SNAPSHOT | Build #3e25823d (2020-09-09)
- PostgreSQL running in Docker
Tests
- JMeter test 4 - Number of containers - default
Results
- Failed at 305,308 resources - no space left on device
summary + 121 in 00:00:26 = 4.7/s Avg: 197 Min: 35 Max: 6224 Err: 1 (0.83%) Active: 0 Started: 1 Finished: 1
summary = 305308 in 04:54:47 = 17.3/s Avg: 53 Min: 3 Max: 7444 Err: 1 (0.00%)
Tidying up ... @ Thu Sep 10 01:10:25 UTC 2020 (1599700225557)Sep 10 01:10:23 ip-10-0-0-234 tomcat9[6964]: java.io.UncheckedIOException: java.nio.file.FileSystemException: /opt/f6/data/staging/c6cd095e-7c40-4659-8532-a08d70342d59: No space left on device
Sep 10 01:10:23 ip-10-0-0-234 tomcat9[6964]: at org.fcrepo.storage.ocfl.DefaultOcflObjectSession.createStagingPath(DefaultOcflObjectSession.java:490)
Sep 10 01:10:23 ip-10-0-0-234 tomcat9[6964]: at org.fcrepo.storage.ocfl.DefaultOcflObjectSession.writeResource(DefaultOcflObjectSession.java:151)
Sep 10 01:10:23 ip-10-0-0-234 tomcat9[6964]: at org.fcrepo.persistence.ocfl.impl.FcrepoOcflObjectSessionWrapper.lambda$writeResource$0(FcrepoOcflObjectSessionWrapper.java:64)
Sep 10 01:10:23 ip-10-0-0-234 tomcat9[6964]: at org.fcrepo.persistence.ocfl.impl.FcrepoOcflObjectSessionWrapper.exec(FcrepoOcflObjectSessionWrapper.java:164)
Sep 10 01:10:23 ip-10-0-0-234 tomcat9[6964]: at org.fcrepo.persistence.ocfl.impl.FcrepoOcflObjectSessionWrapper.writeResource(FcrepoOcflObjectSessionWrapper.java:64)
Sep 10 01:10:23 ip-10-0-0-234 tomcat9[6964]: at org.fcrepo.persistence.ocfl.impl.AbstractRdfSourcePersister.writeRdf(AbstractRdfSourcePersister.java:139)
Sep 10 01:10:23 ip-10-0-0-234 tomcat9[6964]: at org.fcrepo.persistence.ocfl.impl.AbstractRdfSourcePersister.persistRDF(AbstractRdfSourcePersister.java:76)
Sep 10 01:10:23 ip-10-0-0-234 tomcat9[6964]: at org.fcrepo.persistence.ocfl.impl.CreateRdfSourcePersister.persist(CreateRdfSourcePersister.java:75)
Sep 10 01:10:23 ip-10-0-0-234 tomcat9[6964]: at org.fcrepo.persistence.ocfl.impl.OcflPersistentStorageSession.persist(OcflPersistentStorageSession.java:160)
Sep 10 01:10:23 ip-10-0-0-234 tomcat9[6964]: at org.fcrepo.kernel.impl.services.CreateResourceServiceImpl.perform(CreateResourceServiceImpl.java:182)
Sep 10 01:10:23 ip-10-0-0-234 tomcat9[6964]: at org.fcrepo.http.api.FedoraLdp.createObject(FedoraLdp.java:622)
Tests on t2.medium
- Instance type: t2.medium
- 2 vCPUs
- 4 GB memory
MariaDB
Environment
- Fedora deployed into Tomcat 9
- 6.0.0-SNAPSHOT | Build #9ede6c3c (2020-08-07)
- MariaDB running in Docker
Tests
- JMeter test 2 - Size of files - small
Results
- Instance froze (became unresponsive) before 100,000 resources
Tests on t2.xlarge
- Instance type: t2.xlarge
- 4 vCPUs
- 16 GB memory
MariaDB
Environment
- Fedora deployed into Tomcat 9
- 6.0.0-SNAPSHOT | Build #9ede6c3c (2020-08-07)
- MariaDB running in Docker
Tests
- JMeter test 2 - Size of files - small
Results
- Failed at 12,799 resources
summary + 72 in 00:00:10 = 7.0/s Avg: 138 Min: 62 Max: 147 Err: 1 (1.39%) Active: 0 Started: 1 Finished: 1
summary = 12799 in 00:18:29 = 11.5/s Avg: 81 Min: 3 Max: 270 Err: 1 (0.01%)
Sep 08 23:48:10 ip-10-0-0-234 tomcat9[3189]: ERROR 23:48:10.279 (TransactionImpl) Failed to rollback storage in transaction c6af5528-8883-42f1-9d5f-67a352a5c1fb
Sep 08 23:48:10 ip-10-0-0-234 tomcat9[3189]: org.fcrepo.persistence.api.exceptions.PersistentStorageException: Unable to rollback storage session c6af5528-8883-42f1-9d5f-67a352a5c1fb completely due to the following errors:
Sep 08 23:48:10 ip-10-0-0-234 tomcat9[3189]: Cannot rollback object <info:fedora/28cfe2b3-162e-4d7b-ad19-aa36cb4ea95a>. Rollback to previous version not yet implemented.
Sep 08 23:48:10 ip-10-0-0-234 tomcat9[3189]: at org.fcrepo.persistence.ocfl.impl.OcflPersistentStorageSession.rollbackCommittedSessions(OcflPersistentStorageSession.java:415)
Sep 08 23:48:10 ip-10-0-0-234 tomcat9[3189]: at org.fcrepo.persistence.ocfl.impl.OcflPersistentStorageSession.rollback(OcflPersistentStorageSession.java:360)
Sep 08 23:48:10 ip-10-0-0-234 tomcat9[3189]: at org.fcrepo.kernel.impl.TransactionImpl.lambda$rollback$0(TransactionImpl.java:112)
Sep 08 23:48:10 ip-10-0-0-234 tomcat9[3189]: at org.fcrepo.kernel.impl.TransactionImpl.execQuietly(TransactionImpl.java:237)
Sep 08 23:48:10 ip-10-0-0-234 tomcat9[3189]: at org.fcrepo.kernel.impl.TransactionImpl.rollback(TransactionImpl.java:111)
Sep 08 23:48:10 ip-10-0-0-234 tomcat9[3189]: at org.fcrepo.kernel.impl.TransactionImpl.commit(TransactionImpl.java:92)
Sep 08 23:48:10 ip-10-0-0-234 tomcat9[3189]: at org.fcrepo.kernel.impl.TransactionImpl.commitIfShortLived(TransactionImpl.java:177)
Sep 08 23:48:10 ip-10-0-0-234 tomcat9[3189]: at org.fcrepo.http.api.FedoraLdp.createObject(FedoraLdp.java:626)
PostgreSQL
Environment
- Fedora deployed into Tomcat 9
- 6.0.0-SNAPSHOT | Build #9ede6c3c (2020-08-07)
- PostgreSQL running in Docker
Tests
- JMeter test 2 - Size of files - small
Results
- Failed at 264,792 resources
summary + 1249 in 00:00:30 = 41.6/s Avg: 19 Min: 3 Max: 108 Err: 0 (0.00%) Active: 1 Started: 1 Finished: 0
summary = 263967 in 01:43:56 = 42.3/s Avg: 19 Min: 2 Max: 665 Err: 0 (0.00%)
summary + 825 in 00:00:20 = 41.1/s Avg: 20 Min: 17 Max: 583 Err: 1 (0.12%) Active: 0 Started: 1 Finished: 1
summary = 264792 in 01:44:16 = 42.3/s Avg: 19 Min: 2 Max: 665 Err: 1 (0.00%)
Sep 09 01:45:57 ip-10-0-0-234 tomcat9[4233]: INFO 01:45:57.933 (FedoraLdp) Delete resource 'perf/perf_119336445'
Sep 09 01:46:19 ip-10-0-0-234 tomcat9[4233]: INFO 01:46:19.604 (TransactionImpl) Rolling back transaction b5612f9d-57de-47a7-9069-4f579afe62b5
Sep 09 01:46:19 ip-10-0-0-234 tomcat9[4233]: INFO 01:46:19.604 (OcflPersistentStorageSession) Rolling back storage session b5612f9d-57de-47a7-9069-4f579afe62b5
Sep 09 01:46:20 ip-10-0-0-234 tomcat9[4233]: ERROR 01:46:20.054 (WildcardExceptionMapper) Exception intercepted by WildcardExceptionMapper: Failed to commit object <info:fedora/7d4d0ab1-cebc-4822-bf64-3a7f3353a7ce> in session <b5612f9d-57de-47a7-9069-4f579afe62b5>
- No labels