Child pages:


Testing on AWS

2020-12-16

Tester

Test

Results

  • Terminated at 32,203,148 empty containers - no space left on 1-TB device
  • Duration: 5 days 4 hrs 10 mins
    summary = 32203148 in 124:10:42 = 72.0/s Avg: 49 Min: 3 Max: 1126 Err: 1 (0.00%)
Tidying up ... @ Wed Dec 16 03:41:11 UTC 2020 (1608090071444)
  • Grafana dashboard

Environment

Same as 12-09

Database Server:  colocated on instance, running in Docker container

  1. JAVA_OPTS (in /etc/default/tomcat9)
    1. JAVA_OPTS="-Djava.awt.headless=true -server \
-Xms1G -Xmx2G -XX:MaxNewSize=1G \
-Dfcrepo.activemq.directory=/f6/activemq \
-Dfcrepo.home=/f6 \
-Dfcrepo.db.url=jdbc:postgresql://localhost:5432/fcrepo \
-Dfcrepo.db.user=fcrepo-user \
-Dfcrepo.db.password=fcrepo-pw \
-Dfcrepo.persistence.defaultDigestAlgorithm=sha256 \
-Dfcrepo.metrics.enable=true \
-XX:+HeapDumpOnOutOfMemoryError \
-XX:HeapDumpPath=/f6/mem"

2020-12-09

Tester

Test

Results

  • Terminated at 31,593,788 empty containers - no space left on 1-TB device
  • Duration: 5 days 7 hr 49 mins
    summary = 31593788 in 127:49:38 = 68.7/s Avg: 52 Min: 3 Max: 60068 Err: 1 (0.00%)
Tidying up ... @ Wed Dec 09 03:19:59 UTC 2020 (1607483999742)
  • Grafana dashboard

Environment

2020-12-04

Tester

Test

  • Test 6 - Number of Archival Groups with nested binaries
  • nohup jmeter -Dfedora_4_context=/fcrepo/rest -Dcontainer_ag_threads=4 -n -t ~/fcrepo4-jmeter/fedora.jmx > test-out.txt

Results

  • Terminated at 1,143,873 archival groups - no space left on 1-TB device

  • Duration: 6 hrs 3 mins

    summary = 1143873 in 06:03:53 = 52.4/s Avg: 73 Min: 4 Max: 2055 Err: 4 (0.00%)
Tidying up ... @ Fri Dec 04 06:50:21 UTC 2020 (1607064621527)
  • Grafana dashboard

Environment

  1. Instance type: t3.xlarge
    1. 4 vCPUs
    2. 16 GB memory
  2. Fedora
    1. Build #1d26867e (2020-12-03)
  3. PostgreSQL in Docker colocated on instance
    1. v12.3
  4. Tomcat info (java -cp /usr/share/tomcat9/lib/catalina.jar org.apache.catalina.util.ServerInfo)
    1. Server version: Apache Tomcat/9.0.16 (Ubuntu)
Server built: Sep 11 2019 19:47:51 UTC
Server number: 9.0.16.0
OS Name: Linux
OS Version: 5.4.0-1030-aws
Architecture: amd64
JVM Version: 11.0.9.1+1-Ubuntu-0ubuntu1.18.04
JVM Vendor: Ubuntu
  5. JVM (java -version)
    1. openjdk version "11.0.9.1" 2020-11-04
OpenJDK Runtime Environment (build 11.0.9.1+1-Ubuntu-0ubuntu1.18.04)
OpenJDK 64-Bit Server VM (build 11.0.9.1+1-Ubuntu-0ubuntu1.18.04, mixed mode, sharing)
  6. JAVA_OPTS (in /etc/default/tomcat9)
    1. JAVA_OPTS="-Djava.awt.headless=true -server \
-Xms1G -Xmx2G -XX:MaxNewSize=1G \
-Dfcrepo.activemq.directory=/fcrepo/activemq \
-Dfcrepo.home=/fcrepo \
-Dfcrepo.db.url=jdbc:postgresql://localhost:5432/fcrepo \
-Dfcrepo.db.user=*** \
-Dfcrepo.db.password=*** \
-Dfcrepo.metrics.enable=true \
-XX:+HeapDumpOnOutOfMemoryError \
-XX:HeapDumpPath=/fcrepo/mem"

Early tests

The following tests were performed prior to the resolution of ingest memory leaks.

Tester

Tests on t2.large

  1. Instance type: t2.large
    1. 2 vCPUs
    2. 8 GB memory

PostgreSQL

Environment
  1. Fedora deployed into Tomcat 9
    1. 6.0.0-SNAPSHOT | Build #3e25823d (2020-09-09)
  2. PostgreSQL running in Docker
Tests
  1. JMeter test 4 - Number of containers - default
Results
  1. Failed at 305,308 resources - no space left on device

    1. summary + 121 in 00:00:26 = 4.7/s Avg: 197 Min: 35 Max: 6224 Err: 1 (0.83%) Active: 0 Started: 1 Finished: 1
      summary = 305308 in 04:54:47 = 17.3/s Avg: 53 Min: 3 Max: 7444 Err: 1 (0.00%)
      Tidying up ... @ Thu Sep 10 01:10:25 UTC 2020 (1599700225557)


      Sep 10 01:10:23 ip-10-0-0-234 tomcat9[6964]: java.io.UncheckedIOException: java.nio.file.FileSystemException: /opt/f6/data/staging/c6cd095e-7c40-4659-8532-a08d70342d59: No space left on device
      Sep 10 01:10:23 ip-10-0-0-234 tomcat9[6964]: at org.fcrepo.storage.ocfl.DefaultOcflObjectSession.createStagingPath(DefaultOcflObjectSession.java:490)
      Sep 10 01:10:23 ip-10-0-0-234 tomcat9[6964]: at org.fcrepo.storage.ocfl.DefaultOcflObjectSession.writeResource(DefaultOcflObjectSession.java:151)
      Sep 10 01:10:23 ip-10-0-0-234 tomcat9[6964]: at org.fcrepo.persistence.ocfl.impl.FcrepoOcflObjectSessionWrapper.lambda$writeResource$0(FcrepoOcflObjectSessionWrapper.java:64)
      Sep 10 01:10:23 ip-10-0-0-234 tomcat9[6964]: at org.fcrepo.persistence.ocfl.impl.FcrepoOcflObjectSessionWrapper.exec(FcrepoOcflObjectSessionWrapper.java:164)
      Sep 10 01:10:23 ip-10-0-0-234 tomcat9[6964]: at org.fcrepo.persistence.ocfl.impl.FcrepoOcflObjectSessionWrapper.writeResource(FcrepoOcflObjectSessionWrapper.java:64)
      Sep 10 01:10:23 ip-10-0-0-234 tomcat9[6964]: at org.fcrepo.persistence.ocfl.impl.AbstractRdfSourcePersister.writeRdf(AbstractRdfSourcePersister.java:139)
      Sep 10 01:10:23 ip-10-0-0-234 tomcat9[6964]: at org.fcrepo.persistence.ocfl.impl.AbstractRdfSourcePersister.persistRDF(AbstractRdfSourcePersister.java:76)
      Sep 10 01:10:23 ip-10-0-0-234 tomcat9[6964]: at org.fcrepo.persistence.ocfl.impl.CreateRdfSourcePersister.persist(CreateRdfSourcePersister.java:75)
      Sep 10 01:10:23 ip-10-0-0-234 tomcat9[6964]: at org.fcrepo.persistence.ocfl.impl.OcflPersistentStorageSession.persist(OcflPersistentStorageSession.java:160)
      Sep 10 01:10:23 ip-10-0-0-234 tomcat9[6964]: at org.fcrepo.kernel.impl.services.CreateResourceServiceImpl.perform(CreateResourceServiceImpl.java:182)
      Sep 10 01:10:23 ip-10-0-0-234 tomcat9[6964]: at org.fcrepo.http.api.FedoraLdp.createObject(FedoraLdp.java:622)

Tests on t2.medium

  1. Instance type: t2.medium
    1. 2 vCPUs
    2. 4 GB memory

MariaDB

Environment
  1. Fedora deployed into Tomcat 9
    1. 6.0.0-SNAPSHOT | Build #9ede6c3c (2020-08-07)
  2. MariaDB running in Docker
Tests
  1. JMeter test 2 - Size of files - small
Results
  1. Instance froze (became unresponsive) before 100,000 resources

Tests on t2.xlarge

  1. Instance type: t2.xlarge
    1. 4 vCPUs
    2. 16 GB memory

MariaDB

Environment
  1. Fedora deployed into Tomcat 9
    1. 6.0.0-SNAPSHOT | Build #9ede6c3c (2020-08-07)
  2. MariaDB running in Docker
Tests
  1. JMeter test 2 - Size of files - small
Results
  1. Failed at 12,799 resources

    1. summary + 72 in 00:00:10 = 7.0/s Avg: 138 Min: 62 Max: 147 Err: 1 (1.39%) Active: 0 Started: 1 Finished: 1
      summary = 12799 in 00:18:29 = 11.5/s Avg: 81 Min: 3 Max: 270 Err: 1 (0.01%)


      Sep 08 23:48:10 ip-10-0-0-234 tomcat9[3189]: ERROR 23:48:10.279 (TransactionImpl) Failed to rollback storage in transaction c6af5528-8883-42f1-9d5f-67a352a5c1fb
      Sep 08 23:48:10 ip-10-0-0-234 tomcat9[3189]: org.fcrepo.persistence.api.exceptions.PersistentStorageException: Unable to rollback storage session c6af5528-8883-42f1-9d5f-67a352a5c1fb completely due to the following errors:
      Sep 08 23:48:10 ip-10-0-0-234 tomcat9[3189]: Cannot rollback object <info:fedora/28cfe2b3-162e-4d7b-ad19-aa36cb4ea95a>. Rollback to previous version not yet implemented.
      Sep 08 23:48:10 ip-10-0-0-234 tomcat9[3189]: at org.fcrepo.persistence.ocfl.impl.OcflPersistentStorageSession.rollbackCommittedSessions(OcflPersistentStorageSession.java:415)
      Sep 08 23:48:10 ip-10-0-0-234 tomcat9[3189]: at org.fcrepo.persistence.ocfl.impl.OcflPersistentStorageSession.rollback(OcflPersistentStorageSession.java:360)
      Sep 08 23:48:10 ip-10-0-0-234 tomcat9[3189]: at org.fcrepo.kernel.impl.TransactionImpl.lambda$rollback$0(TransactionImpl.java:112)
      Sep 08 23:48:10 ip-10-0-0-234 tomcat9[3189]: at org.fcrepo.kernel.impl.TransactionImpl.execQuietly(TransactionImpl.java:237)
      Sep 08 23:48:10 ip-10-0-0-234 tomcat9[3189]: at org.fcrepo.kernel.impl.TransactionImpl.rollback(TransactionImpl.java:111)
      Sep 08 23:48:10 ip-10-0-0-234 tomcat9[3189]: at org.fcrepo.kernel.impl.TransactionImpl.commit(TransactionImpl.java:92)
      Sep 08 23:48:10 ip-10-0-0-234 tomcat9[3189]: at org.fcrepo.kernel.impl.TransactionImpl.commitIfShortLived(TransactionImpl.java:177)
      Sep 08 23:48:10 ip-10-0-0-234 tomcat9[3189]: at org.fcrepo.http.api.FedoraLdp.createObject(FedoraLdp.java:626)

PostgreSQL

Environment
  1. Fedora deployed into Tomcat 9
    1. 6.0.0-SNAPSHOT | Build #9ede6c3c (2020-08-07)
  2. PostgreSQL running in Docker
Tests
  1. JMeter test 2 - Size of files - small
Results
  1. Failed at 264,792 resources

    1. summary + 1249 in 00:00:30 = 41.6/s Avg: 19 Min: 3 Max: 108 Err: 0 (0.00%) Active: 1 Started: 1 Finished: 0
      summary = 263967 in 01:43:56 = 42.3/s Avg: 19 Min: 2 Max: 665 Err: 0 (0.00%)
      summary + 825 in 00:00:20 = 41.1/s Avg: 20 Min: 17 Max: 583 Err: 1 (0.12%) Active: 0 Started: 1 Finished: 1
      summary = 264792 in 01:44:16 = 42.3/s Avg: 19 Min: 2 Max: 665 Err: 1 (0.00%)


      Sep 09 01:45:57 ip-10-0-0-234 tomcat9[4233]: INFO 01:45:57.933 (FedoraLdp) Delete resource 'perf/perf_119336445'
      Sep 09 01:46:19 ip-10-0-0-234 tomcat9[4233]: INFO 01:46:19.604 (TransactionImpl) Rolling back transaction b5612f9d-57de-47a7-9069-4f579afe62b5
      Sep 09 01:46:19 ip-10-0-0-234 tomcat9[4233]: INFO 01:46:19.604 (OcflPersistentStorageSession) Rolling back storage session b5612f9d-57de-47a7-9069-4f579afe62b5
      Sep 09 01:46:20 ip-10-0-0-234 tomcat9[4233]: ERROR 01:46:20.054 (WildcardExceptionMapper) Exception intercepted by WildcardExceptionMapper: Failed to commit object <info:fedora/7d4d0ab1-cebc-4822-bf64-3a7f3353a7ce> in session <b5612f9d-57de-47a7-9069-4f579afe62b5>

  • No labels

All content on the LYRASIS Wiki is licensed under the CC BY (Attribution) license, unless otherwise noted.