Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Warning
titleOutOfMemoryException when ingesting large files

Currently there seems to be a bug, which creates OutOfMemoryExceptions when ingesting files that are larger than available heap space with certain infinispan configurations (e.g. LevelDB). It seems like this is an issue with the Modeshape project which has been reported at: https://issues.jboss.org/browse/MODE-2103

The following TestCase can be used to reproduce the issue: https://github.com/futures/large-files-test

Workaround

You will need a large heap size for this to work (e.g. -Xmx2048g)

Currently the only known workaround is using a _file_ configuration for infinspan caches e.g.: https://github.com/futures/fcrepo4/blob/34aab66bc26edfca3a4cbabecc4870bfd81f05da/fcrepo-http-commons/src/main/resources/config/single-file/repository.json.

This can be done by setting the following property:

-Dfcrepo.modeshape.configuration=config/single-file/repository.json

...

  • Platform: Linux 3.12.1-1-ARCH #1 SMP PREEMPT x86_64 GNU/Linux
  • Repository Profile: Single-File with an additional external Resource:

    "externalSources" : {
    "home-directory" : {
        "classname" : "org.modeshape.connector.filesystem.FileSystemConnector",
        "directoryPath" : "/tmp/projection",
        "projections" : [ "default:/projection => /" ],
        "readOnly" : true,
        "addMimeTypeMixin" : true
        }
    }

10GB
File SizeFirst Projection Directory Request DurationFirst Projected Node Request Duration

Download Duration

Throughput
2GB2 GB0m35.117s0m34.572s0m8.236s248.66 mb/sec
10 GB    

100 GB

  

 

 100GB
300 GB    
10*10GB10 GB    

 

Related articles

...