1. Create backup

1. curl -X POST http://localhost:8080/fcrepo/rest/fcr:backup

2. You will be given a directory in message body.  For example: C:\Users\Dev\projects\dcs-package-ingest\package-ingest-integration\target\cargo\configurations\tomcat8x\temp\1496329063578-0

3. You will see one or more files named documents_N.bin.gz where N is a number like 000001 (e.g. documents_000001.bin.gz)

2. Unzip and inspect the documents_N.bin.gz files.  Look for modeshape node whose primaryType is ‘mode:namespaces’

1. `jq` is a great tool for inspecting very large json files.  To pretty-print so that tools like grep can be used, try , cat documents_00001.bin | jq .

1. Grep for ‘namespaces’.  You’ll see something like:

   {
     "metadata": {
       "id": "936198c317f1e7mode:namespaces"
     },
     "content": {
       "key": "936198c317f1e7mode:namespaces",
       "parent": "936198c317f1e7jcr:system",
       "properties": {
         "http://www.jcp.org/jcr/1.0": {
           "primaryType": {
             "$name": "mode:namespaces"
           }
         }
       },
       "children": [
         {
           "key": "936198c317f1e7mode:namespaces-http://www.jcp.org/jcr/1.0",
           "name": "jcr"
         },
         {
           "key": "936198c317f1e7mode:namespaces-http://www.jcp.org/jcr/nt/1.0",
           "name": "nt"
         },
         {
           "key": "936198c317f1e7mode:namespaces-http://www.jcp.org/jcr/mix/1.0",
           "name": "mix"
         },
         {
           "key": "936198c317f1e7mode:namespaces-http://www.w3.org/2004/02/skos/core#",
           "name": "skos"
         }
   }

The dump contains a json object whose primary type is “mode:namespaces”.  Nested within it is a “children” array that enumerates the namespaces known to modeshape.  It appears that Modeshape chooses a single random prefix on all internal identifiers.  Acknowledging that prefix, the modeshape namespaces enumeration follows a simple and obvious pattern.

When Fedora encounters a novel rdf namespace and prefix, it adds it to the map.  In many cases, the prefix is auto-generated, e.g. “ns001”.  So if Fedora encountered a SKOS term, at any point, it may contain a namespace entry like:

{
    "key": "936198c317f1e7mode:namespaces-http://www.w3.org/2004/02/skos/core#",
    "name": "ns004"
}

In Fedora 4.7.1 and 4.7.2, a new set of hard-coded namespaces were published in the internal .cnd file used by modeshape.  When started, any pre-existing namespaces that match the hard-coded namespaces, yet differ by prefix, are overwritten.  So the namespaces table may now contain an entry:

{          
    "key": "936198c317f1e7mode:namespaces-http://www.w3.org/2004/02/skos/core#",
    "name": "skos"
}

This overwriting of the namespaces table is problematic.  Objects that existed before the repository had been upgraded to 4.7.1 or 4.7.2 will still contain their _old_ namespace references, such as:

{
  "metadata": {
    "id": "936198c317f1e7/jcr:system/jcr:nodeTypes/{httpwww.w3.org200402skoscore#}Concept"
  },
  "content": {
    "key": "936198c317f1e7/jcr:system/jcr:nodeTypes/{httpwww.w3.org200402skoscore#}Concept",
    "parent": "936198c317f1e7jcr:nodeTypes",
    "properties": {
      "http://www.jcp.org/jcr/1.0": {
        "isQueryable": true,
        "primaryType": {
          "$name": "nt:nodeType"
        },
        "supertypes": [],
        "hasOrderableChildNodes": false,
        "isMixin": true,
        "isAbstract": false,
        "nodeTypeName": "ns004:Concept" << NOTE: THIS IS THE PROBLEMATIC LINE <<
      }
    }
  }

This inconsistency is at the root of the problem.

3. For each dangling namespace, add a new entry to the mode:namespaces node, for example:

    "children": [
      {
        "key": "936198c317f1e7mode:namespaces-http://www.jcp.org/jcr/1.0",
        "name": "jcr"
      },
      {
        "key": "936198c317f1e7mode:namespaces-http://www.jcp.org/jcr/nt/1.0",
        "name": "nt"
      },
      {
        "key": "936198c317f1e7mode:namespaces-http://www.jcp.org/jcr/mix/1.0",
        "name": "mix"
      },
      {
        "key": "936198c317f1e7mode:namespaces-http://www.w3.org/2004/02/skos/core#",
        "name": "skos"
      },
      {
        "key": "936198c317f1e7mode:namespaces-http://www.w3.org/2004/02/skos/core#",
        "name": "ns004"
      }
]

4. After the namespaces node has been fixed, gzip the file so that it retains its original name

5. Restore the backup via fcr:restore

In the case that you have upgraded to either 4.7.1 or 4.7.2 and have stopped your server and are not able to restart your server with an error in your logs like the following:

org.modeshape.jcr.value.ValueFormatException: Error converting "ns00x:Object" from String to a Name

..this fedora-tech mailing list thread provides guidance for remediation.