Upgrading DSpace

Release Notes / Significant Changes

DSpace 7.0 features some significant changes which you may wish to be aware of before beginning your upgrade:

• XMLUI and JSPUI are no longer supported or distributed with DSpace. All users should immediately migrate to and utilize the new Angular User Interface.  TBS:  How?
  
• ElasticSearch Usage Statistics have been removed.  Please use SOLR Statistics or DSpace Google Analytics Statistics.
• Configuration has been upgraded to Apache Commons Configuration version 2. For most users, you should see no effect or difference. No DSpace configuration files were modified during this upgrade and no configurations or settings were renamed or changed. However, if you locally modified or customized the [dspace]/config/config-definition.xml (DSpace's Apache Commons Configuration settings), you will need to ensure those modifications are compatible with Apache Commons Configuration version 2.  See the Apache Commons Configuration's configuration definition file reference for more details.
• Solr is now installed separately due to changes in the packaging of recent Solr releases.  See below.
  

Backup your DSpace

Before you start your upgrade, it is strongly recommended that you create a backup of your DSpace instance. Backups are easy to recover from; a botched install/upgrade is very difficult if not impossible to recover from. The DSpace specific things to backup are: configs, source code modifications, database, and assetstore. On your server that runs DSpace, you might additionally consider checking on your cron/scheduled tasks, servlet container, and database.

Make a complete backup of your system, including:

• Database: Make a snapshot/dump of the database. For the PostgreSQL database use Postgres' pg_dump command. For example:

  pg_dump -U [database-user] -f [backup-file-location] [database-name]

• Assetstore: Backup the directory ([dspace]/assetstore by default, and any other assetstores configured in the [dspace]/config/dspace.cfg "assetstore.dir" and "assetstore.dir.#" settings)
• Configuration: Backup the entire directory content of [dspace]/config.
• Customizations: If you have custom code, such as themes, modifications, or custom scripts, you will want to back them up to a safe location.
• Statistics data: what to back up depends on what you were using before: the options are the default SOLR Statistics, or the legacy statistics. Legacy stats utilizes the dspace.log files, while SOLR Statistics stores data in [dspace]/solr/statistics. A simple copy of the logs or the Solr core directory tree should give you a point of recovery, should something go wrong in the update process. We can't stress this enough:  your users depend on these statistics more than you realize. You need a backup.
• Authority data:  stored in [dspace]/solr/authority.  As with the statistics data, make a copy of the directory tree should enable recovery from errors.

Update Prerequisite Software (as necessary)

DSpace 7.x requires the following versions of prerequisite software:

• Java 7 or 8 (Oracle or OpenJDK)
• Apache Maven 3.0.5 or above
• Apache Ant 1.8 or above
• Database
  • PostgreSQL 9.4 or above (with pgcrypto installed), OR 
  • Oracle 10g or above
• Tomcat 7 or above
• New:  Solr 7.2.1 or above

Refer to the Prerequisite Software section of "Installing DSpace" for more details around configuring and installing these prerequisites.

Upgrade Steps

1. Ensure that your database is compatible: Starting with DSpace 6.x, there are new database requirements for DSpace (refer to the Prerequisite Software section of "Installing DSpace" for full details).
   
   1. PostgreSQL databases: PostgreSQL 9.4 or above is required and the "pgcrypto" extension must be installed.
      1. Notes on installing pgcrypto
         1. On most Linux operating systems (Ubuntu, Debian, RedHat), this extension is provided in the "postgresql-contrib" package in your package manager. So, ensure you've installed "postgresql-contrib".
         2. On Windows, this extension should be provided automatically by the installer (check your "[PostgreSQL]/share/extension" folder for files starting with "pgcrypto")

      2. Enabling pgcrypto on your DSpace database. (Additional options/notes in the Installation Documentation)

         # Login to your "dspace" database as a superuser
         psql --username=postgres dspace
         # Enable the pgcrypto extension on this database
         CREATE EXTENSION pgcrypto;

   2. Oracle databases: Oracle database have no additional requirements at this time.
2. Download DSpace 7.x: Either download DSpace 7.x from DSpace.org or check it out directly from the Github repository. 
   1. NOTE: You will need both the services component (DSpace/DSpace in Github) and the user interface component (DSpace/dspace-angular in Github).
   2. NOTE: If you downloaded DSpace do not unpack it on top of your existing installation. Refer to Installation Instructions, Step 3 for unpacking directives.
3. Merge any User Interface customizations or other customizations (if needed or desired).  If you have made any local customizations to your DSpace installation they must be re-implemented in the new DSpace, because the old UIs have been replaced. 
   1. NOTE: If you are upgrading across many versions of DSpace at once (e.g. from 1.x.x to 7.x), you may find it easier to first upgrade DSpace, and then attempt to migrate over your various customizations. Because each major version of DSpace tends to add new configurations and features to the User Interface, older customizations may require more work to "migrate" to the latest version of DSpace. In some situations, it may even be easier to "start fresh", and just re-customize the brand new User Interface with your local color scheme, header/footer, etc.
   2. Customizations are typically housed in one of the following places:
      1. JSPUI modifications: [dspace-source]/dspace/modules/jspui/src/main/webapp/
These will have to be reimplemented.

      2. XMLUI modifications: [dspace-source]/dspace/modules/xmlui/src/main/webapp/
These will have to be reimplemented.
      3. Config modifications: [dspace]/config

   3. For highly customized DSpace instances, note that the format of the following configuration files has changed.  If you have customized these configuration files, carefully re-integrate your custom settings.

      1. dspace/config/dspace.cfg

         • Note the escaped comma (\,) separator is now needed between element names: https://github.com/DSpace/DSpace/blob/dspace-6.0-rc3/dspace/config/dspace.cfg#L1020
           

      2. dspace/config/spring/api/discovery.xml

         • The following property has been removed
           •  <property name="sortOrder" value="COUNT"/>
         • And has been replaced with the following        
           • <property name="sortOrderSidebar" value="COUNT"/>
           • <property name="sortOrderFilterPage" value="COUNT"/>

4. Replace your old build.properties file with a local.cfg (REQUIRED if upgrading from DSpace 5 or previous): As of DSpace 6.0, the build.properties configuration file has been replaced by an enhanced local.cfg configuration file. Therefore, any old build.properties file (or similar [dspace-source]/*.properties files) WILL BE IGNORED. Instead, you should create a new local.cfg file, based on the provided [dspace-source]/dspace/config/local.cfg.EXAMPLE and use it to specify all of your locally customized DSpace configurations.  This new local.cfg can be used to override ANY setting in any other configuration file (dspace.cfg or modules/*.cfg). To override a default setting, simply copy the configuration into your local.cfg and change its value(s).  For much more information on the features of local.cfg, see the Configuration Reference documentation and the local.cfg Configuration File section on that page.

   cd [dspace-source]
   cp dspace/config/local.cfg.EXAMPLE local.cfg
   
   # Then edit the local.cfg, specifying (at a minimum) your basic DSpace configuration settings.
   # Optionally, you may copy any settings from other *.cfg configuration files into your local.cfg to override them.
   # After building DSpace, this local.cfg will be copied to [dspace]/config/local.cfg, where it will also be used at runtime.

5. Build DSpace. Run the following commands to compile DSpace :

   cd [dspace-source]/dspace/
   mvn -U clean package

   The above command will re-compile the DSpace source code and build its "installer". You will find the result in [dspace-source]/dspace/target/dspace-installer

   Defaults to PostgreSQL settings

   Without any extra arguments, the DSpace installation package is initialized for PostgreSQL. If you use Oracle instead, you should build the DSpace installation package as follows:
   mvn -Ddb.name=oracle -U clean package

6. Dump your authority and statistics Solr cores.  (Only when upgrading from DSpace 6 or older.)
   

   [dspace]/bin/dspace solr-export-statistics -i authority
   [dspace]/bin/dspace solr-export-statistics -i statistics

   The dumps will be written to the directory [dspace]/solr-export.  This may take a long time and require quite a lot of storage.  In particular, the statistics core is likely to be huge, perhaps double the size of the content of solr/statistics/data.  You should ensure that you have sufficient free storage.
   
   This is not the same as the disaster-recovery backup that was done above.  These dumps will be reloaded into new, reconfigured cores later.
   
   If you are sharding your statistics data, you will need to dump each shard separately.  The index names for prior years will be statistics-YYYY (for example:  statistics-2017 statistics-2018 etc.)  The current year's statistics shard is named statistics and you should dump that one too.

7. Stop Tomcat (or servlet container). Take down your servlet container. 

   1. For Tomcat, use the $CATALINA_HOME/shutdown.sh script. (Many Unix-based installations will have a startup/shutdown script in the /etc/init.d or /etc/rc.d directories.)

8. Update DSpace Installation. Update the DSpace installation directory with the new code and libraries. Issue the following commands:

   cd [dspace-source]/dspace/target/dspace-installer
   ant update

9. Update your DSpace Configurations and/or move them to local.cfg (REQUIRED if upgrading from DSpace 5 or previous): You should review your configuration for new and changed configurations in DSpace 6.x. 
   1. As mentioned above, DSpace 6.0 now includes a new local.cfg Configuration File.  So, rather than editing the dspace.cfg (or any of the modules/*.cfg), it's recommended to simply override the default values in your own local.cfg. That way, your local.cfg can serve as the record of which configurations you have actually tweaked in your DSpace, which may help to simplify future upgrades.
      1. WARNING: in order to create this powerful ability to override configurations in your local.cfg, all modules/*.cfg files had their configurations renamed to be pre-pended with the module name.  As a basic example, all the configuration settings within the modules/oai.cfg configuration now start with "oai.".  Unfortunately, these means that DSpace 5.x configuration files are NOT guaranteed to be compatible with DSpace 6. For more information on configurations in DSpace 6 see our updated Configuration Reference.
   2. Search/Browse requires Discovery: As of DSpace 6, only Discovery (Apache Solr) is supported for search/browse. Support for Legacy Search (using Apache Lucene) and Legacy Browse (using database tables) has been removed, along with all their configurations.
   3. XPDF media filtering no longer exists: XPDF media filtering, deprecated in DSpace 5, has been removed.  If you used this, you will need to reconfigure using the remaining alternatives (e.g. PDF Text Extractor and/or ImageMagick PDF Thumbnail Generator)
   4. Upgrading Configurable/XML Workflow may require minor configuration updates. If you are currently running the DSpace XMLUI with Configurable/XML Workflow enabled, you may need to re-enable its configurations in the DSpace 6 configuration files prior to upgrading. As with past releases, DSpace 6 defaults to using Basic (Traditional) Workflow. Therefore, you should double check the settings required to enable Configurable Workflow in DSpace 6. Pay close attention to the fact that, to enable Configurable Workflow in DSpace 6, all BasicWorkflow settings must also be commented out (in several configs).
   5. It is recommended to review all configuration changes that exist in the config directory, and its subdirectories. It is helpful to compare your current configs against a clean checkout of your current version to see what you have customized. You might then also want to compare your current configs with the configs of the version you are upgrading to. A tool that compares files in directories such as Meld or DiffMerge is useful for this purpose.
      1. After reviewing which configurations you've changed, we recommend moving all your customized configurations into your local.cfg file, as described above. Examples of how this might be accomplished are provided in the Configuration Reference.
   6. CHANGE IN DSpace 6.3: IP Address to geographic location database has been renamed.  The old [dspace]/config/GeoLiteCity.dat file is no longer maintained by its provider.  You can delete it.  The new file is named GeoLite2-City.mmdb by default. The upgrade process will automatically download a copy of the new database if you don't already have it.  If you have configured a different name and/or location for this file, you should check the setting of usage-statistics.dbfile in [dspace]/config/modules/usage-statistics.cfg (and perhaps move your custom setting to local.cfg).
   7. tm-extractors media filtering (WordFilter) no longer exists:  the PoiWordFilter plugin now fulfills this function.  If you still have WordFilter configured, remove from dspace.cfg and/or local.cfg all lines referencing org.dspace.app.mediafilter.WordFilter and uncomment all lines referencing org.dspace.app.mediafilter.PoiWordFilter.
   8. Re-configure Solr URLs:  change the value of solr.server to point at your new Solr v7 service.  It will probably become something like solr.server = https://${dspace.hostname}:8983/solr.  Also review the values of
      • discovery.search.server
      • oai.solr.url
      • solr.authority.server
      • solr.statistics.server

10. Decide which DSpace Web Applications you want to install. DSpace comes with a variety of web applications (in [dspace]/webapps), each of which provides a different "interface" to your DSpace.  Which ones you install is up to you, but there are a few that we highly recommend (see below):

    1. "server" = This contains all of the standard back-end services:  REST, SWORD, SWORDv2, RDF, OAI.  This is required.

    2. "rest" = This is the deprecated, old DSpace REST API.  The current REST API is part of "server".  It is optional.  If you use applications written to the pre-v7 REST API, you will need this until you have converted them.

11. Enable DSpace Web Applications. If necessary, copy the web applications from your [dspace]/webapps directory to the subdirectory of your servlet container (e.g. Tomcat):

    cp -R [dspace]/webapps/* [tomcat]/webapps/
    

    See the installation guide for full details.  Also move or delete superseded webapps (oai, sword, swordv2, rdf) that you may have here.

12. Upgrade your database (optional, but recommended for major upgrades). As of DSpace 5 (and above), the DSpace code will automatically upgrade your database (from any prior version of DSpace).  By default, this database upgrade occurs automatically when you restart Tomcat (or your servlet container).  However, if you have a large repository or are upgrading across multiple versions of DSpace at once, you may wish to manually perform the upgrade (as it could take some time, anywhere from 5-15 minutes for large sites).

    1. First, you can optionally verify whether DSpace correctly detects the version of your DSpace database. It is very important that the DSpace version is detected correctly before you attempt the migration:

       [dspace]/bin/dspace database info
       # Look for a line at the bottom that says something like:
       # "Your database looks to be compatible with DSpace version ___"

    2. In some rare scenarios, if your database's "sequences" are outdated, inconsistent or incorrect, a database migration error may occur (in your DSpace logs). While this is seemingly a rare occurance, you may choose to run the "update-sequences" command PRIOR to upgrading your database. If your database sequences are inconsistent or incorrect, this "update-sequences" command will auto-correct them (otherwise, it will do nothing).

       # General PostgreSQL example
       psql -U [database-user] -f [dspace]/etc/postgres/update-sequences.sql [database-name]
       
       # Example for a PostgreSQL database named "dspace", and a user account named "dspace"
       # psql -U dspace -f [dspace]/etc/postgres/update-sequences.sql dspace

    3. Then, you can upgrade your DSpace database to the latest version of DSpace. (NOTE: check the DSpace log, [dspace]/log/dspace.log.[date], for any output from this command)

       [dspace]/bin/dspace database migrate

       If you are upgrading from DSpace 6 or earlier, there are database changes which were optional but now are mandatory.  Instead of (or after) the above command:

       [dspace]/bin/dspace database migrate ignored

       to apply these changes.

    4. The database migration should also automatically trigger your metadata/file registries to be updated (based on the config files in [dspace]/config/registries/).  However, if this update was NOT triggered, you can also manually run these registry updates (they will not harm existing registry contents) as follows:

       [dspace]/bin/dspace registry-loader -metadata [dspace]/config/registries/dcterms-types.xml
       [dspace]/bin/dspace registry-loader -metadata [dspace]/config/registries/dublin-core-types.xml
       [dspace]/bin/dspace registry-loader -metadata [dspace]/config/registries/eperson-types.xml
       [dspace]/bin/dspace registry-loader -metadata [dspace]/config/registries/local-types.xml
       [dspace]/bin/dspace registry-loader -metadata [dspace]/config/registries/sword-metadata.xml
       [dspace]/bin/dspace registry-loader -metadata [dspace]/config/registries/workflow-types.xml
       

    5. If the database upgrade process fails or throws errors, then you likely have manually customized your database structure (and/or backported later DSpace features to an older version of DSpace). In this scenario, you may need to do some manual migrations before the automatic migrations will succeed. The general process would be something like this:
       
       1. Revert back to your current DSpace database
       2. Manually upgrade just your database past the failing migration.  For example, if you are current using DSpace 1.5 and the "V1.6" migration is failing, you may need to first manually upgrade your database to 1.6 compatibility. This may involve either referencing the upgrade documentation for that older version of DSpace, or running the appropriate SQL script from under [dspace-src]/dspace-api/src/main/resources/org/dspace/storage/rdbms/sqlmigration/)
       3. Then, re-run the migration process from that point forward (i.e. re-run ./dspace database migrate)
    6. More information on the "database" command can be found in Database Utilities documentation.

    By default, your site will be automatically reindexed after a database upgrade

    If any database migrations are run (even during minor release upgrades), then by default DSpace will automatically reindex all content in your site. This process is run automatically in order to ensure that any database-level changes are also immediately updated within the search/browse interfaces. See the notes below under "Restart Tomcat (servlet container)" for more information.

    However, you may choose to skip automatic reindexing. Some sites choose to run the reindex process manually in order to better control when/how it runs.

    To disable automatic reindexing, set discovery.autoReindex = false in config/local.cfg or config/modules/discovery.cfg.
    

    As you have disabled automatic reindexing, make sure to manually reindex your site by running [dspace]/bin/dspace discovery -b (This must be run after restarting Tomcat)

    WARNING: It is not recommended to skip automatic reindexing, unless you will manually reindex at a later time, or have verified that a reindex is not necessary. Forgetting to reindex your site after an upgrade may result in unexpected errors or instabilties.

13. Sites with Oracle database backends (and Configurable Workflow enabled) may need to run a "repair" on your database.

    1. In version 6.3, we fixed an Oracle migration issue related to Configurable (XML) Workflow. See DS-3788.
       

    2. If you are upgrading an Oracle-based site to 6.3 from 6.0, 6.1 or 6.2 AND had Configurable Workflow already enabled, then you will need to manually "repair" your database to align it with the latest schema. This does not affect PostgreSQL-based backends or any sites that are upgrading from 5.x or below.

    3. Simply run the following to repair your Oracle database:  [dspace]/bin/dspace database repair

14. Install new Solr cores and rebuild your indexes.  (Only necessary if upgrading from 6.x or below)

    1. Copy the new, empty Solr cores to your new Solr instance.

       cp -r [dspace]/solr/* [solr]/server/solr
       chown -R solr:solr [solr]/server/solr/authority [solr]/server/solr/oai [solr]/server/solr/search [solr]/server/solr/statistics

    2. Start Solr, or restart it if it is running, so that these new cores are loaded.

    3. Load authority and statistics from the dumps that you made earlier (not the disaster-recovery backup).

       [dspace]/bin/dspace solr-import-statistics -i authority
       [dspace]/bin/dspace solr-import-statistics -i statistics

       This could take quite some time.
       
       If you had sharded your statistics, you will need to load the dump of each shard separately.  As when dumping, the index names will be ... statistics-2017 statistics-2018 statistics.
       
       

    4. Rebuild the oai and search cores.

       [dspace]/bin/dspace oai import
       [dspace]/bin/dspace index-discovery

       If you have a great deal of content, this could take a long time.

15. Update Handle Server Configuration. If you are using the built-in Handle server, you'll need to add the follow to the server_config section of your config.dct file:

    "enable_txn_queue" = "no"

16. (optional) Set up lookup of clients' locations.  If you wish to (continue to) record the geographic origin of client activity, you will need to make new arrangements to provide an updated GeoLite database.
    • You will need to set up a free account at MaxMind
    • You will need to arrange regular downloads of the GeoLite2 database.  MaxMind offers an updater tool to do the downloading, and a number of Linux distributions package it.  Use it before restarting DSpace, to get an up-to-date database.
    • You should check the value of usage-statistics.dbfile in config/modules/usage-statistics.cfg – it should point to the path where the updater puts the database.  A typical value (on a Linux host) is commented out.
    • You can discard any old GeoLite database(s) found in the config/ directory.

17. Restart Tomcat (servlet container). Now restart your servlet container (Tomcat/Jetty/Resin) and test out the upgrade.

    1. Upgrade of database: If you didn't manually upgrade your database in the previous step, then your database will be automatically upgraded to the latest version. This may take some time (seconds to minutes), depending on the size of your repository, etc. Check the DSpace log ([dspace]/log/dspace.log.[date]) for information on its status.

    2. Reindexing of all content for search/browse: If your database was just upgraded (either manually or automatically), all the content in your DSpace will be automatically re-indexed for searching/browsing. As the process can take some time (minutes to hours, depending on the size of your repository), it is performed in the background; meanwhile, DSpace can be used as the index is gradually filled. But, keep in mind that not all content will be visible until the indexing process is completed. Again, check the DSpace log ( [dspace]/log/dspace.log.[date]) for information on its status.

       1. If you wish to skip automatic reindexing, please see the Note above under the "Upgrade your Database" step.

18. Check your cron / Task Scheduler jobs.  In recent versions of DSpace, some of the scripts names have changed. 

    1. Check the Scheduled Tasks via Cron documentation for details.  Especially pay attention to the Solr Index optimization commands, which ideally should be run regularly (as noted in the previous step).

    2. WINDOWS NOTE: If you are running the Handle Server on a Windows machine, a new [dspace]/bin/start-handle-server.bat script is available to more easily startup your Handle Server.

Troubleshooting Upgrade Issues

"Ignored" Flyway Migrations

In very rare instances, a Flyway database migration will be "ignored." One known instance of this is documented in DS-3407. If you are upgrading from DSpace 5.x to a later version of DSpace, the migration put in place to address DS-2818 will be "ignored" because it is not necessary. There is a special command you can run which will un-flag this migration as "ignored."

 dspace database migrate ignored