Upgrading DSpace

Backup your DSpace

Before you start your upgrade, it is strongly recommended that you create a backup of your DSpace instance. Backups are easy to recover from; a botched install/upgrade is very difficult if not impossible to recover from. The DSpace specific things to backup are: configs, source code modifications, database, and assetstore. On your server that runs DSpace, you might additionally consider checking on your cron/scheduled tasks, servlet container, and database.

Make a complete backup of your system, including:

• Database: Make a snapshot/dump of the database. For the PostgreSQL database use Postgres' pg_dump command. For example:

  pg_dump -U [database-user] -f [backup-file-location] [database-name]

• Assetstore: Backup the directory ([dspace]/assetstore by default, and any other assetstores configured in the [dspace]/config/dspace.cfg "assetstore.dir" and "assetstore.dir.#" settings)
• Configuration: Backup the entire directory content of [dspace]/config.
• Customizations: If you have custom code, such as themes, modifications, or custom scripts, you will want to back them up to a safe location.

Update Prerequisite Software (as necessary)

DSpace 5.0 requires the following versions of prerequisite software (see Prerequisite Software section of "Installing DSpace" for more details):

• Java 7 (Oracle or OpenJDK)
• Maven 3.0.5 or above
• Database
  • PostgreSQL 9.1 or above, OR 
  • Oracle 10g or above
• Tomcat 7 or above

Upgrade Steps

1. Download DSpace 5.0: Either download DSpace 5.0 from DSpace.org or check it out directly from the Github repository. 
   1. NOTE: If you downloaded DSpace do not unpack it on top of your existing installation. Refer to Installation Instructions, Step 3 for unpacking directives.
2. Merge any customizations (if needed).  If you have made any local customizations to your DSpace installation they may need to be migrated over to the new DSpace. 
   1. NOTE: If you are upgrading across many versions of DSpace at once (e.g. from 1.x.x to 5.x), you may find it easier to first upgrade DSpace, and then attempt to migrate over your various customizations. Because each major version of DSpace tends to add new configurations and features, older customizations may require more work to "migrate" to the latest version of DSpace.
   2. Customizations are typically housed in one of the following places:
      1. JSPUI modifications: [dspace-source]/dspace/modules/jspui/src/main/webapp/
      2. XMLUI modifications: [dspace-source]/dspace/modules/xmlui/src/main/webapp/
      3. Config modifications: [dspace]/config

3. Edit the build.properties file (if needed) ([dspace-source]/build.properties).  Any settings changed in this build.properties file are automatically copied over to the final dspace.cfg file during the "Build DSpace" process (in the next step).  For more information on the build.properties file, see "The build.properties Configuration Properties File" section of the Configuration Reference documentation.

4. Build DSpace. Run the following commands to compile DSpace :

   cd [dspace-source]/dspace/
   mvn -U clean package

   The above command will re-compile the DSpace source code and build its "installer". You will find the result in [dspace-source]/dspace/target/dspace-installer 

   Defaults to PostgreSQL settings

   Without any extra arguments, the DSpace installation package is initialized for PostgreSQL. If you use Oracle instead, you should build the DSpace installation package as follows:
   mvn -Ddb.name=oracle -U clean package

   Enabling and building the DSpace 5 Mirage 2 theme

   Mirage 2 is a responsive theme for the XML User Interface, added as a new feature in DSpace 5. It has not yet replaced the Mirage 1 theme as the XMLUI default theme.

   To enable Mirage 2, add the following to the <themes> section of src/dspace/config/xmlui.xconf, replacing the currently active theme:

   <theme name="Mirage 2" regex=".*" path="Mirage2/" /> 

   It is important to do this before executing the maven build.

   Mirage 2 is not yet activated in the default "mvn package" build. To include it as part of the build, run: 

   mvn -U clean package -Dmirage2.on=true

   The speed of this specific step of the build can be increased by installing local copies of the specific dependencies required for building Mirage 2. The Mirage 2 developer documentation provides detailed instructions for these installations. After the installation of these dependencies, you can choose to run:

   mvn -U clean package -Dmirage2.on=true -Dmirage2.deps.included=false

5. Stop Tomcat (or servlet container). Take down your servlet container. 
   1. For Tomcat, use the $CATALINA_HOME/shutdown.sh script. (Many Unix-based installations will have a startup/shutdown script in the /etc/init.d or /etc/rc.d directories.)

6. Update DSpace Installation. Update the DSpace installation directory with the new code and libraries. Issue the following commands:

   cd [dspace-source]/dspace/target/dspace-installer
   ant update

   
   

7. Update your DSpace Configurations. You should review your configuration for new and changed configurations in DSpace 5.0. 
   1. In the specific case of dspace.cfg it is recommended to start with a fresh copy of the file from the new version and copy your site-specific settings from the old file.  Read the new file carefully to see if you need (or want) other alterations. 
   2. Please notice that as of DSpace 4, the default search and browse support has changed from the old Lucene/DBMS-based method to Discovery.

8. Decide which DSpace Web Applications you want to install. DSpace comes with a variety of web applications (in [dspace]/webapps), each of which provides a different "interface" to your DSpace.  Which ones you install is up to you, but there are a few that we highly recommend (see below):

   1. "xmlui" = This is the XML-based User Interface, based on Apache Cocoon. It comes with a variety of out-of-the-box themes, including Mirage 1 (the default) and Mirage 2 (based on Bootstrap). Between the "xmlui" and "jspui", you likely only need to choose one.

   2. "jspui" = This is the JSPUI-based User Interface, which is based on Bootstrap. Between the "xmlui" and "jspui", you likely only need to choose one.

   3. "solr" (required) = This is Apache Solr web application, which is used by both the "xmlui" and "jspui" for search & browse functionality. It must be installed in support of either UI.

   4. "oai" = This is the DSpace OAI interface. It allows for Metadata and Bitstream (content-file) harvesting, supporting OAI-PMH (Protocol for Metadata Harvest) and OAI-ORE (Object Reuse and Exchange) protocols
   5. "rest" = This is the DSpace REST API
   6. "sword" = This is the DSpace SWORDv1 interface. More info on SWORD protocol and its usage.
   7. "swordv2" = This is the DSpace SWORDv2 interface. More info on SWORD protocol and its usage.
   8. "rdf" (new) = This is the DSpace RDF interface.
   9. "lni" (deprecated) = This is the DSpace Lightweight Networking Interface, supporting WebDAV / SOAP / RPC API.  It is disabled by default as we recommend using REST or SWORD for most activities. In order to build it you must rebuild DSpace with the following flag: mvn package -Pdspace-lni

9. Deploy DSpace Web Applications. If necessary, copy the web applications from your [dspace]/webapps directory to the subdirectory of your servlet container (e.g. Tomcat):

   cp -R [dspace]/webapps/* [tomcat]/webapps/
   

   See the installation guide for full details.

10. Upgrade your database (optional, but recommended). As of DSpace 5, the DSpace code will automatically upgrade your database (from any prior version of DSpace.  By default, this database upgrade occurs automatically when you restart Tomcat (or your servlet container).  However, if you have a large repository or are upgrading across multiple versions of DSpace at once, you may wish to manually perform the upgrade (as it could take some time, anywhere from 5-15 minutes for large sites).

    1. First, you can optionally verify whether DSpace correctly detects the version of your DSpace database. It is very important that the DSpace version is detected correctly before you attempt the migration:

       [dspace]/bin/dspace database info
       # Look for a line at the bottom that says something like:
       # "Your database looks to be compatible with DSpace version ___"

    2. Then, you can upgrade your DSpace database to the latest version of DSpace. (NOTE: check the DSpace log, [dspace]/log/dspace.log.[date], for any output from this command)

       [dspace]/bin/dspace database migrate

    3. More information on this new "database" command can be found in Database Utilities documentation.
11. Restart Tomcat (servlet container). Now restart your servlet container (Tomcat/Jetty/Resin) and test out the upgrade.
    1. Upgrade of database: If you didn't manually upgrade your database in the previous step, then your database will be automatically upgraded to the latest version. This may take some time (seconds to minutes), depending on the size of your repository, etc. Check the DSpace log ([dspace]/log/dspace.log.[date]) for information on its status.
    2. Reindexing of all content for search/browse: If your database was just upgraded (either manually or automatically), all the content in your DSpace will be automatically re-indexed for searching/browsing. As the process can take some time (minutes to hours, depending on the size of your repository), it is performed in the background; meanwhile, DSpace can be used as the index is gradually filled. But, keep in mind that not all content will be visible until the indexing process is completed. Again, check the DSpace log ( [dspace]/log/dspace.log.[date]) for information on its status.

12. Upgrade Solr Indexes. After upgrading DSpace, you should optimize your Solr indexes to ensure that they are in the current format. Solr can upgrade recent formats to its current one. Doing this whenever you upgrade DSpace should keep the index format current.  

    1. Upgrading from DSpace 1.6.x, 1.7.x or 1.8.x: In DSpace 1.x versions, we used and older version of Solr which is no longer compatible with the current version of Solr. Unfortunately, this involves a manual upgrade process, as there's no easy way to automate it.

       1. If you are using an older version of DSpace, you will see errors similar to this one until you manually upgrade your index:

          Caused by: org.apache.lucene.index.IndexFormatTooOldException: Format version is not supported (resource: segment _386q in resource ChecksumIndexInput(MMapIndexInput(path="/space/dspace/solr/statistics/data/index/segments_37m6"))): 2.x. This version of Lucene only supports indexes created with release 3.0 and later.

       2. Manually upgrading your Solr index involves temporarily downloading an older version of Lucene (on which Solr is based), and calling its IndexUpgrader script, e.g.

          # Download Lucene 3.5.0, which can upgrade older Solr/Lucene indexes
          wget "http://search.maven.org/remotecontent?filepath=org/apache/lucene/lucene-core/3.5.0/lucene-core-3.5.0.jar" -O lucene-core-3.5.0.jar
          
          # Then, actually upgrade the indexes by loading the lucene-core-3.5.0.jar and calling IndexUpgrader
          
          # Upgrade the Usage Statistics index. Run this if you have Solr Usage Statistics enabled in your UI.
          java -cp lucene-core-3.5.0.jar org.apache.lucene.index.IndexUpgrader [dspace]/solr/statistics/data/index/
           
          # Upgrade the OAI-PMH indexes. Run this if you use the "oai" webapp.
          java -cp lucene-core-3.5.0.jar org.apache.lucene.index.IndexUpgrader [dspace]/solr/oai/data/index/
           
          # NOTE: You do not need to upgrade the Discovery Search and Browse indexes as they will be automatically rebuilt on upgrade (See previous upgrade step)

       3. At this point in time, your older indexes will now be compatible with Solr / Lucene 3.5.  At this point they are readable by the latest version of Solr. However, you should still optimize each of these indexes using the commands detailed in the "Upgrading from DSpace 3.x or Above" step below.

    2. Upgrading from DSpace 3.x or above: DSpace provides optimization commands for all Solr indexes. Which ones you need to run depend on which features you are using in DSpace. 

       # Optimize Usage Statistics (based on Solr). Run this if you have Usage Statistics enabled in your UI.
       [dspace]/bin/dspace stat-util -o
        
       # Optimize OAI-PMH indexes (based on Solr). Run this if you use the "oai" webapp.
       [dspace]/bin/dspace oai import -o
        
       # NOTE: You should not need to optimize the Discovery Search and Browse indexes, as they will be automatically rebuilt on upgrade (See previous upgrade step)
       # However, you still may wish to schedule optimizing of Discovery Search & Browse (via cron or similar)
       # [dspace]/bin/dspace index-discovery -o

13. Check your cron / Task Scheduler jobs.  In recent versions of DSpace, some of the scripts names have changed. 

    1. Check the Scheduled Tasks via Cron documentation for details.  Especially pay attention to the Solr Index optimization commands, which ideally should be run regularly (as noted in the previous step).