Search Engine Optimization

Table of Contents:

Ensuring your DSpace is indexed

Anyone who has analyzed traffic to their DSpace site (e.g. using Google Analytics or similar) will notice that a significant (and in many cases a majority) of visitors arrive via a search engine such as Google or other search engines. Hence, to help maximize the impact of content and thus encourage further deposits, it is important to ensure that your DSpace instance is indexed effectively.

DSpace comes with tools that ensure major search engines (Google, Bing, Google Scholar) are able to easily and effectively index all your content. However, many of these tools provide some basic setup.  Here's how to ensure your site is indexed.

For the optimum indexing, you should:

1. Check SEO Validator status to detect any obvious issues

2. Keep your DSpace up to date. We are constantly adding new indexing improvements in new releases
3. Ensure your DSpace is visible to search engines.

4. Ensure your proxy is passing X-Forwarded headers to the User Interface

5. Ensure the user interface is using server-side rendering (enabled by default)
6. Ensure the sitemaps feature is enabled. (enabled by default)
7. Ensure your robots.txt allows access to item "splash" pages and full text.
8. Ensure item metadata appears in HTML headers correctly.
9. Avoid redirecting file downloads to Item landing pages
10. Turn OFF any generation of PDF cover pages
11. As an aside, it's worth noting that OAI-PMH is generally not useful to search engines.  OAI-PMH has its own uses, but do not expect search engines to use it.

Check SEO Validator status

DSpace now has a basic Search Engine Optimization (SEO) validator which can provide you feedback on how well your site may align with the some of these Search Engine Optimization policies. 

At this time, this validation tool can only check three things:

• Ensure the user interface is using server-side rendering
• Ensure the sitemaps feature is enabled and appear to be configured correctly
• Ensure your robots.txt exists and links correctly to the sitemaps (may also detect basic issues with a proxy not passing X-Forwarded headers)

This validation tool can be found in the Admin User Interface on the "Health" page.  Look for the section named "SEO".  If everything looks good, you'll see a green checkbox similar to this:

If there are issues detected, you'll see a red warning with details on what needs to be addressed.

If issues are detected, you should use the documentation on this wiki page to address the detected issues.

Keep your DSpace up to date

We are constantly adding new indexing improvements to DSpace.  In order to ensure your site gets all of these improvements, you should strive to keep it up-to-date. For example:

• As of DSpace 9.0, 8.2, and 7.6.4, a basic SEO validator is now provided on the "Health" page.
• As of DSpace 7.0, Sitemaps are enabled by default (see below)
• As of DSpace 5.0, the DSpace robots.txt file now includes references to Sitemaps by default (see https://github.com/DSpace/DSpace/issues/5302), and also blocks known bad bots (see https://github.com/DSpace/DSpace/issues/5701).
• As of DSpace 4.0, DSpace has provided several enhancements, which were requested by the Google Scholar team. These included providing users (and web indexers) a way to browse content by the date it was added to DSpace (see https://github.com/DSpace/DSpace/issues/4851), ensuring the "dc.date.issued" field is set more accurately (see https://github.com/DSpace/DSpace/issues/4850), and enhancing the logic behind the "citation_pdf_url" HTML <meta> tag (see https://github.com/DSpace/DSpace/issues/4852)
• As of DSpace 1.7, DSpace has improved how its Item-level metadata is made available to Google Scholar. For the 1.7.0 release, the DSpace Developers worked directly with the Google Scholar developers, to ensure DSpace is generating the "citation_*" HTML "<meta>" tags (i.e. Highwire Press tags) that Google Scholar recommends in their Indexing Guidelines.
• As of DSpace 1.5, DSpace has support for sitemaps (both simple HTML pages of links, as well as the sitemaps.org protocol). It also includes item metadata in the HTML HEAD element of item display pages, ensuring that the metadata can be effectively indexed no matter what changes you might have made to your DSpace's layout or style.
• As of DSpace 1.4, DSpace has support for the "if-modified-since" HTTP header. This basically means that if an item (or bitstream therein) has not changed since the last time a search engine's crawler indexed it, that item/bitstream does not have to be re-retrieved, sparing your server.

Additional minor improvements / bug fixes have been made to more recent releases of DSpace.

Ensure your DSpace is visible to search engines

First ensure your DSpace instance is visible, e.g. with: https://www.google.com/webmasters/tools/sitestatus

If your site is not indexed at all, all search engines have a way to add your URL, e.g.:

• Google: http://www.google.com/addurl
• Google Scholar: https://scholar.google.com/intl/en/scholar/inclusion.html
• Bing: https://www.bing.com/webmasters/about

Ensure your proxy is passing X-Forwarded headers to the User Interface

Some HTML tags important for SEO, such as the "citation_pdf_url" tag, require the full URL of your site.  The DSpace user interface will automatically attempt to "discover" that URL using HTTP Headers. 

Because most DSpace sites use some sort of proxy (e.g. Apache web server or Nginx or similar), this requires that the proxy be configured to pass along proper X-Forwarded-* headers, especially X-Forwarded-Host and X-Forwarded-Proto.  For example in Apache HTTPD, you can do something like this:

# This lets DSpace know it is running behind HTTPS and what hostname is currently used
# (requires installing/enabling mod_headers)
RequestHeader set X-Forwarded-Proto https
RequestHeader set X-Forwarded-Host my.dspace.edu

Ensure the user interface is using server-side rendering

In DSpace, Angular's Server Side Rendering (SSR) feature is enabled by default (only when running in production mode). However, it's important to ensure you do not disable it in production mode, as most search engine bots cannot index your site if SSR is disabled.  Per the frontend Installation instructions, you MUST also be running your user interface in production mode (via either npm run serve:ssr or npm start). 

Because the DSpace user interface is based on Angular.io (which is a javascript framework), you MUST have server-side rendering enabled (which is the default) for search engines to fully index your side.  Server-side rendering allows your site to still function even when Javascript is turned off  in a user's browser.  Many web crawlers and bots do not support Javascript (e.g. Google Scholar), so they will only interact with this server-side rendered content.

If you are unsure if server-side rendering (SSR) is enabled, you can check to see if your site is accessible when Javascript is turned off.  For example, in Chrome, you should be able to do the following:

1. Open your site in the Chrome browser
2. Turn off (disable) Javascript using the Chrome instructions: https://developer.chrome.com/docs/devtools/javascript/disable/
3. Click reload in your browser window to reload your site.
   1. If SSR is enabled, then you will still see your site's contents. You should be able to browse & search the site. (Keep in mind, pages may take longer to load because every request requires SSR.)  However, all dynamic menus or actions obviously will not work, as all pages will be static HTML.
   2. If SSR is disabled, then you will see a blank white page. You will not be able to see any content on your site.
4. Don't forget to re-enable Javascript after you are done testing (see link above, or just close that window & reopen a new one)

DSpace use Angular's Server Side Rendering (SSR) for server-side rendering, and it's enabled by default in Production mode via our production environment initialization in src/environments/environment.production.ts:

// Angular Server Side Rendering Settings
ssr: {
  enabled: true,
  ...
},

SSR Performance Tuning

While enabling SSR is extremely important for Search Engine Optimization, it can also be very resource intensive for large pages or highly active sites.  Server Side Rendering involves building the entire HTML for the page in Node.js (on your server) before sending the page back to the client/user.  Most humans only encounter SSR briefly, when they initially visit your site.  However, bots may only interact with SSR, especially if they are unable to process Javascript.  This is true even for Google Scholar, whose bots will only use SSR generated pages to index your site.

In order to maximum the performance of SSR, by default, DSpace will minimize the pages and Angular components that are processed during server side rendering.  You may wish to review the default settings to ensure they are appropriate for your site.  See the "Server Side Rendering (SSR) settings" in User Interface Configuration

Ensure the sitemaps feature is enabled

As of DSpace 7, sitemaps are enabled by default and automatically update on a daily basis.  This is the recommended setup to prefer proper indexing. So, there's nothing you need to do unless you wish to either change their schedule, or disable them.

In the dspace.cfg, the Sitemap generation schedule is controlled by this setting

# By default, sitemaps regenerate daily at 1:15am server time
sitemap.cron = 0 15 1 * * ?

You can modify this schedule by using the Cron syntax defined at https://www.quartz-scheduler.org/api/2.3.0/org/quartz/CronTrigger.html .  Any modifications can be placed in your local.cfg.

If you want to disable this automated scheduler, you can either comment it out, or set it to a single "-" (dash) in your local.cfg

# This disables the automatic updates
sitemap.cron = -

Again, we highly recommend keeping them enabled.  However, you may choose to disable this scheduler if you wish to define these in your local system cron settings.

Once you've enabled your sitemaps, they will be accessible at the following URLs:

• HTML Sitemaps: ${dspace.ui.url}/sitemap_index.html
• XML Sitemaps: ${dspace.ui.url}/sitemap_index.xml

So, for example, if your "dspace.ui.url = https://mysite.org" in your "dspace.cfg" configuration file, then the HTML Sitemaps would be at: "http://mysite.org/sitemap_index.html"

By default, the Sitemap URLs also will appear in your UI's robots.txt (in order to announce them to search engines):

# The URL to the DSpace sitemaps
# XML sitemap is listed first as it is preferred by most search engines
Sitemap: [dspace.ui.url]/sitemap_index.xml
Sitemap: [dspace.ui.url]/sitemap_index.html

The generate-sitemaps command

If you wanted to generate your sitemaps manually, you can use a commandline tool to do so.

WARNING: Keep in mind, you do NOT need to run these manually in most situations, as sitemaps are autoupdated on a regular schedule (see documentation above)

# Commandline option (run from the backend)
[dspace]/bin/dspace generate-sitemaps

This command accepts several options:

You can configure the list of "all search engines" by setting the value of sitemap.engineurls in dspace.cfg.

Create a good robots.txt

DSpace's robots.txt file can be found in the UI's codebase at "src/robots.txt.ejs".  This is an "embedded javascript template" (ejs) file, which simply allows for us to insert variable values into the "robots.txt" at runtime.  It can be edited as a normal text file.

The trick here is to minimize load on your server, but without actually blocking anything vital for indexing. Search engines need to be able to index item, collection and community pages, and all bitstreams within items – full-text access is critically important for effective indexing, e.g. for citation analysis as well as the usual keyword searching.

If you have restricted content on your site, search engines will not be able to access it; they access all pages as an anonymous user.

Ensure that your robots.txt file is at the top level of your site: i.e. at http://repo.foo.edu/robots.txt, and NOT e.g. http://repo.foo.edu/dspace/robots.txt. If your DSpace instance is served from e.g. http://repo.foo.edu/dspace/, you'll need to add /dspace to all the paths in the examples below (e.g. /dspace/browse-subject).

NEVER BLOCK THESE PATHS

Some URLs can be disallowed without negative impact, but be ABSOLUTELY SURE the following URLs can be reached by crawlers, i.e. DO NOT put these on Disallow: lines, or your DSpace instance might not be indexed properly.

• /bitstreams

• /browse/*  (UNLESS USING SITEMAPS)

• /collections

• /communities

• /community-list (UNLESS USING SITEMAPS)

• /entities/*

• /handle

• /items

Example good robots.txt

DSpace comes with an example robots.txt file (which is copied below).  This file can be found at "src/robots.txt.ejs" in the DSpace UI.  This is an "embedded javascript template" (ejs) file, which simply allows for us to insert variable values into the "robots.txt" at runtime.  It can be edited as a normal text file.

The highly recommended settings are uncommented.  Additional, optional settings are displayed in comments – based on your local configuration you may wish to enable them by uncommenting the corresponding "Disallow:" line.

# The URL to the DSpace sitemaps
# XML sitemap is listed first as it is preferred by most search engines
# NOTE: The <%= origin %> variables below will be replaced by the fully qualified URL of your site at runtime.
Sitemap: <%= origin %>/sitemap_index.xml
Sitemap: <%= origin %>/sitemap_index.html

##########################
# Default Access Group
# (NOTE: blank lines are not allowable in a group record)
##########################
User-agent: *
# Disable access to Discovery search and filters; admin pages; processes; submission; workspace; workflow & profile page
Disallow: /search
Disallow: /admin/*
Disallow: /processes
Disallow: /submit
Disallow: /workspaceitems
Disallow: /profile
Disallow: /workflowitems
# Crawlers should be able to access entity pages, but not the facet search links present on entity pages
Disallow: /entities/*?f

# Heavy Discovery facet queries on browse pages
Disallow: /collections/*?f
Disallow: /communities/*?spc.page

# Optionally uncomment the following line ONLY if sitemaps are working
# and you have verified that your site is being indexed correctly.
# Disallow: /browse/*
#
# If you have configured DSpace (Solr-based) Statistics to be publicly
# accessible, then you may not want this content to be indexed
# Disallow: /statistics
#
# You also may wish to disallow access to the following paths, in order
# to stop web crawlers from accessing user-based content
# Disallow: /contact
# Disallow: /feedback
# Disallow: /forgot
# Disallow: /login
# Disallow: /register

# NOTE: The default robots.txt also includes a large number of recommended settings to avoid misbehaving bots.
# For brevity, they have been removed from this example, but can be found in src/robots.txt.ejs

To identify if a specific user agent has access to a particular URL, you can use this handy robots.txt tester.

For more information on the robots.txt format, please see the Google Robots.txt documentation.

Ensure Item Metadata appears in the HTML HEAD

Google Scholar Metadata in HTML HEAD

In addition to Dublin Core <meta> tags in the HTML HEAD, DSpace also includes Google Scholar specific metadata fields in each item's HTML display page.

<meta property="citation_author" content="Tansley, Robert; Donohue, Timothy"/>
<meta property="citation_title" content="Ensuring your DSpace is indexed" />

These meta tags are the "Highwire Press tags" which Google Scholar recommends.  If you have heavily customized your metadata fields, or wish to change the default "mappings" to these Highwire Press tags, you may do so by modifying https://github.com/DSpace/dspace-angular/blob/main/src/app/core/metadata/head-tag.service.ts (see for example the "setCitationAuthorTags()" method in that service class)

Much more information is available in the Configuration section on Google Scholar Metadata Mappings.

Avoid redirecting file downloads to Item landing pages

Make sure that you never redirect "direct file downloads" (i.e. users who directly jump to downloading a file, often from a search engine) to the associated Item's splash/landing page.  In the past, some DSpace sites have added these custom URL redirects in order to facilitate capturing statistics via Google Analytics or similar.

While these URL redirects may seem harmless, they may be flagged as cloaking or spam by Google, Google Scholar and other major search engines. This may hurt your site's search engine ranking or even cause your entire site to be flagged for removal from the search engine.

If you have these URL redirects in place, it is highly recommended to remove them immediately. If you created these redirects to facilitate capturing download statistics in Google Analytics, you should consider upgrading to DSpace 5.0 or above, which is able to automatically record bitstream downloads in Google Analytics (see https://github.com/DSpace/DSpace/issues/5454) without the need for any URL redirects.

Turn OFF any generation of PDF cover pages

While DSpace offers a PDF Citation Cover Page option, this option may affect your content's visibility in search engines like Google Scholar.  Google Scholar (and possibly other search engines) specifically extracts metadata by analyzing the contents of the first page of a PDF.  Dynamically inserting a custom cover page can break the metadata extraction techniques of Google Scholar and may result in all or much of your site being dropped from the Google Scholar search engine.

For more information, please see the "Indexing Repositories: Pitfalls and Best Practices" talk from Anurag Acharya (co-creator of Google Scholar) presented at the Open Repositories 2015 conference.

In general, OAI-PMH is not useful to Search Engines

Feel free to support OAI-PMH, but be aware that in general it is not useful for search engines:

• No reliable way to determine OAI-PMH base URL for a DSpace site.
• No standard or predictable way to get to item display page or full text from an OAI-PMH record, making effective indexing and presenting meaningful results difficult.
• In most cases provides only access to simple Dublin Core, a subset of available metadata.
• NOTE: Back in 2008, Google officially announced they were retiring support for OAI-PMH based Sitemaps. So, OAI-PMH will no longer help you get better indexing through Google. Instead, you should be using the DSpace 'generate-sitemaps' feature described above.

Troubleshooting indexing issues in Google Scholar

DSpace Committers are in touch with the Google Scholar team on a regular basis to improve the coverage of DSpace sites within Google Scholar.  Based on those discussions, there are a variety of common mistakes which Google Scholar has reported may cause issues with indexing your DSpace site.

Please note, if you have followed the SEO guidelines above, then your site should be easy for Google Scholar to index.  However, this troubleshooting section provides additional suggestions, if you are having ongoing coverage problems in Google Scholar.

Be aware of the troubleshooting guidelines published by Google Scholar

These Troubleshooting guidelines can be found in the sidebar menu of the Google Scholar "Inclusion" page.  It's important to read that document to ensure you are aligning with the guidelines published by Google Scholar.  There's also a "Contact us" email provided in these guidelines, if you have more detailed questions for the Google Scholar team.

Be aware that the Google Scholar "site:" operator does not accurately report site coverage

As documented in bullet 2 of the "Troubleshooting" guide published by Google Scholar (emphasis our own)

Keep in mind that the result count of the "site:" operator is not a good indicator of coverage of your website in Google Scholar. First, this operator currently only searches primary versions of the papers. If you're not the primary publisher, some of the papers that you host may not be counted. Second, the result count is usually estimated based on searching a small fraction of the index (the purpose of the result count is to help users refine their queries and not coverage checking). As a result, this estimate may not be accurate. If you're alarmed that the result count for your site is low, please confirm the problem with a more detailed check. We recommend trying to find several dozen of sample papers using search by title.

Because this "site:" operator doesn't provide accurate results, Google Scholar recommends searching a subset (e.g. a few dozen) of representative papers in your DSpace repository by their title to verify whether those papers are findable in Google Scholar or not.  If those papers are all findable, then your coverage is likely reasonable.  If many of those papers are missing, then you should check the other recommendations in this guide to ensure your site is properly configured.

Additionally, keep in mind that Google Scholar will only index content in DSpace which appears to be a scholarly paper/publication. So, if your DSpace uses Entities, then Person, Project, OrgUnit and other similar Entities will not be indexed by Google Scholar. Instead, Google Scholar will concentrate on indexing your Publication entities (or generic Items or other entity types that seem to be papers/publications)

Ensure your site is not blocking the Googlebot crawler

With the increase in the number of aggressive bots/crawlers, some DSpace sites have highly restricted bot access to their DSpace site via firewalls or blocked their DSpace to campus-only access.

If the Googlebot crawler is not able to access your site, then your site will not be included in Google Scholar.

First, double check your "robots.txt" file (e.g. [dspace.ui.url]/robots.txt) to verify that Googlebot is allowed to access your site.  So, for example, if your "dspace.ui.url = https://mysite.org" in your "local.cfg" configuration file, then the robots.txt MUST be available at: "https://mysite.org/robots.txt"

Second, verify (e.g. from your access logs in Apache / Nginx or similar) that you can see the Googlebot is accessing pages on your site, especially your "robots.txt". If you see no signs in your access logs that Googlebot has accessed your site recently, then you may have accidentally blocked its access via firewall settings or similar.

Google provides additional resources for identifying Google common crawlers (like Googlebot) via their domain names or IP ranges.  These resources may allow you to validate which crawler requests are from Google so that you can ensure Googlebot is able to access your site.

Ensure sitemaps are enabled and linked from your "robots.txt"

Make sure you have sitemaps enabled in your DSpace per the instructions above. Google Scholar uses sitemaps heavily to find all the content on your site.

To test if sitemaps are enabled, attempt to visit one of these paths on your DSpace site:

• HTML Sitemaps: [dspace.ui.url]/sitemap_index.html (e.g. https://demo.dspace.org/sitemap_index.html)
• XML Sitemaps: [dspace.ui.url]/sitemap_index.xml (e.g. https://demo.dspace.org/sitemap_index.xml)

So, for example, if your "dspace.ui.url = https://mysite.org" in your "local.cfg" configuration file, then the HTML Sitemap MUST be at: "http://mysite.org/sitemap_index.html"

If visiting those URLs in your DSpace shows a visible HTML/XML file (or downloads a file), then sitemaps are enabled on your site.  However, if those URLs return a 404 error, then this means that Sitemaps are disabled or misconfigured and you should follow the instructions above on how to enable them.

Finally, you should verify that your Sitemap URLs appear in your robots.txt (as this will announce the location of your sitemaps to the Googlebot crawler). If they are not listed in your robots.txt, then Google Scholar may be able to find your sitemaps, which can decrease your coverage.

# The URL to the DSpace sitemaps
# XML sitemap is listed first as it is preferred by most search engines
# Make sure to replace "[dspace.ui.url]" with the URL of your DSpace user interface
# as configured in the "dspace.ui.url" configuration in your local.cfg
Sitemap: [dspace.ui.url]/sitemap_index.xml
Sitemap: [dspace.ui.url]/sitemap_index.html

...

Ensure server-side rendering is enabled

Google Scholar requires that server-side rendering is enabled in your DSpace. Without it, the Googlebot crawler will not be able to access the content on your site.

If you are unsure if server-side rendering (SSR) is enabled, you can check to see if your site is accessible when Javascript is turned off.  For example, in Chrome, you should be able to do the following:

1. Open your site in the Chrome browser
2. Temporarily, turn off (disable) Javascript using the Chrome instructions: https://developer.chrome.com/docs/devtools/javascript/disable/
3. Click reload in your browser window to reload your DSpace site.
   1. If SSR is enabled, then you will still see your site's contents. You should be able to visit the Homepage, Community/Collection pages and Item pages and still see their basic content. However, all dynamic menus or actions obviously will not work, as all pages will be static HTML. (Keep in mind, pages may take longer to load because every request requires SSR.) 
   2. If SSR is disabled, then you will see a blank white page. You will not be able to see any content on the Homepage, Community/Collection pages or Item pages.
4. Don't forget to re-enable Javascript after you are done testing (see link above, or just close that window & reopen a new one)

If you find that server-side rendering is disabled then you MUST enable it via the instructions above, if you want Google Scholar to index your site.