Page History

...

In a faceted search, a user can modify the list of displayed search results by specifying additional "filters" that will be applied on the list of search results. In DSpace, a filter is a contain condition applied to specific facets. In the example below, a user started with the search term "approachhealth", which yielded 15 500 results. By After applying the filter "economicspublic" on the facet "Subject". After applying this filter, only 6 227 results remain. Each time a user selects a sidebar facet it will be added as a filter. Active filters can be altered or removed in the 'filters' section of the search interface.

Image AddedImage Removed

Another example: would beUsing the standard search operation, a user would search for something like \[*wetland + "dc.author=Mitsch, William J" + dc.subject="water quality"* \]. With filtered search, athey user can start by searching for \[*wetland* \], and then filter the results by the other attributes, author and subject.

Discovery Features

• Configurable sidebar Sidebar browse facets that can display be configured to use contents from any metadata field
  • Dynamically generated timespans for dates
• Customizable recent submissions display view on the repository homepage, collection and community pages
• Auto-complete on search termsHit highlighting & search snippets

DSpace 1.8 Improvements

• Configuration moved from dspace.cfg into config/modules/discovery.cfg and config/spring/discovery/spring-dspace-addon-discovery-configuration-services.xml
• Individual communities and collections can have their own Discovery configuration.
• Tokenization for Auto-complete values (see SearchFilter)
• Alphanumeric sorting for Sidebarfacets
• Possibility to avoid indexation of specific metadata fields.
• Grouping of multiple metadata fields under the same SidebarFacet

DSpace 3 Improvements

• Hit highlighting and search snippets support
• Hierarchical facets sidebar facets
• Improved & more intuitive user interface
• Access rights based results
• More like this

Enabling Discovery

As with any upgrade procedure, it is highly recommend that you backup your existing data thoroughly. Although upgrades in versions of Solr/Lucene do tend to be forwards compatible for the data stored in the Lucene index, it is always a best practice to backup your {{\[dspace-install-dir\]/solr/statistics}} cores to assure no data is lost.

1. Enable the Discovery Aspects in the XMLUI by changing the following settings in configconfig/xmlui.xconf
   1. Comment out: SearchArtifacts
   2. Uncomment: Discovery

      Code Block
      XMLxmlXML xml
      <xmlui> <aspects> <!-- @deprecated: the Artifact Browser has been devided into ViewArtifacts, BrowseArtifacts, SearchArtifacts <aspect name="Artifact Browser" path="resource://aspects/ArtifactBrowser/" /> --> <aspect name="Displaying Artifacts" path="resource://aspects/ViewArtifacts/" /> <aspect name="Browsing Artifacts" path="resource://aspects/BrowseArtifacts/" /> <!--<aspect name="Searching Artifacts" path="resource://aspects/SearchArtifacts/" />--> <aspect name="Administration" path="resource://aspects/Administrative/" /> <aspect name="E-Person" path="resource://aspects/EPerson/" /> <aspect name="Submission and Workflow" path="resource://aspects/Submission/" /> <aspect name="Statistics" path="resource://aspects/Statistics/" /> <!-- To enable Discovery, uncomment this Aspect that will enable it within your existing XMLUI Also make sure to comment the SearchArtifacts aspect as leaving it on together with discovery will cause UI overlap issues--> <aspect name="Discovery" path="resource://aspects/Discovery/" /> <!-- This aspect tests the various possible DRI features, it helps a theme developer create themes --> <!-- <aspect name="XML Tests" path="resource://aspects/XMLTest/"/> --> </aspects>

2. Enable the Discovery Indexing Consumer that will update Discovery Indexes on changes to content in XMLUI, JSPUI, SWORD, and LNI in config/dspace.cfg
   1. Add discovery to the list of event.dispatcher.default.consumers

      Code Block
      # default synchronous dispatcher (same behavior as traditional DSpace) event.dispatcher.default.class = org.dspace.event.BasicDispatcher #event.dispatcher.default.consumers = search, browse, eperson, harvester event.dispatcher.default.consumers = search, browse, discovery, eperson, harvester

   2. Change recent.submissions.count to zero

      Code Block
      #Put the recent submissions count to 0 so that discovery can use it's recent submissions, # not doing this when discovery is enabled will cause UI overlap issues #How many recent submissions should be displayed at any one time #recent.submissions.count = 5 recent.submissions.count = 0

3. Check that the port is correct for solr.search.server in config/modules/discovery.cfg
   1. If all of your traffic runs over port 80, then you need to remove the port from the URL

      Code Block
      ##### Search Indexing ##### solr.search.server = http://localhost/solr/search

4. From the command line, navigate to the dspace directory and run the command below to index the content of your DSpace instance into Discovery.

   Code Block
   ./bin/dspace update-discovery-index

   Panel
   NOTE: This step may take some time if you have a large number of items in your repository.

5. Verify if that you now can see the Sidebar Facets on your DSpace homepage. Note that these are only visible when you have items in your repository.

...

• Wiki Markup
  General settings: The *{{discovery.cfg*}} file located in the {{\[dspace-install-dir\]/config/modules directory}}.

• Wiki Markup
  User Interface Configuration: The *{{spring-dspace-addon-discovery-configuration-services.xml*}} file is located in {{\[dspace-install-dir\]/config/spring/discovery/}} directory.

General Discovery settings (config/modules/discovery.cfg)

The {{discovery.cfg}} file is located in the {{\[dspace-install-dir\]/config/modules}} directory and contains following properties:

Modifying the Discovery User Interface (config/spring/spring-dspace-addon-discovery-configuration-services.xml)

The {{spring-dspace-addon-discovery-configuration-services.xml}} file is located in the {{\[dspace-install-dir\]/config/spring}} directory.

Structure Summary

Because this This file is in XML format, you should be familiar with XML before editing this file. The configurations are organized together in beans, depending on the purpose these properties are used for.
This purpose can be derived from the class of the beans. Here's a short summaries summary of classes you will encounter throughout the file and what the corresponding properties in the bean are used for.

...

Default settings

Default settings

In addition to the In addition to the summarized descriptions of the default values, following details help you to better understand these defaults. If you haven't yet, download the configuration file and review it together with the following parameters.
The file contains one default configuration that defines following sidebar facets, search filters, sort fields and recent submissions display:

• Sidebar facets
  • sidebarFacetAuthorsearchFilterAuthor:  groups the metadata fields dc.contributor.author & dc.creator with a facet limit of 10, sorted by occurrence count
  • sidebarFacetSubjectsearchFilterSubject: groups all subject metadata fields (dc.subject.*) with a facet limit of 10, sorted by occurrence count
  • sidebarFacetDateIssuedsearchFilterIssued: contains the dc.date.issued metadata field, which is identified with the type "date" and sorted by specific date values
• Search filters
  • searchFilterTitle: contains the dc.title metadata field and has a tokenized autocomplete
  • searchFilterAuthor: contains the dc.contributor.author & dc.creator metadata fields and has a non tokenized autocomplete configured
  • searchFilterSubject: contains the dc.subject.* metadata fields and has a non tokenized autocomplete configured
  • searchFilterIssued: contains the dc.date.issued metadata field with the type "date" and has a tokenized autocomplete
• Sort fields
  • sortTitle: contains the dc.title metadata field
  • sortDateIssued: contains the dc.date.issued metadata field, this sort has the type date configured.
• defaultFilterQueries
  • The default configuration contains no defaultFilterQueries
  • The default filter queries are disabled by default but there is an example in the default configuration in comments which allows discovery to only return items (as opposed to also communities/collections).
• Recent Submissions
  • The recent submissions are sorted by dc.date. accessioned which is a date and a maximum number of 5 recent submissions are displayed.
• Hit highlighting
  • The fields dc.title, dc.contributor.author & dc.subject can contain hit highlighting.
  • The dc.description.abstract & full text field are used to render search snippets.

Many of the properties contain lists that use references to point to the configuration elements. Many of the properties contain lists which use references to point to the configuration elements. This way a certain configuration type can be used in multiple discovery configurations so there is no need to duplicate thesethem.

...

Search filters & sidebar facets Customization

This section explains the properties of an individual SidebarFacet, like SidebarFacetAuthor, SidebarFacetSubject and SidebarFacetDateIssued from the default configuration. In order to create custom SidebarFacets, you can either modify specific properties of those that already exist or create a totally new one from scratch.

Here's what the SidebarFacetAuthor looks like:

for search filters & sidebar facets. Each sidebar facet must occur in the reference list of the search filters. Below is an example configuration of a search filter that is not used as a sidebar facet.

<bean id="sidebarFacetAuthorsearchFilterTitle" class="org.dspace.discovery.configuration.SidebarFacetConfigurationDiscoverySearchFilter">
        <property name="indexFieldName" value="authortitle"/>
        <property name="metadataFields">
            <list>
                <value>dc.contributor.author<title</value>
        </list>
    </property>
</bean>

The id & class attributes are mandatory for this type of bean. The properties that it contains are discussed below.

• indexFieldName (Required): A unique search filter name, the metadata will be indexed in SOLR under this field name.
• metadataFields (Required): A list of the metadata fields that need to be included in the facet.

Sidebar facets extend the search filter and add some extra properties to it, below is an example of a search filter that is also used as a sidebar facet.

<bean id="searchFilterAuthor" class="org.dspace.discovery.configuration.SidebarFacetConfiguration"    <value>dc.creator</value>
            </list>
        </property>
        <property name="facetLimit" value="10"/>
        <property name="sortOrderindexFieldName" value="COUNTauthor"/>
        <property name="type" value="text"/>
    </bean>

The id & class attributes are mandatory for this type of bean. The properties that it contains are discussed below.

...

metadataFields">
            <list>
                <value>dc.contributor.author</value>
                <value>dc.creator</value>
            </list>
        </property>
        <property name="facetLimit" value="10"/>
        <property name="sortOrder" value="COUNT"/>
        <property name="type" value="text"/>
    </bean>

Note that the class has changed from DiscoverySearchFilter to SidebarFacetConfiguration this is needed to support the extra properties

...

.

• facetLimit (optional): The maximum number of values to be shown. This property is optional, if none is specified the default value '10' will be used. When a type of date is givenIf the filter has the type date, this property will not be used since dates are automatically grouped together.
• sortOrder (optional): The sort order for the sidebar facets, it can either be COUNT or VALUE. If none is given the COUNT value is used as a defaultThe default value is COUNT.
  • COUNT Facets will be sorted by the amount of times they appear in the repository
  • VALUE Facets will be sorted alphanumericalphabetically
• type (optional): the type of the sidebar facet it can either be 'date' or 'text, if none is defined text will be used',  'text' is the default value.
  • text: The facets will be treated as is
  • date: Only the year will be identified from the values and stored in the SOLR index. These years are automatically grouped together and offered as a drill-down browse.

SearchFilter Customization

This section explains the properties of an individual SearchFilter, like searchFilterTitle, searchFilterAuthor, searchFilterSubject and searchFilterIssued from the default configuration. In order to create custom Search Filters, you can either modify specific properties of those that already exist or create a totally new one from scratch.

Here's what the searchFilterAuthor looks like:

• • displayed in ranges that get smaller when you select one.

Hierarchical (taxonomies based) sidebar facets

Hierarchical facets further extend the sidebar facet concept, each metadata field is composed of multiple nodes which are combined by a certain splitter. e.g. group::sub-group::sub-sub-group. The sidebar will only display the top level facets, when clicking on view more all the facet options will be displayed.

<bean id="searchFilterSubject" class="org.dspace.discovery.configuration.HierarchicalSidebarFacetConfiguration

<bean id="searchFilterAuthor" class="org.dspace.discovery.configuration.DiscoverySearchFilter">
    <property name="indexFieldName" value="authorsubject"/>
    <property name="metadataFields">
        <list>
            <value>dc.contributor.author<subject</value>
            <value>dc.creator</value>
        </list>
    </property>
    <property name="fullAutoCompletesortOrder" value="COUNT"/>
    <property name="splitter" value="true::"/>
    <property name="typeskipFirstNodeLevel" value="textfalse"/>
</bean>

The id & class attributes are mandatory for this type of bean. The properties that it contains are discussed below.

• indexFieldName (Required): A unique search filter field name, the metadata will be indexed under this field name
• metadataFields (Required): A list containing the metadata fields which can be used in this filter
• fullAutoComplete (optional): If set to true the values indexed for autocomplete will not be tokenized, if set to false tokenization will occur. Tokenization is the process of breaking up text strings in individual words. In this case, with tokenization activated, a title like "Medical Guidelines" will respond both to the "M" and to the "G", because both words are indexed individually for auto-completion.
• type (optional): the type of the search filter it can either be date or text, if none is defined text will be used.
  • text: The metadata will be treated as is
  • date: With a type of date the dates will receive the following format: yyyy-MM-dd (2011-07-01)

Sort option customization for search results

This section explains the properties of an individual SortConfiguration, like sortTitle and sortDateIssued from the default configuration. In order to create custom sort options, you can either modify specific properties of those that already exist or create a totally new one from scratch.

Here's what the sortTitle SortConfiguration looks like:

Note that the class has changed from SidebarFacetConfiguration to HierarchicalSidebarFacetConfiguration this is needed to support the extra properties.

• splitter (required): The splitter used to split up the separate nodes
• skipFirstNodeLevel (optional): Whether or not to show the root node level. For some hierarchical data there is a single root node. In most cases it doesn't need to be shown since it isn't relevant. This property is true by default.

Sort option customization for search results

This section explains the properties of an individual SortConfiguration, like sortTitle and sortDateIssued from the default configuration. In order to create custom sort options, you can either modify specific properties of those that already exist or create a totally new one from scratch.

Here's what the sortTitle SortConfiguration looks like:

<bean

<bean id="sortTitle" class="org.dspace.discovery.configuration.DiscoverySortFieldConfiguration">
        <property name="metadataField" value="dc.title"/>
        <property name="type" value="text"/>
 </bean>

...

Below is an example of how one of these lists can be configured. It's important that each of the bean references corresponds with to the exact name of the earlier defined Facetsfacets, filters or sort options.

<property name="sidebarFacets">
    <list>
        <ref bean="sidebarFacetAuthor" />
        <ref bean="sidebarFacetSubject" />
        <ref bean="sidebarFacetDateIssued" />
    </list>
</property>

...

<property name="recentSubmissionConfiguration">
    <bean class="org.dspace.discovery.configuration.DiscoveryRecentSubmissionsConfiguration">
        <property name="metadataSortField" value="dc.date.accessioned"/>
        <property name="type" value="date"/>
        <property name="max" value="5"/>
    </bean>
</property>

The property name & the bean class are mandatory. The property field names are discusses below.

...

>
    </bean>
</property>

The property name & the bean class are mandatory. The property field names are discusses below.

• metadataSortField (mandatory): The metadata field to sort on to retrieve the recent submissions
• max (mandatory): The maximum number of results to be displayed as recent submissions
• type (optional): the type of the search filter. It can either be date or text, if none is defined text will be used.

Customizing hit highlighting & search snippets

The hit highlighting configuration element contains all the settings to display search snippets & enable hit highlighting.

Below is an example configuration of the hit highlighting.

<property name="hitHighlightingConfiguration">
    <bean class="org.dspace.discovery.configuration.DiscoveryHitHighlightingConfiguration">
        <property name="metadataFields">
            <list>
                <bean class="org.dspace.discovery.configuration.DiscoveryHitHighlightFieldConfiguration">
                    <property name="field" value="dc.title"/>
                    <property name="snippets" value="5"/>
                </bean>
                <bean class="org.dspace.discovery.configuration.DiscoveryHitHighlightFieldConfiguration">
                    <property name="field" value="dc.contributor.author"/>
                    <property name="snippets" value="5"/>
                </bean>
                <bean class="org.dspace.discovery.configuration.DiscoveryHitHighlightFieldConfiguration">
                    <property name="field" value="dc.subject"/>
                    <property name="snippets" value="5"/>
                </bean>
                <bean class="org.dspace.discovery.configuration.DiscoveryHitHighlightFieldConfiguration">
                    <property name="field" value="dc.description.abstract"/>
                    <property name="maxSize" value="250"/>
                    <property name="snippets" value="2"/>
                </bean>
                <bean class="org.dspace.discovery.configuration.DiscoveryHitHighlightFieldConfiguration">
                    <property name="field" value="fulltext"/>
                    <property name="maxSize" value="250"/>
                    <property name="snippets" value="2"/>
                </bean>
            </list>
        </property>
    </bean>
</property>

The property name & the bean class are mandatory. The property field names are:

• field (mandatory): The metadata field to be highlighted (can also be * if all the metadata fields should be highlighted).
• maxSize (optional): Limit the number of characters displayed to only the relevant part (use metadata field as search snippet).
• snippets (optional): The maximum number of snippets that can be found in one metadata field.

Hit highlighting technical details

The org.dspace.discovery.DiscoveryQuery object has a setter & getter for the hit highlighting configuration configured in the discovery configuration. If this configuration is given the resolveToSolrQuery method located in the org.dspace.discovery.SolrServiceImpl class will use the standard solr highlighting feature (http://wiki.apache.org/solr/HighlightingParameters). The org.dspace.discovery.DiscoverResult class has a method to set the highlighted fields for each object & field.

The rendering of search results is no longer handled by the mets format but uses a special type of list named "TYPE_DSO_LIST". Each metadata field (& fulltext if configured) is added in the DRI and IF the field contains hit higlighting the java code will split up the string & add _DRI highlights_ to the list. The xsl for the themes also contains special rendering xsl for the DRI, for Mirage the changes have been located in the _discovery.xsl_ file. For themes using the old structural.xsl look for the template matching "_dri:list\[@type='dsolist'\]_".

More like this configuration

The 'more like this'-configuration element contains all the settings for displaying the related items on an item display page.
Below is an example of the more like this configuration.

<property name="moreLikeThisConfiguration">
    <bean class="org.dspace.discovery.configuration.DiscoveryMoreLikeThisConfiguration">
        <property name="similarityMetadataFields">
            <list>
                <value>dc.contributor.author</value>
                <value>dc.creator</value>
                <value>dc.subject</value>
            </list>
        </property>
        <!--The minimum number of matching terms accross the metadata fields above before an item is found as related -->
        <property name="minTermFrequency" value="5"/>
        <!--The maximum number of related items displayed-->
        <property name="max" value="3"/>
    </bean>
</property>

The property name & the bean class are mandatory. The property field names are discusses below.

• similarityMetadataFields: the metadata fields checked for similarity
• minTermFrequency: The minimum number of matching terms accross the metadata fields above before an item is found as related
• max: The maximum number of related items displayed

More like this technical details

The org.dspace.discovery.SearchService object has received a getRelatedItems() method. This method requires an item & the more-like-this configuration bean from above. This method is implemented in the org.dspace.discovery.SolrServiceImpl which uses the item as a query & uses the default Solr parameters for more-like-this to pass the bean configuration to solr (http://wiki.apache.org/solr/MoreLikeThis). The result will be a list of items or if none found an empty list. The rendering of this list is handled in the org.dspace.app.xmlui.aspect.discovery.RelatedItems class.

Access item based results

The items returned by discovery are all the items the user logged in has access to. So the results may differ if you are logged in. This feature can be switched off it isn't requested by going to the \[dspace.dir\]/config/spring/discovery/spring-dspace-addon-discovery-solr-plugin-services.xml file & commenting out the bean & the alias shown below.

<bean class="org.dspace.discovery.SolrServiceResourceRestrictionPlugin" id="solrServiceResourceIndexPlugin"/>

<alias name="solrServiceResourceIndexPlugin" alias="org.dspace.discovery.SolrServiceResourceRestrictionPlugin"/>

Access item based results technical details

The DSpaceObject class has an updateLastModified() method which will be triggered each time an authorization policy changes. This method is only implemented in the item class where the last_modified timestamp will be updated and a modify event will be fired. By doing this we ensure that the discovery consumer is called and the item is reindexed. Since this feature can be switched off a separate plugin has been created: the SolrServiceResourceRestrictionPlugin. Whenever we reindex a DSpace object all the read rights will be stored in the read field. We make a distinction between groups and users by adding a 'g' prefix for groups and the 'e' prefix for epersons.

When searching in discovery all the groups the user belongs to will be added as a filter query as well as the users identifier. If the user is an admin all items will be returned since an admin has read rights on everything

...

.

Discovery SOLR Index Maintenance

...

Discovery is built as an application layer on top of the Open Source Enterprise Search Server SOLR. ThereforeTherefor, SOLR configuration can be applied to the SOLR cores that are shipped with DSpace.
The DSpace SOLR instance itself now runs two cores. One for collection DSpace Solr based "statistics", the other for Discovery Solr based "search".

...