Introduction

DSpace users have expressed the need for DSpace to be able to provide more support for different types of digital objects related to open access publications, such as authors/author profiles, data sets etc. Configurable Entities are designed to meet that need.

In DSpace, an Entity is a special type of Item which often has Relationships to other Entities.  Breaking it down with more details...

Entities and their Relationships are also completely configurable. DSpace provides some sample models out of the box, which you can use directly or adapt as needed.

The Entity model also has similarities with the Portland Common Data Model (PCDM), with an Entity roughly mapping to a "pcdm:Object" and existing Communities and Collections roughly mapping to a "pcdm:Collection".  However, at this time DSpace Entities concentrate more on building a graph structure of relationships, instead of a tree structure.  

Default Entity Models

DSpace currently comes with the following Entity models, both of which are defined in [dspace]/config/entities/relationship-types.xml. These Entity models are not used by default, but may be enabled as described below.

Research Entities

Research Entities include Person, OrgUnit, Project and Publication. They allow you to create author profiles (Person) in DSpace, and relate those people to their department(s) (OrgUnit), grant project(s) (Project) and works (Publication).

Journals

Journal Entities include Journal, Journal Volume, Journal Issue and Publication (article).  They allow you to represent a Journal hierarchy more easily within DSpace, starting at the overall Journal, consisting of multiple Volumes, and each Volume containing multiple Issues.  Issues then link to all articles (Publication) which were a part of that journal issue. 

NOTE: that this model includes the same "Publication" entity as the Research Entities model described above.  This Entity overlap allows you to link an article (Publication) both to its author (Person) as well as the Journal Issue it appeared in.

Enabling Entities

By default, Entities are not used in DSpace. But, as described above several models are available out-of-the-box that may be optionally enabled.  

Keep in mind, there are a few DSpace import/export features that do not yet support Entities in DSpace 7.0.  These will be coming in future 7.x releases.  See DSpace Release 7.0 Status for prioritization information, etc.

1. Configure your entity model (optionally)

As described above, DSpace provides two default entity models defined in [dspace]/config/entities/relationship-types.xml. These models may be used as-is, or modified. 

You can also design your own model from scratch (see "Designing your own model" section below).  So, feel free to start by modifying relationship-types.xml, or creating your own model based on the relationship-types.dtd.

A simple XML configuration file is used to determine the relations. This configuration is read into the database (making modeling the data from the UI possible in future versions as well). A sample of how to define a relationship between Publication and Person is shown below:

<!-- This relationship defines both the Publication and Person Entities, along with how they are related together. -->
<relationships>
    <type>
        <leftType>Publication</leftType>
        <rightType>Person</rightType>
        <leftLabel>isAuthorOfPublication</leftLabel>
        <rightLabel>isPublicationOfAuthor</rightLabel>
        <leftCardinality>
            <min>0</min>
            <!--<max></max> not specified, unlimited-->
        </leftCardinality>
        <rightCardinality>
            <min>0</min>
            <!--<max></max> not specified, unlimited-->
        </rightCardinality>
    </type>
</relationships>

2. Import entity model into the database

In order to enable a defined entity model, it MUST be imported into the DSpace database This is achieved by using the "initialize-entities" script.  The example below will import the "out-of-the-box" entity models into your DSpace installation

# The -f command requires a full path to an Entities model configuration file.
[dspace]/bin/dspace initialize-entities -f [dspace]/config/entities/relationship-types.xml

If an Entity (of same type name) already exists, it will be updated with any new relationships defined in relationship-types.xml

If an Entity (of same type name) doesn't exist, the new Entity type will be created along with its relationships defined in relationship-types.xml

Once imported into the Database, the overall structure is as follows:

Keep in mind, your currently enabled Entity model is defined in your database, and NOT in the "relationship-types.xml".  Anytime you want to update your data model, you'd update/create a configuration (like relationship-types.xml) and re-run the "initialize-entities" command.

3. Configure Collections for each Entity type

Because all Entities are Items, they MUST belong to a Collection.  Therefore, the recommended way to create a different submission forms per Entity type (e.g. Person, Project, Journal, Publication, etc) is to ensure you create a Collection for each Entity Type (as each Collection can have a custom Submission Form).  

  1. Create at least one Collection for each Entity Type needing a custom Submission form. For example, a Collection for "Person" entities, and a separate one for "Publication" entities.
  2. Edit the Collection. On the "Edit Metadata" page, use the "Entity Type" dropdown to select the Entity Type for this Collection.
    1. This "Entity Type" selection will ensure that every Item submitted to this collection is automatically assigned that Entity type.  So, it ties this Collection to that type of Entity (i.e. no other type of Entity can be submitted to this Collection). 
      1. NOTE: Entity Type is currently not modifiable after being set. This is because changing the Entity type may result in odd behavior (or errors) with in-progress submissions (as they will continue to use the old Entity Type).  If you really need to modify the Entity Type, you can do so by changing the "dspace.entity.type" metadata value on the Collection object. At this time, changing that metadata field would need to be done at the database level.
    2. NOTE: In 7.0, this "Entity Type" dropdown did not exist. In that release, you have to create a "Template Item" from that page. In the In the Template Item, add a single metadata field "dspace.entity.type".  Give it a value matching the Entity type (e.g. Publication, Person, Project, OrgUnit, Journal, JournalVolume, JournalIssue).  This value IS CASE SENSITIVE and it MUST match the Entity type name defined in relationship-types.xml
      1. As of 7.1 (or above) , if you previously created a Template Item in 7.0, the "dspace.entity.type" field value will be migrated to the "Entity Type" dropdown automatically (via a database migration).
  3. In the Edit Collection page, switch to the "Assign Roles" tab and create a "Submitters" group.  Add any people who should be allowed to submit/create this new Entity type.
    1. If you only want Administrators to create this Entity type, you can skip this step. Administrators can submit to any Collection.
  4. If you want to hide this Collection, you can choose to only make it visible to that same Submitters group (or Administrators). This does NOT hide the Entities from search or browse, but it will hide the Collection itself.
    1. In the Edit Collection page, switch to the "Authorizations" tab.
    2. Add a new Authorization of TYPE_CUSTOM, restricting "READ" to the Submitters group created above (or Administrators if there is no Submitters group).  You can also add multiple READ policies as needed. WARNING: The Submitters group MUST have READ privileges to be able to submit/create new Entities.
    3. Remove the default READ policy giving Anonymous permissions.
    4. Assuming you want the Entities to still be publicly available, make sure the DEFAULT_ITEM_READ policy is set to "Anonymous"!

Each of these Collections must be contained in a Community.  The simplest thing to do would be to create a new top-level Community, place your entity Collections there, and (optionally) hide the Community in a fashion similar to "hide this Collection" above.

Obviously, how you organize your Entity Types into Collections is up to you.  You can create a single Collection for all Entities of that type (e.g. an "Author Profiles" collection could be where all "Person" Entities are submitted/stored).  Or, you could create many Collections for each Entity Type (e.g. each Department in your University may have it's own Community, and underneath have a "Staff Profiles" Collection where all "Person" Entities for that department are submitted/stored).  A few example structures are shown below.

Example Structure based on the departments:

OR 

Example Structure based on the publication type:

4. Configure Submission Forms for each Entity type

You should have already created Entity-specific Collections in the previous step. Now, we just need to map those Collections to Submission processes specific to each Entity.

On the backend, you will now need to modify the [dspace]/config/item-submission.xml to "map" this Collection (or Collections) to the submission process for this Entity type.

Once your modifications to the submission process are complete, you will need to quickly reboot Tomcat (or your servlet container) to reload the current settings.


4.1 Use of collection-entity-type attribute for default Submission forms per Entity Type

Alternatively to a collection's Handle, Entities Types can be used as an attribute. So, instead of specifying the collection handle, you will need to use the collection-entity-type attribute and what Entity Type to use (like: Person, Project). Please mind that your Collections with Entity Type need to be previously created.

<name-map collection-entity-type="Publication" submission-name="Publication"/>
<name-map collection-entity-type="Person" submission-name="Person"/>
<name-map collection-entity-type="Project" submission-name="Project"/>
<name-map collection-entity-type="OrgUnit" submission-name="OrgUnit"/>
<name-map collection-entity-type="Journal" submission-name="Journal"/>
<name-map collection-entity-type="JournalVolume" submission-name="JournalVolume"/>
<name-map collection-entity-type="JournalIssue" submission-name="JournalIssue"/>


Once your modifications to the submission process are complete, you will need to quickly reboot Tomcat (or your servlet container) to reload the current settings.


Due to the way SubmissionConfigReader is loaded into memory (on a initialize process) currently there is no implemented way to reload submission forms. So, every time you assign an entity type to a collection, or create a new collection with an associated entity type,  you will need to do a Tomcat restart for that collection to be available at the item submission config. There is an on going fix for that.


DSpace 7.6.1 adds a way to reload Submission Configs, so you no longer need to do a Tomcat Restart after creating a new collection with an entity type, or assigning to a existing one.

5. Configure Workflow for each Entity type (optionally)

The DSpace workflow can be used for reviewing all objects in the Object Model since these objects are all Items, and separate collections can be used. The workflow used for e.g. a Person Object can be configured to be identical to a publication, different from a publication, or use no workflow at all.

See Configurable Workflow for more information on configuring workflows per Collection.

6. Configure Virtual Metadata to display for related Entities (optionally)

"Virtual Metadata" is metadata that is dynamically determined (at the time of access) based on an Entity's relationship to other Entities.  A basic example is displaying a Person Entity's name in the "dc.contributor.author" field of a related Publication Entity.  That "dc.contributor.author" field doesn't actually exist on the Publication, but is dynamically added as "virtual metadata" simply because the Publication is linked to the Person (via a relationship).

Virtual Metadata is configurable for all Entities and all relationships.  DSpace comes with default settings for its default Entity model, and those can be found in [dspace]/config/spring/api/virtual-metadata.xml. In that Spring Bean configuration file, you'll find a map of each relationship type to a metadata field & its value.  Here's a summary of how it works:

If the default Virtual Metadata looks good to you, no changes are needed.  If you make any changes, be sure to restart Tomcat to update the bean definitions.

Designing your own Entity model

When using a different entities model, the new model has to be configured an loaded into your repository

Thinking about the object model

First step: identify the entity types

Second step: identify the relationship types

Third step: visualize your model

Configuring the object model

Configure the model in relationship-types.xml

Configuring the metadata fields

Determining the metadata fields to use

Configure the submission forms

Configuring the item display pages

Configuring virtual metadata

Configuring discovery

Additional Technical Details

The original Entities design document is available in Google Docs at: https://docs.google.com/document/d/1wEmHirFzrY3qgGtRr2YBQwGOvH1IuTVGmxDIdnqvwxM/edit  While this content has been copied into the DSpace wiki, the file remains available as needed.

A talk on Configurable Entities was also presented at DSpace 7 at OR2021

Database Storage

The actual type of entity is stored in the "dspace.entity.type" metadata field. 

Flyway will create tables to store the entity type and relationship configuration. The entity type table contains a unique ID and name for the entity type. This table typically only contains a few rows. It uses database indexes to easily and quickly find the entity types.

 Column |         Type          | Modifiers 
--------+-----------------------+-----------
 id     | integer               | not null
 label  | character varying(32) | not null

The relationship type table contains a unique ID, the 2 entity type IDs (foreign keys), the labels for the relation, and the cardinality details (min and max occurrences in each direction). This table typically only contains a few rows. It uses database indexes to easily and quickly find the relations types.

        Column         |         Type          | Modifiers 
-----------------------+-----------------------+-----------
 id                    | integer               | not null
 left_type             | integer               | not null
 right_type            | integer               | not null
 left_label            | character varying(32) | not null
 right_label           | character varying(32) | not null
 left_min_cardinality  | integer               |
 left_max_cardinality  | integer               | 
 right_min_cardinality | integer               | 
 right_max_cardinality | integer               | 

The actual relations are stored in another new database table, containing a row per relation between two entities.

   Column    |  Type   | Modifiers 
-------------+---------+-----------
 id          | integer | not null
 left_id     | uuid    | not null
 type_id     | integer | not null
 right_id    | uuid    | not null
 left_place  | integer | 
 right_place | integer | 

It contains a unique ID, the ID of the relationship type (a foreign key to the table above), and the left and right item (foreign keys to the actual items), and place values to keep track of the position of the entity for the given relationship type. It uses database indexes to easily and quickly find all relations from the current item.

Ordering of Relations (place column)

The “place” columns for relations are similar to the "place" column in the metadatavalue table, which also determines the order of the values. It gets updated automatically with the next number if a relation is created, similar to the current solution to populate the "place" column in the metadatavalue table.

The reason why there's a left and right place separately, is because we don't want the model to restrict which part of the relation has a "place".

Unidirectional example

For a journal, there are relations to the journal volumes as visible in REST. The relevant section is:

https://demo.dspace.org/entities/journalvolume/f9b89a11-b44e-4a64-a3b4-ab24a33553c7

"leftId": "a23eae5a-7857-4ef9-8e52-989436ad2955",
"rightId": "f9b89a11-b44e-4a64-a3b4-ab24a33553c7",
"leftPlace": 1,
"rightPlace": 1,

https://demo.dspace.org/entities/journalvolume/343d3263-2733-4367-9dc4-216a01b4a461

"leftId": "a23eae5a-7857-4ef9-8e52-989436ad2955",
"rightId": "343d3263-2733-4367-9dc4-216a01b4a461",
"leftPlace": 2,
"rightPlace": 1,

For this particular relation, the leftPlace is relevant because the leftId contains the current item ID. First the item with leftPlace 1 is displayed. 
Hereafter the item with leftPlace 2 is displayed.

Bidirectional example

For an article, there are relations to the OrgUnits visible in REST. The relevant section is:

https://demo.dspace.org/entities/orgunit/d30de96b-1e76-40ae-8ef9-ab426b6f9763

"leftId": "96715576-3748-4761-ad45-001646632963",
"rightId": "d30de96b-1e76-40ae-8ef9-ab426b6f9763",
"leftPlace": 1,
"rightPlace": 2,

https://demo.dspace.org/entities/orgunit/c216201f-ed10-4361-b0e0-5a065405bd3e

"leftId": "96715576-3748-4761-ad45-001646632963",
"rightId": "c216201f-ed10-4361-b0e0-5a065405bd3e",
"leftPlace": 2,
"rightPlace": 2,

For this particular relation, the leftPlace is relevant because the leftId contains the current item ID. First the item with leftPlace 1 is displayed. 
Hereafter the item with leftPlace 2 is displayed.

In the other direction, on the OrgUnit item page there are 6 publications visible in REST. The relevant section is:

https://demo.dspace.org/entities/publication/e98b0f27-5c19-49a0-960d-eb6ad5287067

"leftId": "e98b0f27-5c19-49a0-960d-eb6ad5287067",
"rightId": "d30de96b-1e76-40ae-8ef9-ab426b6f9763",
"leftPlace": 1,
"rightPlace": 1,

https://demo.dspace.org/entities/publication/96715576-3748-4761-ad45-001646632963

"leftId": "96715576-3748-4761-ad45-001646632963",
"rightId": "d30de96b-1e76-40ae-8ef9-ab426b6f9763",
"leftPlace": 1,
"rightPlace": 2,

https://demo.dspace.org/entities/publication/2f4ec582-109e-4952-a94a-b7d7615a8c69

"leftId": "2f4ec582-109e-4952-a94a-b7d7615a8c69",
"rightId": "d30de96b-1e76-40ae-8ef9-ab426b6f9763",
"leftPlace": 2,
"rightPlace": 3,

For this particular relation, the rightPlace is relevant because the rightId contains the current item ID. First the item with rightPlace 1 is displayed. Hereafter the item with rightPlace 2 is displayed, …

The relation with rightPlace 2 is the same relation as mentioned for the article above

The default DSpace database tables will not need to be modified as the entity type is part of the regular metadata.

Tilted relationships

The tilted relationships are a default DSpace 7 feature, developed in https://github.com/DSpace/DSpace/pull/3134

It’s designed to improve performance when an entity has 1000s of relationships. It will avoid loading the relationships in the configured direction unless explicitly requested to retrieve them. It’s mainly used for setups where there are so many related items that it doesn’t make sense to list them all, and they are rather made available via search. Using tilted relationships, those setups can get a big performance boost.

The behavior can most easily be explained using a real-life example

Use case: OrgUnit vs Publication relationship

DSpace 7 has an isOrgUnitOfPublication relationship type defined by default. When there are OrgUnits with 1000s or more Publications linked to them, the default isOrgUnitOfPublication relationship can run into performance issues.

When setting the tilted to left, where the left entity type is Publication and the right entity type is OrgUnit, you are specifying that the Publication should still load the related OrgUnits (expected to be a small amount anyway), but the OrgUnit is not supposed to load all the related Publications.

The implications of this setup are:

Impact of the tilted relationships

Performance tests were performed in https://github.com/DSpace/DSpace/pull/3134 (some tests were specific to improvements which are independent from tilted relationships, and which are already applied in all cases)

In general:

Using tilted relationships, there are production environments with single entities having over 40k relationships using a single relationship type. Using the tilted relationships, this setup doesn’t impact the performance.

Using the OrgUnit to Person relationship, it’s possible to have an OrgUnit with 40k Person entities, and the virtual metadata of the Person entity contains the Org Unit’s name, ID, ….

Versioning Support

DSpace entities fully support versioning. For the most part, this works like any other item. For example, when creating a new version of an item, a new item is created and all metadata values of the preceding item are copied over to the new item. Special care was taken to version relationships between entities.

Example of the latest status of a relationship (technical details)

To understand how versioning between entities with relationships works, let's walk through the following example:

Consider Volume 1.1 (left side) and Issue 1.1 (right side). Both are archived and both are the first version. Note that on the arrow, representing the relation between the volume and the issue, two booleans and two numbers are indicated.

With the groundwork out of the way, let's see what happens when we create a new version of volume 1.1. The new version is not yet archived, because it still has to be edited in the submission UI.

At this moment, when viewing the item page of issue 1.1, the user should only see volume 1.1 (as volume 1.2 is not yet archived). When viewing the item page of volume 1.1, nothing has changed: only a link to issue 1.1 will appear. When viewing the item page of volume 1.2 (e.g. as an admin), a link to issue 1.1 will appear as well.

As soon as volume 1.2 is deposited (archived), the "latest status" of both volume 1.1 and volume 1.2 are updated. When viewing the item page of issue 1.1, volume 1.2 should be visible. When viewing the item pages of the volumes, nothing has changed.

Let's create another version of the volume (not archived):

And after archiving volume 1.3:

What happens if we create a new version of issue 1.1?

Only the relationship with volume 1.3 is copied. For issue 1.1, no relationship was displayed with volume 1.1 and 1.2. (The relationships still exist in the database, but are not visible in the UI.). For volume 1.1, a relationship to issue 1.1 remains present, but it should not be updated to issue 1.2. For issue 1.2, these relationships are longer relevant, so they are not copied.

On the item pages of volume 1.1, volume 1.2 and volume 1.3, you should see issue 1.1 (as 1.2 is not archived yet)

Because issue 1.2 is not yet archived, all volumes are still pointing to issue 1.1. Let's archive it:

Now on the item pages of volume 1.1 and volume 1.2, you should see issue 1.1; it's the latest issue at the time that those volumes were superseded by volume 1.3. On the item page of volume 1.3, you'll see issue 1.3. On the item page of issue 1.1 you'll still see volume 1.3 as well.

Metadata fields that represent relations

If you have a closer look at items with relationships, you'll notice two categories of metadata fields that are controlled by DSpace:

Metadata fields of the first category (relation.*) contain all uuids of related items that the current item can see. I.e. a relationship has to exist between the current item and the other item, and the other item needs to have "latest status" for that specific relationship.

As an example take the following state of the previous section:

Item issue 1.1 will contain metadata field relation.isJournalVolumeOfIssue with as value the uuid of volume 1.3. Volume 1.1 and 1.2 are not included because they don't have "latest status" on the relevant relationships.

Metadata fields of the second category (relation.*.latestForDiscovery) contain all uuids of the items for which the current item is visible. I.e. a relationship has to exist between the current item and the other item, and the current item needs to have "latest status" for that specific relationship. These fields are particularly important for indexing and search, because they allow to us to surface all the items that a particular item is referring to.

Continuing on the example above, issue 1.1 will have metadata field relation.isJournalVolumeOfIssue.latestForDiscovery containing the uuids of volume 1.1 and 1.2.

With issue 1.1 containing volume 1.1 and 1.2 in relation.isJournalVolumeOfIssue.latestForDiscovery, a search on the volume 1.1 page for all issues containing volume 1.1 will display issue 1.1 thanks to this setup.

Configure versioning for an entity type

DSpace contains a bunch of example entity types that support versioning out of the box. What follows is an overview of the requirements to make entity versioning work.

  1. when introducing a relationship type, make sure to add four new metadata fields to config/registries/relationship-formats.xml. E.g. relation.isAuthorOfPublication, relation.isAuthorOfPublication.latestForDiscovery, relation.isPublicationOfAuthor and relation.isPublicationOfAuthor.latestForDiscovery
  2. when introducing an entity type, filter items on latestVersion:true in discovery.xml. This will be the default search, which ensures older versions are not shown

Note that versioning support is enabled by default, but can be turned off by setting versioning.enabled = false in versioning.cfg or local.cfg. For more details on item versioning, see: https://wiki.lyrasis.org/display/DSDOC7x/Item+Level+Versioning.