Advanced Authority Control (previously Editable Authority Control)

Introduction

Editable Authority Control provides a solution for the management of assigned Authority Control records used within DSpace authority control and assigned to DSpace Item metadata. This solution automatically assigns a unique Authority record that can be used to locally preserve externally controlled vocabulary information when assigned to DSpace Item metadata fields.

The model is designed to provide:

• Workflow for Concepts and Terms: Additional support for designating that a Concept, Term or Thesaurus is ready for “publication and use” via status as “candidate”, “accepted” and so-on.
• Access Control for Metadata Attributes: Additional support to limit viewing of specific metadata attributes (for example authors email addresses, phone numbers, so-on) to only repository admins is supported via a “hidden” attribute.
• Extensibility has been provided to support additional metadata fields on Authority Thesaurus, Concepts, and Terms through Metadata registries and value tables similar to DSpace Item Metadata.
• Reuse of existing DSpace XMLUI Admin UI have been employed to support management and viewing of Authority records. Admin UI Interfaces are now provided to support:
• Creation and Management of new thesauri (for example: dc.types, dc.language, authors, organizations, and so on). Repository Admins will be able to easily produce new Thesauri for use in Authority Control via this use interface.

Creation and Management of Concepts within the Thesauri, including support for creating and assigning Preferred Terms, Non-Preferred Terms and Hidden Terms in DSpace Authority Control Concepts. In addition support for managing the hierarchical and associative relationships between Authority Control Concepts will be available. This will allow support to create and manage hierarchical vocabularies (taxonomies) as well as relate concepts across thesauri (for example, assigning organization membership to authors).
Finally, support for assigning additional metadata to both Concepts and Terms will be have in a manner equivalent to attaching “Notes” to Term and Concept objects (skos and skos-xl namespaces will be employed to facilitate thesaurus specific metadata attributes. Additional attributes will be able to be defined in namespaces decided on my the Repository Administrators (Dublin Core, FoaF, MADS, MODS, so-on)

Versions

DSpace 6.x : A recent production rollout of the 6.0 version of this codebase is deployed on the TXState repository with additional UI enhancements. Added Features in DSpace 6 version of solution include:

• Authority Concept Linking in Discovery: Links will be traversable to profile and used in filters for associated Authority Concept. Authority Concepts can be reached by clicking on a glyphicon located next to the Authority Concepts name. Non administrative users will only have access to view the public Authority Concept profile page. Administrators will be able to view all Concept data.

• Public/Private Authority Concept View (Author / Organization Profiles): Allow for administrator to identify if a Profile should be publicly viewable or not. Feature will be controlled by setting Authority Concept as Public / Private in Authority Concept Record. 

• Hidden Metadata Fields: Allow for Repository Sys Admin to Configure specific Concept Metadata Fields to be publicly Visible or not. Similar to Item Provenance metadata visibility.

• Advanced Edit Forms for Authority Concepts: Similar in design to the Community and Collection Edit forms, this interface will allow for configuration of the fields that should be populated when creating a new Authority Concept manually. Configuration will allow for different forms per Authority Scheme in DSpace (person, organization, type, subject, etc).

  • Example Case: Person (Scheme) Specific Create and Edit Page:

  • Support a standard set of fields that may be configured specifically for Scheme.

    • Required and Optional Fields

    • Multiple Value Fields

    • Fields for Relationships (IsMemberOf Organization)

    • Field titles and Help instructions

    • Support for Controlled Vocabularies, Value Lists

• Merging Authority Concepts: Many Concepts in the current Authority Control are duplicates due to name variants not being matched on Item Deposit.  This required a solution in the Admin UI to support to assure that Duplicates can be easily resolved by merging them with other pre-existing Authority records. Provided support in the Concept view to search for and merge other concepts in the same Scheme. Support will included Curation task queue scheduling to assure that updates to existing Item metadata and discovery indexes.

  
  

DSpace 5.x : Code for AAC was released in concert with DSpace LoD Sesame Repository support for DSpace 5.x located in the following fork of the DSpace codebase: https://github.com/dspace-oceanlink/DSpace/tree/oceanlink-5_x/dspace-aac it is currently deployed into WHOAS DSpace Repository with LoD features included, solution provides:

• Creation and management of "Authority Records" in DSpace database. 
• Integrates Sesame with dspace RDF service to provide out-of-box triplestore support
• Extends DSpace RDF support to include RDF representations of a number of objects in DSpace, including Communities, Collections, Items, Bitstreams, BitstreamFormats, as well as Authority Concepts, Terms and Schemes as SKOS-XL entities.

Data Model Overview

The database schema is available here to see how the two solutions differ between DSpace versions.

DSpace 6.x Data Model

Simplifies previous 5.x Schema for DSpace AAC modelDSpace 5.x EAC data model

• reuse Metadata For All tables, 
• Use UUID in model
• Improved AAC Scheme, Concept and Term model to use new Hibernate support
• Leveraged existing design patterns to create DAO Services for each object type.

-------------------------------------------------------
-- Sequences for creating new IDs (primary keys) for tables.
-------------------------------------------------------
CREATE SEQUENCE conceptrelationtype_seq;
CREATE SEQUENCE conceptrelation_seq;

--------------------------------------------------------------------------------------------------------------
-- ADVANCED AUTHORITY CONTROL ENTITIES
--------------------------------------------------------------------------------------------------------------
CREATE TABLE scheme
(
  uuid              UUID PRIMARY KEY DEFAULT gen_random_uuid() REFERENCES dspaceobject(uuid),
  created           TIMESTAMP WITH TIME ZONE,
  modified          TIMESTAMP WITH TIME ZONE,
  name              VARCHAR(256) UNIQUE,
  lang              VARCHAR(24)
);

CREATE TABLE term
(
  uuid              UUID PRIMARY KEY DEFAULT gen_random_uuid() REFERENCES dspaceobject(uuid),
  created           TIMESTAMP WITH TIME ZONE,
  modified          TIMESTAMP WITH TIME ZONE,
  source            VARCHAR(256),
  status            VARCHAR(256),
  literalForm       TEXT,
  lang              VARCHAR(24),
  hidden            BOOL,
  discoverable      BOOL
);

CREATE TABLE concept
(
  uuid              UUID PRIMARY KEY DEFAULT gen_random_uuid() REFERENCES dspaceobject(uuid),
  created           TIMESTAMP WITH TIME ZONE,
  modified          TIMESTAMP WITH TIME ZONE,
  status            VARCHAR(256),
  lang              VARCHAR(24),
  source            VARCHAR(256),
  hidden            BOOL,
  replaced_by       UUID REFERENCES concept(uuid),
  preferred_term    UUID REFERENCES term(uuid)
);

CREATE TABLE conceptrelationtype
(
  id                INT PRIMARY KEY DEFAULT NEXTVAL('conceptrelationtype_seq'),
  hierarchical      BOOL,
  incoming_label    VARCHAR(64) UNIQUE,
  outgoing_label    VARCHAR(64) UNIQUE
);

--------------------------------------------------------------------------------------------------------------
-- ADVANCED AUTHORITY CONTROL RELATIONS
--------------------------------------------------------------------------------------------------------------
CREATE TABLE scheme2concept
(
  scheme_id   	    UUID REFERENCES scheme(uuid),
  concept_id   	    UUID REFERENCES concept(uuid),
  PRIMARY KEY(scheme_id, concept_id)
);

CREATE TABLE concept2concept
(
  id                INT PRIMARY KEY DEFAULT NEXTVAL('conceptrelation_seq'),
  incoming_id       UUID REFERENCES concept(uuid),
  outgoing_id       UUID REFERENCES concept(uuid),
  relation_type     INT REFERENCES conceptrelationtype(id)
);

CREATE TABLE concept2term
(
  concept_id   	    UUID REFERENCES concept(uuid),
  term_id   	      UUID REFERENCES term(uuid),
  PRIMARY KEY(concept_id, term_id)
);

Functional and Abstract Concepts behind Data Model Approach

Original design background for the Editable Authority Control addon is based research completed by Pete Johnston on aligning ISO-25964 and SKOS-EX as published in the following eFoundations blog articles:

http://efoundations.typepad.com/efoundations/2011/02/term-based-thesauri-and-skos-part-1.html

http://efoundations.typepad.com/efoundations/2011/03/term-based-thesauri-and-skos-part-2-linked-data.html

http://efoundations.typepad.com/efoundations/2011/03/term-based-thesauri-and-skos-part-3-change-over-time-i.html

http://efoundations.typepad.com/efoundations/2011/03/term-based-thesauri-and-skos-part-4-change-over-time-ii.html

The EAC datamodel is based in its general design on important parts of SKOS-XL datamodel.

More  specifically , on the Scheme, Concept and Term entity types that are defined by the model. SKOS-XL extends on SKOS by making Terms entities rather than literal values, allowing for additional descriptive statements to be provided on the model.

The database model for the EAC had been expressed below, in this model, MetadataTerms can be created within MetadataConcept that are  contained  within a MetadataThesaurus (Scheme). All are treated as core DSpaceObject types and as such, also have metadata tables that can be associated with them.

DSpace 6.X Advanced Authority Control

(to be documented)

DSpace 5.X Editable Authority Control

Enable the Editable Authority Control System

Add the flowing configuration into xmlui.xconf to enable the authority management ui.

  <aspect name="Authority" path="resource://aspects/Authority/" />

If not logged in as admin, users can still browse the existing authority objects by using the link

https:// dspace-host /scheme/[id]
https://dspace-host /concept/[id]
https://dspace-host /term/[id]

After login, the management link "Manage Scheme" will show up in the navigation section on the left side of the page

Configuring Choices in the Editable Authority Control System

In the dspace.cfg, add new choices plugins to use new authority scheme as the data type in the item submission.

eg. use the author scheme as the "dc.contributor.author" field type

authority.minconfidence = ambiguous
choices.plugin.dc.contributor.author = SolrAuthorAuthority
choices.presentation.dc.contributor.author = lookup
authority.controlled.dc.contributor.author = true
authority.controlled.txstate.person.email = true
authority.controlled.txstate.person.institution = true
authority.author.indexer.field.1=dc.contributor.author

Database changes

Run the following script ([dspace]/etc/postgres/authority.sql) to add all the new data structure into database.

Migrate Data into Database

Original SolrAuthority data is preserved in Authority Solr Core, a migration tool ([dspace]/bin/dspace import-authority) is provided on the commandline to import the data from the authority into the EAC tables.

This command will take the following solr record

<doc>
<date name="creation_date">2014-06-30T12:37:36.679Z</date>
<str name="email">wjgeerts@txstate.edu</str>
<str name="field">Author</str>
<str name="id">3877CFB67867B422038A1E3970837AFE</str>
<date name="last_modified_date">2014-06-30T12:37:36.717Z</date>
<str name="value">Geerts, Wilhelmus J.</str>
<str name="institute">Louisiana State University</str>
</doc>

The importer will break each record up and record it into its appropriate scheme.

After creation all the concepts, each solr record is updated with additional data from the created Concept.

eg. an "Institute" Authority Control Database Record

TODO

and its associated Solr Authority Index Record.

<doc>
<date name="creation_date">2014-06-30T12:37:36.455Z</date>
<str name="field">Organization</str>
<str name="id">05bdbef2752f48818440235ea05e2399</str>
<date name="last_modified_date">2014-06-30T12:37:36.456Z</date>
<str name="value">Louisiana State University</str>
</doc>
<doc>

eq. an "Author" Authority Control Record\

TODO

and its associated Solr Authority Index Record.

<doc>
<date name="creation_date">2014-06-30T12:37:36.786Z</date>
<str name="field">Author</str>
<str name="id">9E82D970E95930836EBDEF6B0E091A28</str>
<date name="last_modified_date">2014-06-30T12:37:36.788Z</date>
<str name="value">Jackson, Nicole</str>
</doc>

User Interface

After enabling the authority aspect, there will be an link ("Manage Scheme") in the Administration menu representing the management page of the authority system.

Click on this link will show the scheme management page where admin users can create , edit or delete an authority object.

Scheme

Main Page

After click on the "manage scheme" link, the scheme main page will show all the schemes currently in database.

eg. the current database contains two scheme: institute and author.

The institute scheme contains all the instantiation concepts and the author scheme contains all the authors concepts. A author concept can relate to a institute concept if the author is the member of this institute.

Search a scheme

You can search a particular scheme by its identifier.

Type the key word in the search box and hit search will return all the matching scheme on the same page in the search result section.

Create new scheme

Click on the "create new scheme link" on the main management page will lead the admin user to the scheme creation page.

After filling in the information , the system will generate a unique identifier for the new scheme. The new scheme in the main management page.

The new scheme can be used as a source for Authority Control fields in the metadata when configured properly in DSpace choice configuration

TODO

r submission.

eg. the metadata field "dc.contributor.author" is using the author scheme as a data set.

Add the new scheme in the dspace.cfg to configure it as mentioned in the previous section "Configure the choice authority system"

Delete a scheme

On the main scheme management page, check the box in front of each scheme and click delete will delete the whole scheme include all metadata , concepts and terms in the scheme.

View a scheme

Choose one of the scheme from the search result or type the url to the scheme(https://https://digital-test.library.txstate.edu/scheme/[id]) directly in the web browser will to go to the scheme view page:

This page contains all the information about the current scheme including the name, identifier,create date, status, all the metadata fields and a list lists all the concepts that in this scheme.

eg.

The institute scheme contains some institutes concepts that is refereed in the dspace object metadata as "txstate.person.institute".

The author scheme contains some authors concepts that is refereed in the dspace object metadata as "dc.contributor.author".

In the scheme view page, if logged in as administrator, there will be a list of operation links show in the context menu. The current actions relate to a scheme will be "edit scheme attribute", "edit scheme metadata value","search & add concepts".

Edit Scheme Attribute Page

If click on the "edit scheme attibute", you can change the status , identifier , language of a scheme or even delete the scheme.

Edit Metadata Page

Click on the "edit scheme metadata value" will lead the admin user to the metadata editing page for the scheme. Here you can add all the metadata fields that existing in dspace to the current scheme.

Manage Concepts in Current Scheme Page

In order to manage all the concepts in the current scheme, go to the "search & add concepts" link in the scheme view page.

The user will be sent to the main concepts management page.

Search a concept inside a scheme

In the main concepts management page, admin user can search for a concept inside the current scheme by its identifier, prefered term or id.

The page will show all the concepts in the current scheme as a result when first entered, 20 results per page. Admin users can browse all the concepts by hitting the "next page" link. Or type the key word in the search box and hit search will give the user all the matching concepts in this scheme.

Create a New Concept and Add it to Current Scheme

On the main concept management page, admin user can add a new concept to the current scheme.

File up the concept value, top concept,status,language information and hit create concept will generate a concept and a preferred term under the current scheme.

The concept value will be turned into the literal form of the preferred term of this concept.

The status will be used to decide whether this concept should be add to the choice list for user to choose or not. If you want the user to use this concepts value then select "Accepted".

The identifier for this concept will be automatically generated and stored , this value will be used as the authority of the metadata field value.

Delete a concept

On the main concept management page, admin user can delete concepts from the current scheme by checking the check box in front of each concept and click delete. This will delete the concept , metadata fields and its terms.

Concept

Edit

To view a concept, admin user can either type in the concept link

directly in the web browser or find the concept in the scheme and then click the link to the concept in the result list of the main concept management page. See "Search a concept inside a scheme" to get how to find a concept in scheme.

On the concept view page, admin user can see all the information about this concept including parent scheme,attribute(identifier, create date,status,source),preferred terms,alternative terms,metadata(author email),parent concept(eg.institution is the parent concept to the author concept),child concept(eg. author concept is the child concept to the institution concept).

If login in as a admin user, you will see the management link for this concept in the navigation section on the left side of the page. The actions we can do on this concept is "Edit concept attribute","Edit concept metadata value","Add related concept" and "Search & add terms".

Edit Concept Attribute Page

Click on "edit concept attribute" link on the concept view page will lead admin users to the editing page about this concept including changing the status,top concept and language information. Once the concept is created , the identifier will be generated automatically and assigned to it , we don't allow change the identifier of the concept.

Edit Concept Metadata Page

In the editing metadata page for the concept, admin user can add metadata to the concept.

eg. We use metadata field for the author concept to store the email information of the author.We add the metadata field "txstate.person.email" to the author concept.

Add relationships between concepts

In the add concept relationships page, there will be a list of concepts. We only allow add child relationships so if you want to add a parent concept to current concept, you have to go to the parent concept and search for the concept you ware working on.

Type the child concepts' identifier , preferred term or id in the search box and search for the concept you want to add as a child concept of the current concept, and check the check box behind it and hit add. The you will see the new child concept show in the concept view page.

There are three types of relationships between concept and concept: "associate","equal","broader/narrower".

eg. The concept in institute scheme could be the parent of the concept in author scheme, and their relationship type is "Broder/Narower".

Manage Terms in Current Concept Page

In order to manage all the terms in the current concept, go to the "search & add terms" link in the concept view page.

The user will be sent to the main terms management page.

Search a term inside a concept

In the main term management page, admin user can search for a term  inside the current concept by its identifier, literal form or id.

The page will show all the terms in the current concept as a result when first entered, 20 results per page. Admin users can browse all the terms by hitting the "next page" link. Or type the key word in the search box and hit search will give the user all the matching terms  in this concept.

Create a New Term and Add it to Current Concept

On the main term management page, admin user can add a new term to the current concept.

File up the term literal form, preferred term,status,source,language information and hit create term will generate a new term under the current concept.

The identifier for thisterm  will be automatically generated and stored .

Term

Edit

Term View Page

To view a term, admin user can either type in the term link

directly in the web browser or find the term in the concept and then click the link to the term in the result list of the main term management page. See "Search a term inside a concept" to get how to find a term in concept.

On the term view page, admin user can see all the information about this term including parent concept,attribute(identifier, create date,status,literal form),metadata values,parent concept.

If login in as a admin user, you will see the management link for this term in the navigation section on the left side of the page. The actions we can do on this term is "Editterm  attribute","Edit term metadata value".

Edit Term Attribute Page

To change a term's information , go to the edit term attribute page. Here admin users can change the term's literal form, status,source,language or even delete the term. Note we don't allow user to change the identifier of the term once it is generated automatically and stored. Change the literal form of the preferred term will affect the search result for its parent concept.

Edit Term Metadata Page