Page History
Warning |
---|
UNMAINTAINED. This documentation has been moved to the official DSpace documentation at COAR Notify - LDN Services |
Table of Contents | ||
---|---|---|
|
Introduction
TheThe Arcadia-funded COAR Notify Project is developing and accelerating community adoption of a standard, interoperable, and decentralised approach to linking research outputs hosted in the distributed network of repositories with resources from external services such as overlay-journals and open peer review services, using linked data notifications. As part of this project, COAR is funding the development of platforms and systems to support the exchange of linked data notifications across partner organisations and the workflows to manage notifications in those platforms and systems. As the largest adopted repository platform in the World one of the first platforms to be addressed is DSpace the implementation of which has been entrusted to 4Science.
The application can receive and send LDN messages concerning items with external systems. The LDN system is the protocol of message exchanging; the Quality Assurance system is the mechanism used to approve or reject item updates.toc
How to enable
Configuration properties involved:
...
ldn.enabled = true|false |
If false the COAR Notify logo in the footer linked to the informative page will not appear.
...
true => message is received and managed the server responds with an HTTP 202 code | |
ldn.notify.inbox = <ldn inbox endpoint> | where the ldn inbox rest service is mapped on the current DSpace instance. IE: ${dspace.server.url}/ldn/inbox |
coar-notify.ip-range.enabled = true|false |
...
enables the validation against the IP of received ldn message against the registered range |
Relation with the Quality Assurance Correction Service
The LDN system, as a message exchanging i/o system, has an inbox and an outbox. Every LDN message refers to a Notify Service: all the Notify Services are configured manually from the admin application form. A Notify Service is just like an authority labelled on LDN messages.
The Quality Assurance system is the implementation of item metadata updates operations. A Quality Assurance Event contains informations for item metadata updates: QAEvent are stored into QAEvent solr collection. All of the QAEvent are shown on an administration form. Every QAEvent can be accepted, ignored or removed: if accepted some metadata of the referred item are modified, if ignored or removed nothing about the item is modified. Every QAEvent has a property named source : qa events created by processing an LDNMessage has source="coar-notify".
To process an LDN message means to create a QAEvents; as soon as the QAEvent is accepted the referred item is updated. The match between LDN message type and the QAEvent topic is configured onto We do it by routing LDN Messages from the queue to the LDN Router. The LDN Router is designed into the spring file /dspace/config/spring/api/ldn-coar-notify.xml on the bean ldnRouter. The router de-multiplex two attributes read from the LDN Message (two string values inside the array called type) content to java classes called Processors: see org.dspace.app.ldn.processor.* package to view them all.
The match between LDN message type and the QAEvent topic (suddenly created by the evaluation of the ldn message from the queue) is configured onto ldn-coar-notify.xml spring file. Every Processor owns a list of actions. An action is often an email to be sent and an action to create the qa event. The LDN processor receives the QAEvent topic as a parameter and creates the relative QAEvent spring file. Every LDN message type has an action related, which owns a list of actions. An action is often an email to be sent and an LDNProcessor to call. The LDN processor receives the QAEvent topic as a parameter and creates the relative QAEvent.
LDN Autodiscovery mechanism
Third party system can retrieve the location of the repository LDN InBox via the LDN autodiscovery mechanism, nevertheless to be able to interact with DSpace they need to be approved by a Repository Administrators and listed in the LDN Services Directory (see next paragraph); otherwise their messages will be flagged as untrusted and not processed at all.
LDN Services Directory
...
(Registering Services)
Anchor | ||||
---|---|---|---|---|
|
You need to register the external services to trust allow the handling of the incoming LDN messages:.
These services are also used to send LDN notification during the submission process of an item (Please take a look at LDN Inbound pattern)
Administrators can do that manage services using the menu LDN Services. Creating
The following page is used to submit a new Notify Service means to submit the following formLDN Service:
Every LDN message is going to have a link referring to the stored notify service. At time of writing it is the LDN Inbox URL. If the incoming LDN message has the LDN inbox url equals to one of the stored Notify Service' ldn_url - the LDN message is stored and queued as a trusted message, otherwise it’s flagged as untrusted.
Untrusted LDN messages are not processed.
LDN Inbox queueing
LDN incoming messages are stored into the ldn_message database table. The incoming LDN messages is stored into a logical queue. The queue_status column of the table contains the status of the LDN message inside the queue. If valued as queued the LDN message will be processed. The LDN Message Extractor is an asynchronous DSpace task that reads the oldest manageable LDN message and route it to be processed; the extractor instance ends as soon as the LDN message extracted is routed and processed or failed to be.
Please consider that this means that the corresponding QAEvent is not automatically created! The corresponding QAEvent is created by the processor instance of the LDN message, invoked by the extractor.
LDN message processing
All the possible incoming LDN messages patterns are documented in the official link: COAR Notify Protocol: Notification Patterns Every LDN message json ends with the array named Type. Inside the DSpace spring configuration settings we store a list of LDN message types to be routed to certain processors. See dspace/config/spring/api/ldn-coar-notify.xml the bean ldnRouter and its incomingProcessors map property. When the extractor finds an LDN message with a Type mismatching any of the processors, the LDN message queue status moves to unmapped.
Unmapped LDN messages are not processed.
At time of writing the processing of an LDN message corresponds to the creation of a QAEvent.
LDN Messages Queue is maintained by two different tasks:
Message Extractor
Timeout Checker
The Extractor pick an LDN message from the queue, assigns a processor and launches it. If the processor ends successfully the LDN message is out out of the queue, updating its queue_status to processed. Also, queue_last_start_time and queue_timeout are updated: queue_last_start_time is updated with the current timestamp, queue_timeout is updated with a value current timestamp + the configuration value of the key ldn.processor.queue.msg.timeout. Also, increases queue_attempts by 1. The extractor does not consider the LDN messages with attempts already done >= the value of the configuration key ldn.processor.max.attempts
The Timeout looks for timed-out LDN messages with queue_status as processing. If attempts < ldn.processor.max.attempts the message is enqueued again, moving its status to queued; otherwise its status is set to failed.
Offer, Acknowledgement and Announce
Considering these possible scenarios here at: COAR Notify Protocol: Example Scenarios
We have to keep the user updated about the item situation. We do it with colored boxes on its landing (handle) page.
When the Offer message (being it review, endorsement or ingest) has been sent as an outgoing LDN message, and nothing else about it has been received, the yellow box is shown.
When the Offer message has been followed by a related incoming Acknowledgement message: if the ack is an accept/tentative accept the box shown is blue, if the ack is a tentative rejct the box shown is red.
When Offer message has been followed by a related Announce incoming message, there are no boxes of this kind shown, because it’s expected for the boxes about suggestions to be shown!
Automatic meaning
Automatic is a flag true/false of Inbound Patterns and are strictly linked to item filter. The Inbound Pattern is inbound from the NotifyService point of view. So they are configuration for LDN messages of the DSpace outbox. Automatic triggers an Outbox LDN message targeting the Notify service and targeting the item in submission. The automatic flag involves only the submission phase of an item. If no item filter is set - the flag is applied on all submissions.
Level of Trust
It’s a number 0 < # < 1. Triggers an automatic approval of QAEvent once the LDN message is extracted. On qaevents.xml spring configuration files the bean qaScoreEvaluation describes three different boundaries defaults to manage the level of trust:
<property name="scoreToReject" value="0.3" />
<property name="scoreToIgnore" value="0.5" />
<property name="scoreToApprove" value="0.8" />
<= rejection deletes the QAEvent;
<= ignore discards the QAEvent;
>= approval accepts the QAEvent automatically, so nobody will see it from the Quality Assurance Page because it’s created and right after approved.
Test calls
Let's trigger an Announce Relationship (see COAR official pattern documentation here).
Firstly configure a Notify Service such as:
By clicking the + New green button fill the form as:
Pick the following POST call from the Postman project (see below to download it and import it to your installed PostMan instance):
Let's evaluate the request' json body (some field is commented) according to the COAR official documentation of this pattern:
Name: a label for the Notify Service used on the UI |
Description: the description of the Notify Service - add details here |
Service URL: the url of the remote Notify Service. This is mostly used as a descriptive URL when sending notification to an external system. The service URL must not be confused with the inbox url. As said this url is descriptive so we expect the main application URL to be added here |
Level of Trust: floating point number value accepted between 0 and 1. This value is used to assign to the service a TRUST value that describes how much reliable the external service is. This value is not used if the automatic processing of QA Events is disabled Check the configuration file dspace/config/spring/api/qaevents.xml to enable the automatic approval When the automatic approval is configured the trust value of the LDN Service is check against configured score thresholds. Thresholds for automatic Approval/Rejection/Ignoring can be set. Also within a specified range QA Events will need a manual approval |
Service IP Range: two IP addresses expected as minimum and maximum. This range is used to validate the received LDN Messages. When the LDN notification doesn't match the configured IP Range the notification is stored as UNTRUSTED |
LDN Inbox URL: this is the url used to send the LDN Notifications. This url is also used when a notification is received to retrieve the registered LDN service it belongs to This URL must is unique among the registered LDN Services |
Inbound Pattern: the section for Inbound pattern is the section which describes what operations/actions are supported by the external service The pattern itself can be selected from the dropdown as there's a list of pattern. To better understand pattern usage please The item filter is used to describe which item can be processed for the current LDN Service/Inbound pattern. The automatic flag when set to true the ldn message exchange is performed automatically right after the item submission has finished. |
LDN Inbox queueing
LDN incoming messages are stored into the ldn_message database table. As far as the property ldn.enabled is true and the incoming json is valid, the LDN Message is stored on the table. Together with the storage of the record, the queue status of the message is initialized. The queue_status column of the table contains the status of the LDN message inside the queue. All the possible queue_status values are described into the java class org.dspace.app.ldn.LDNMessageEntity as integer constants.
Status name | Value in DB | Description |
QUEUE_STATUS_UNTRUSTED_IP QUEUE_STATUS_UNTRUSTED | 0 5 | Message must not be routed as it is not trusted. This may occur if the IP address of the notifications' sender doesn't match the provided "IP Range" or if the service inbox URL in the origin section of the message doesn't match with any registered LDN Service entry's Inbox URL. |
QUEUE_STATUS_QUEUED | 1 | Message is waiting in the queue to be processed by the Extractor |
QUEUE_STATUS_PROCESSING | 2 | Message is currently being processed by the Extractor |
QUEUE_STATUS_PROCESSED | 3 | Message has been successfully processed by the Extractor |
QUEUE_STATUS_FAILED | 4 | Message has been evaluated but its routing has failed |
If all validations are run successfully the LDN Notification status is set as QUEUE_STATUS_QUEUED and the notification will be processed as soon as the extractor retrieves it from the queue.
The LDN Message logical queue is managed by:
- The LDN Message Extractor is an asynchronous DSpace task which retrieves the oldest processable LDN message.
A LDN notification is processable by the Extractor only if its queue status is QUEUE_STATUS_QUEUED other status will not be considered by the extractor.
The extractor process ends as soon as the LDN extracted message is routed and processed (either with success of failure).
- The LDN Message Timeout Checker is an asynchronous DSpace task that looks for timed-out messages with attempts less than Y, where Y= the value of the configuration property ldn.processor.max.attempts. Cron configuration at property ldn.queue.timeout.checker.cron
Each notification if not successfully processed is retrieved in the next execution of the Checker and attempts are kept up to date (increased by 1 and the timeout is increased by X minutes), where X= the value of the configuration property ldn.processor.queue.msg.timeout | defaulted to 60.
Please consider that this means that the corresponding QAEvent is not automatically created as soon as the LDN Message is received.
The QA Event related to the LDN Notification will be created only once the notification is successfully processed by the extractor (LDN queue status set to QUEUE_STATUS_PROCESSED).
Notify status boxes
Considering these possible scenarios here at: COAR Notify Protocol: Example Scenarios
We have to keep the user updated about the item situation. We do it with colored boxes on its landing (handle) page.
When the Offer message (being it review, endorsement or ingest) has been sent as an outgoing LDN message, and nothing else about it has been received, the yellow box is shown.
When the Offer message has been followed by a related incoming Acknowledgement message: if the ack is a tentative rejct the box shown is red.
When Offer message has been followed by a related Announce incoming message, the box shown is blue! It is blue because receiving an Announce means - then the message is extracted - we produce a new QA Event. As every QA Event it is shown on Item Page and myDSpace page.
Configuring automatic QA evaluation using the Level of Trust
Score is a number 0 < # < 1.
If the proper configuration is enabled an automatic check for approval of QAEvents is run once the LDN message is extracted.
On qaevents.xml file the bean qaScoreEvaluation we can configure three different boundaries to automatically approve/reject/ignore the level of trust:
<property name="scoreToReject" value="0.3" />
<property name="scoreToIgnore" value="0.5" />
<property name="scoreToApprove" value="0.8" />
- if score <= rejection deletes the QAEvent;
- if score <= ignore discards the QAEvent;
- if score >= approval accepts the QAEvent automatically
- if score respects the following:
scoreToIgnore
< score <scoreToApprove
no automatic action is performed and the user must check manually the QA event
This feature must not be confused with the automatic triggering for LDN Inbound pattern
Sending LDN Notifications during the submission of an item
In the example below it is shown the submission of an Item with a new section for COAR-Notify.
In this section we can manually choose ldn-services (which were not marked as automatic) in order to ask for Review, Endorsement and Ingest to external services.
If No (non-automatic) pattern has been configured for the LDN Service this service will not be displayed in any of the dropdown.
Automatic Inbound pattern triggering
When adding new LDN Inbound pattern to a LDN Service we can select if a pattern has to be automatic or not.
Flagging a pattern as automatic means that the request described by the current pattern is sent automatically to the external service:
- No action is required during the submission to select the LDN Service for a specific pattern
- The service in not even listed in the COAR notify step of the submission
Item filters for Inbound pattern
When adding inbound pattern to a service we have the possibility to add an item-filter for each described pattern
An item filter allows to filter items according to specified criteria.
- If the pattern is flagged as automatic the item filter is evaluated automatically
- if the item filter is respected the LDN Notification is stored and queued for sending
- if the item filter is not respected the LDN Notification is not even generated an nothing will be sent to the external service
- If the pattern is not flagged as automatic the item filter is evaluated when choosing the service from the dropdown for Notify in the submission
- if the pattern is not respected an error message will be displayed saying the selected LDN service cannot be used for the current item
- if the pattern is respected no error message is displayed
Item Filter is a non-mandatory field. It activates a filter to be applied to the item during the submission.
If the item matches the item filter condition the submission will be successful: otherwise the submission will fail with a talkative message.
Visible item filters are configured into a static list mapped into item-filters.xml file - on a bean named ldnItemFilters. We established for the following item filter to be shown:
Has one Bitstream = the item must have exactly one bundle named "ORIGINAL";
Item is public = anonymous user can see/read the item;
Title starts with pattern = the dc.title metadata of the item must start with 'Pattern';
Type equals dataset = the dc.type metadata of the item must be 'Dataset';
Type equals dataset = the dc.type metadata of the item must be 'Journal Article';
item-filter.xml has all item filters declarations: only the ones declared inside the ldnItemFilters map are shown.
The LDN Consumer
The LDN Consumer is the consume responsible for generating the outgoing LDN Notification.
This consumer is triggered any time an Item is successfully installed and a check against the registered LDN Services/patterns is done to generate the appropriate LDN Notification
This consumer is responsible for the handling of both automatic a non-automatic patterns
event.consumer.ldnmessage.class = org.dspace.app.ldn.LDNMessageConsumer event.consumer.ldnmessage.filters = Item+Install |
---|
Understanding the structure of the LDN Notification
The LDN Notification includes some important section many of these if improperly set might led to errors in evaluations on the notification
Let's take a deeper look at the LDN Notification structure
@context
The context section is the same for any LDN Notification
|
---|
actor
This section describes the actor which performs the request
This is a descriptive section and doesn't include important information
|
---|
context
This is one of the most important section.
In this section we have a description of the Item being involved in the LDN Notification
- id → is the identifier URL of the item in the repository
this field is really important to identify and link the Item to the current notification
other fields are descriptive fields for the item
|
---|
id
The identifier of the LDN notification (Each notification has a different id)
"id": "urn:uuid:94ecae35-dcfd-4182-8550-22c7164fe23f" |
---|
inReplyTo
This field is used to reply to other LDN notification so that the system knows if the current notification is a response
"inReplyTo": "urn:uuid:0370c0fb-bb78-4a9b-87f5-bed307a509dd" |
---|
object
This is a descriptive section of the received Review, Endorsement etc..
- id → is the URL of the review/endorsement etc.
|
---|
origin
This is an important section since it is used (for incoming notifications) to determine the service among the registered ones.
If the service is not recognized as registered the notification is untrusted
The field being used for this check is inbox
|
---|
target
This section is used to describe the target system involved in the notification workflow
|
---|
type
This type section is important since it's determining how to process/evaluate any received notification
|
---|
...
Testing
Request Endorsement (LDN Message on item submission)
Let's configure a new Notify Service such as:
and save.
(NOTE: If you plan to test with Docker, the "Service IP Range" should likely include 172.23.0.1 – this is the IP that your Host machine will likely appear as when running the backend in Docker.)
Create a new Community: | |
Create a new Collection inside the community created above. It is important for this community to have no Entity Type set. Take a look into the item-submission.xml file for the "traditional" steps sequence configuration. Here we've set the coarnotify-step! The traditional step sequence is used as the default, and the default is used for Collection with no entity type associated. If the coar-notify step is needed for every collection possible, the entry-step must be added onto every <submission-process> described into item-submission.xml | |
Create a new Item, select the collection created above. As expected we see the COAR Notify section: here it is the ldn request configuration: comboes for Review, Endorsement and Ingest are filled with the corresponding configured LDN Services. For this example we see only one value for the Endorsement combo (see the Inbound Pattern configured onto the Notify Service named NotifyService Score0.123 - first screenshot of this paragraph) A new LDN Message of type Offer: Endorsement Action is stored and enqueued! As soon as the Extractor manages it, it is enrouted to the Processor configured at ldn-coar-notify.xml (as one of the <outcomingProcessors>). The processor is the bean named outcomingAnnounceEndorsementAction, which lists the SendLDNMessageAction. So, it is only at this point, when the action SendLDNMessageAction is executed - that the notification is sent out to the external Notify Service. To whom the notification is sent out? To the url configured as the LDN Inbox property of the Notify Service (see above: https://notify-inbox.info/inbox/). | |
as soon as our Offer is correctly received on the Notify Service side, the Notify Service has to provide us - asynchronously as a separate Acknowledge LDN message - the Acceptance or Rejection of the Endorsement we offered. When Acceptance is received: The Acknowledge LDN Message - when received - is managed by the Processor configured into the bean named acceptAckAction. When Rejection is received: The Acknowledge LDN Message - when received - is managed by the Processor configured into the bean named rejectAckAction. | Until an Acknowledge or an Announce is received for this item, on the item page you can see the upper yellow window as: |
at time of writing COAR service for test purposes is responding an HTTP 500 as we sent the Offer. This error did not exist back in December 2023 when we firstly developed the LDN implementation. |
Testing Announce Review against current repository
Let's trigger an Request Review (see COAR official pattern documentation here).
For this specific example we can set just the first Inbound Pattern as Request Review and leave the item filter blank. This test is not about item submission.
Pick the following POST call from the Postman project (see below to download it and import it to your installed PostMan instance): let's emulate the Notify Service external system sending us an LDN Message!
For this test to succeed you must update the json below based on your system configuration. See the "IMPORTANT" lines.
{ "@context":[ "https://www.w3.org/ns/activitystreams", "https://purl.org/coar/notify" ], "actor":{ "id":"https://orcid.org/0000-0002-1825-0097", "name":"Josiah Carberry", "type":"Person" }, "id":"urn:uuid:0370c0fb-bb78-4a9b-87f5-bed307a50dff", //// IMPORTANT!! this MUST be a unique value because it's going to be stored as uuid into the ldn_message table. "object":{ |
...
...
1122", ////IMPORTANT!! this |
...
MUST be the URL of an Item in your DSpace repository. It can be either a handle-based URL or a UUID-based URL. "ietf:cite-as":"https://doi.org/10.5555/12345680", "type":"sorg:AboutPage", "url":{ "id":"https:// |
...
...
...
...
...
...
pdf", "mediaType":"application/ |
...
pdf", "type":[ "Article", "sorg: |
...
ScholarlyArticle" ] } } |
...
, "origin":{ "id":"https://research-organisation.org/repository", "inbox":" |
...
https://research-organisation.org/inbox", /// IMPORTANT!! this field MUST match the "LDN Inbox URL" for one of the LDN/Notify Services you've added to your repository. (See previous screenshot) "type":"Service" }, "target":{ "id":"https:// |
...
...
...
com/system", " |
...
inbox":" |
...
...
notify-inbox.info/inbox", "type":"Service" }, " |
...
type": |
...
[ "Offer", "coar-notify:ReviewAction" ] } |
See below the successful response:
This means the LDN Message has been received.
At this point there's not feedback if the notification has been stored & enqueued or just stored as untrusted.
Please remember the enqueued ldn messages must be evaluated by the Extractor for a QA Event (with source=coar-notify) to be created.
As soon as it is processed by the extractor, a blue popup is shown on Item and myDSpace pages:
by clicking on the Check button we are redirected to the Quality Assurance Page for the source coar-notify:
If we navigate to the previous Quality Assurance page we can see the counter for QA Event of coar-notify source increased by 1!
Here it is the page of the specific QA Event in example:
Let's accept it with a click on the Green Check.
Go back to the item page to see the new Review metadata!
Postman Collection for testing COAR-Notify Feature
See below the successful response:
This means the LDN Message has been enqueued. Please remember the enqueued ldn messages must be evaluated by the Extractor.
...
Here’s the Postman collection for all test purposesscenarios:
View file | ||||
---|---|---|---|---|
|
...