...
Goal of using Kafka
What is Kafka?
...
see also https://kafka.apache.org/intro
Messaging system | Event streaming |
---|---|
Event streaming thus ensures a continuous flow and interpretation of data so that the right information is at the right place, at the right time. |
|
...
ORCID to VIVO Dataflow through Kafka
...
Dataflow execution
Children Display |
---|
Results
...
In summary
- It has been shown that it is possible to use Kafka to populate VIVO from ORCID
Several points require special attention
- The ORCID ontology needs to be refined and clarified.
- The mapping between ORCID and VIVO also needs to be worked on
- The structure of the Kafka message has to be designed to respect the add/delete/modify record actions
- Several minor bugs need to be fixed in the scripts.
...
Future Plan
- Building a POC VIVO → Kafka → ORCID
- Proving the architecture to operate in event-driven and real-time mode
- Getting POCs to Java
- Redesigning the mapping process, ORCID ontology structure and message structure