Summary

Initial step: Initializing Kafka

Install Kafka according to these instructions: https://computingforgeeks.com/configure-apache-kafka-on-ubuntu/

heon@ubuntu:$ kafka-start.sh 
● kafka.service - Apache Kafka Server
Loaded: loaded (/etc/systemd/system/kafka.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2020-12-10 18:23:22 EST; 4 days ago
Docs: http://kafka.apache.org/documentation.html
Main PID: 830 (java)
Tasks: 82 (limit: 14243)
Memory: 799.9M
CGroup: /system.slice/kafka.service
└─830 java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -XX:MaxInlineLevel=15 -Djava.awt.headless=true -Xloggc:/opt/kafka/bin/../l>

.....

heon@ubuntu:$ ./kafka-create-topic.sh 
Created topic vivo-dataconnect-topic

heon@ubuntu:$ ./kafka-list-topic.sh 
vivo-dataconnect-topic

heon@ubuntu:$ ./kafka-describe-topic.sh
Topic: vivo-dataconnect-topic PartitionCount: 1 ReplicationFactor: 1 Configs: segment.bytes=1073741824
Topic: vivo-dataconnect-topic Partition: 0 Leader: 0 Replicas: 0 Isr: 0

heon@ubuntu:$ ./kafka-show-numberof-message.sh 
0

4) Send to Kafka

./kafka-start-producer.sh < ../onto_uqam.n3 
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ... >>>>>>>>>>>>>>>>>>>>heon@ubuntu:$ 

heon@ubuntu:$ ./kafka-show-numberof-message.sh 
25171

5) Receive data and Send to VIVO

Edit the file example_credential.sh and rename it credential.sh

./kafka-populate_vivo.sh 
process offset = 0
There are still 25171 messages to process
Processed a total of 1000 messages
HTTP/1.1 200 
Set-Cookie: JSESSIONID=74DFB2C9827B04BA17D15036751F0EED;path=/vivo;HttpOnly
X-FRAME-OPTIONS: SAMEORIGIN
Content-Type: text/html;charset=UTF-8
Content-Length: 37
Date: Wed, 16 Dec 2020 10:13:11 GMT

<H1>200 SPARQL update accepted.</H1>
process offset = 1000
There are still 24171 messages to process
Processed a total of 1000 messages