Summary
Initial step: Initializing Kafka
Install Kafka according to these instructions: https://computingforgeeks.com/configure-apache-kafka-on-ubuntu/
Code Block |
---|
heon@ubuntu:$ kafka-start.sh
● kafka.service - Apache Kafka Server
Loaded: loaded (/etc/systemd/system/kafka.service; enabled; vendor preset: enabled)
Active: active (running) since Thu 2020-12-10 18:23:22 EST; 4 days ago
Docs: http://kafka.apache.org/documentation.html
Main PID: 830 (java)
Tasks: 82 (limit: 14243)
Memory: 799.9M
CGroup: /system.slice/kafka.service
└─830 java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -XX:MaxInlineLevel=15 -Djava.awt.headless=true -Xloggc:/opt/kafka/bin/../l>
.....
heon@ubuntu:$ ./kafka-create-topic.sh
Created topic vivo-dataconnect-topic
heon@ubuntu:$ ./kafka-list-topic.sh
vivo-dataconnect-topic
heon@ubuntu:$ ./kafka-describe-topic.sh
Topic: vivo-dataconnect-topic PartitionCount: 1 ReplicationFactor: 1 Configs: segment.bytes=1073741824
Topic: vivo-dataconnect-topic Partition: 0 Leader: 0 Replicas: 0 Isr: 0
heon@ubuntu:$ ./kafka-show-numberof-message.sh
0
|
4) Send to Kafka
Code Block |
---|
./kafka-start-producer.sh < ../onto_uqam.n3
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ... >>>>>>>>>>>>>>>>>>>>heon@ubuntu:$
heon@ubuntu:$ ./kafka-show-numberof-message.sh
25171 |
5) Receive data and Send to VIVO
Edit the file example_credential.sh and rename it credential.sh
Code Block |
---|
./kafka-populate_vivo.sh
process offset = 0
There are still 25171 messages to process
Processed a total of 1000 messages
HTTP/1.1 200
Set-Cookie: JSESSIONID=74DFB2C9827B04BA17D15036751F0EED;path=/vivo;HttpOnly
X-FRAME-OPTIONS: SAMEORIGIN
Content-Type: text/html;charset=UTF-8
Content-Length: 37
Date: Wed, 16 Dec 2020 10:13:11 GMT
<H1>200 SPARQL update accepted.</H1>
process offset = 1000
There are still 24171 messages to process
Processed a total of 1000 messages |