Summary
Initial step: Initializing Kafka
Install Kafka according to these instructions: https://computingforgeeks.com/configure-apache-kafka-on-ubuntu/
heon@ubuntu:$ kafka-start.sh ● kafka.service - Apache Kafka Server Loaded: loaded (/etc/systemd/system/kafka.service; enabled; vendor preset: enabled) Active: active (running) since Thu 2020-12-10 18:23:22 EST; 4 days ago Docs: http://kafka.apache.org/documentation.html Main PID: 830 (java) Tasks: 82 (limit: 14243) Memory: 799.9M CGroup: /system.slice/kafka.service └─830 java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -XX:MaxInlineLevel=15 -Djava.awt.headless=true -Xloggc:/opt/kafka/bin/../l> ..... heon@ubuntu:$ ./kafka-create-topic.sh Created topic vivo-dataconnect-topic heon@ubuntu:$ ./kafka-list-topic.sh vivo-dataconnect-topic heon@ubuntu:$ ./kafka-describe-topic.sh Topic: vivo-dataconnect-topic PartitionCount: 1 ReplicationFactor: 1 Configs: segment.bytes=1073741824 Topic: vivo-dataconnect-topic Partition: 0 Leader: 0 Replicas: 0 Isr: 0 heon@ubuntu:$ ./kafka-show-numberof-message.sh 0
4) Send to Kafka
./kafka-start-producer.sh < ../onto_uqam.n3 >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> ... >>>>>>>>>>>>>>>>>>>>heon@ubuntu:$ heon@ubuntu:$ ./kafka-show-numberof-message.sh 25171
5) Receive data and Send to VIVO
Edit the file example_credential.sh and rename it credential.sh
./kafka-populate_vivo.sh process offset = 0 There are still 25171 messages to process Processed a total of 1000 messages HTTP/1.1 200 Set-Cookie: JSESSIONID=74DFB2C9827B04BA17D15036751F0EED;path=/vivo;HttpOnly X-FRAME-OPTIONS: SAMEORIGIN Content-Type: text/html;charset=UTF-8 Content-Length: 37 Date: Wed, 16 Dec 2020 10:13:11 GMT <H1>200 SPARQL update accepted.</H1> process offset = 1000 There are still 24171 messages to process Processed a total of 1000 messages