All methods in vault-account are triggered by gRPC calls, thus Kafka is not a prerequisite for initiating any actions. Kafka is used to communicate events between microservices. Events are:
generated by vault-account for the vault-account-processor to execute asynchronous jobs.
generated by vault-account-processor for further operations (such as account migrations).
published eagerly by vault-account if the config flag events_disable_eager_publishing is set to FALSE.
published at least once for those recorded in the event journals tables (using the event orchestrators and journal pollers).
picked up by the Core Streaming service to be streamed out to the public.
Account processor events are the most relevant to the communication between vault-account and vault-account-processor. Account processor events for account updates are published by vault-account to the internal topic vault.core.accounts.account_processor.events in an eager manner if enabled; the account-events-orchestrator, account-journal-poller, plan-events-orchestrator and plan-journal-poller are required to publish all account processor events for account updates and plan updates eventually. vault-account-processor awaits jobs by listening to the activity on this topic.
The processor also uses the topic vault.core.accounts.account_update_batch_completed in order to manage account migrations and account update batches.
Non-event topics are also used.
xxxxxxxxxx
[Produces] -> vault.core.account.events
[Produces] -> vault.core.account_update.events
[Produces] -> vault.core.account_update_batch.events
[Produces] -> vault.core.account_plan_assoc.events
[Produces] -> vault.core.accounts.account_processor.events
[Produces] -> vault.core.accounts.account_update_batch_completed
[Produces] -> vault.core.plan.events
[Produces] -> vault.core.plan_update.events
[Produces] -> vault.core.plan_update_batch.events
[Produces] -> vault.core.plan_update_batch_completed
[Produces] -> vault.core.plan_migration.events
[Produces] -> vault.core.flag.events
[Produces] -> vault.core.restriction_set.events
[Produces] -> vault.core.payment_device.events
[Produces] -> vault.core.payment_device_link.events
xxxxxxxxxx
Apache Kafka is an open-source distributed event streaming platform used by thousands of companies for high-performance data pipelines, streaming analytics
Key Features:-
High Throughput Support for millions of messages with modest hardware
Scalability Highly scalable distributed systems with no downtime
Replication Messages are replicated across the cluster to provide support for multiple subscribers and balances the consumers in case of failures
Durability Provides support for persistence of message to disk
Stream Processing Used with real-time streaming applications like Apache Spark & Storm
Data Loss Kafka with proper configurations can ensure zero data loss
xxxxxxxxxx
$ bin/kafka-topics.sh --describe --topic quickstart-events --bootstrap-server localhost:9092
Topic: quickstart-events TopicId: NPmZHyhbR9y00wMglMH2sg PartitionCount: 1 ReplicationFactor: 1 Configs:
Topic: quickstart-events Partition: 0 Leader: 0 Replicas: 0 Isr: 0
xxxxxxxxxx
$ bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server localhost:9092
This is my first event
This is my second event
xxxxxxxxxx
# Start the ZooKeeper service
$ bin/zookeeper-server-start.sh config/zookeeper.properties
xxxxxxxxxx
# Kafka, kafkajs
## Prerequisite
- Knowledge
- Node.JS Intermediate level
- Experience with designing distributed systems
- Tools
- Node.js: [Download Node.JS](https://nodejs.org/en)
- Docker: [Download Docker](https://www.docker.com)
- Kafkajs: [Download VSCode](https://kafka.js.org)
## Commands
- Start Zookeper Container and expose PORT `2181`.
```bash
docker run -p 2181:2181 zookeeper
```
- Start Kafka Container, expose PORT `9092` and setup ENV variables.
```bash
docker run -p 9092:9092 \
-e KAFKA_ZOOKEEPER_CONNECT=<PRIVATE_IP>:2181 \
-e KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://<PRIVATE_IP>:9092 \
-e KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR=1 \
confluentinc/cp-kafka
```
## Code
`client.js`
```js
const { Kafka } = require("kafkajs");
exports.kafka = new Kafka({
clientId: "my-app",
brokers: ["<PRIVATE_IP>:9092"],
});
```
`admin.js`
```js
const { kafka } = require("./client");
async function init() {
const admin = kafka.admin();
console.log("Admin connecting...");
admin.connect();
console.log("Adming Connection Success...");
console.log("Creating Topic [rider-updates]");
await admin.createTopics({
topics: [
{
topic: "rider-updates",
numPartitions: 2,
},
],
});
console.log("Topic Created Success [rider-updates]");
console.log("Disconnecting Admin..");
await admin.disconnect();
}
init();
```
`producer.js`
```js
const { kafka } = require("./client");
const readline = require("readline");
const rl = readline.createInterface({
input: process.stdin,
output: process.stdout,
});
async function init() {
const producer = kafka.producer();
console.log("Connecting Producer");
await producer.connect();
console.log("Producer Connected Successfully");
rl.setPrompt("> ");
rl.prompt();
rl.on("line", async function (line) {
const [riderName, location] = line.split(" ");
await producer.send({
topic: "rider-updates",
messages: [
{
partition: location.toLowerCase() === "north" ? 0 : 1,
key: "location-update",
value: JSON.stringify({ name: riderName, location }),
},
],
});
}).on("close", async () => {
await producer.disconnect();
});
}
init();
```
`consumer.js`
```js
const { kafka } = require("./client");
const group = process.argv[2];
async function init() {
const consumer = kafka.consumer({ groupId: group });
await consumer.connect();
await consumer.subscribe({ topics: ["rider-updates"], fromBeginning: true });
await consumer.run({
eachMessage: async ({ topic, partition, message, heartbeat, pause }) => {
console.log(
`${group}: [${topic}]: PART:${partition}:`,
message.value.toString()
);
},
});
}
init();
```
## Running Locally
- Run Multiple Consumers
```bash
node consumer.js <GROUP_NAME>
```
- Create Producer
```bash
node producer.js
```
```bash
> tony south
> tony north
```