Working With Kafka Topics & Messages

by

Overview

This post covers the fundamental, yet important operations that you or your applications interact with kafka. The main purpose of kafka, to me, is to delivery messages. We are dealing with messages handling in this post.

Setup

As I’ve setup a kafka instance in the previous post, I’m going to use it in this demonstration. I use docker so I can do docker exec to access the kafka’s shell.

As of the time of this writing, the shell commands for kafka are at /bin (kafka confluent image)

As I will work with kafka in the terminal, I need to type the part --bootstrap-server localhost:9092 quite frequently so I prefer to create a variable like so to avoid excessive typing.

export BS="--bootstrap-server localhost:9092"

Let’s get started

Topic managements

List topics

./kafka-topics --bootstrap-server localhost:9092 --list

Create a new topic

./kafka-topics $BS --create --topic user-events --partitions 3 --replication-factor 1

Here I create a topic with 3 partitions and replication factor is 1. As I have only one broker, I cannot specify a replication factor > 1.

You can omit --partitions 3 --replication-factor 1 and the default values are 1.

Describe a topic

You can get the information of a topic using the describe command

./kafka-topics $BS --describe --topic user-events
Describe a topic

increase partition count

As you may already know, one partition can only be consumed by one consumer in a consumer group. If you want to have more throughput, you need to increase the parition count before increasing the consumer.

You can only increase the partition count (no decreasing).

./kafka-topics $BS --alter --topic user-events --partitions 6

Now, if I describe the topic again, I would see 6 partitions

increase partition count to 6

Delete a topic

You can, obviously, delete a topic. Doing so would also delete all the messages.

./kafka-topics $BS --delete --topic user-events

Publishing messages

You can publish a message to a topic using the following command

./kafka-console-producer $BS --topic user-events --property "parse.key=true" --property "key.separator=:"

This will open a prompt for you to enter your message. When publishing a message with a key, it will be used to distribute messages to partitions.

You can exit the publishing prompt by ctrl+c

Consuming messages

Let’s open a separate terminal and start the consumer

./kafka-console-consumer --bootstrap-server localhost:9092 --topic user-events --from-beginning --property "print.key=true" --property "key.separator= -> "

Here, I started a consumer and start consuming from the beginning. You can also start a consumer and specifying a partition and an offset (you cannot specify an offset without a partition)

 ./kafka-console-consumer $BS --topic user-events --partition 0 --offset 0 --property "print.key=true" --property
 "key.separator= -> "

Conclusion

In this post, I’ve shown you some of the fundamental, yet important operations in Kafka.

Leave a Comment