Apache Kafka CLI Tutorial (2025 Updates)

Apache Kafka, a distributed streaming platform, is a cornerstone of modern data infrastructure. While application code often interacts with Kafka, command-line proficiency is crucial for initial setup, testing, debugging, and cluster health monitoring. This article serves as a practical guide to the essential Kafka command-line interface (CLI) commands, empowering you to manage your Kafka environment effectively.

Setting Up Your Environment

Before embarking on the journey of Kafka CLI commands, ensure your environment is properly configured. This involves verifying that both ZooKeeper and your Kafka brokers are running.

ZooKeeper Validation: The first step is to ensure your ZooKeeper service is active and listening on the expected port (typically 2181). This can be quickly verified using the telnet command:

      telnet localhost 2181
    

A successful connection signifies that ZooKeeper is operational and ready to provide metadata services to Kafka. If you encounter an error, troubleshoot your ZooKeeper installation and configuration.

Kafka Broker Validation: Next, validate that your Kafka broker is running and listening on its designated port (typically 9092). Again, use telnet:

      telnet localhost 9092
    

A successful connection indicates that the Kafka broker is accepting connections. Resolve any connection issues before proceeding.

Locating the Kafka CLI Scripts: Once you've confirmed both services are running, navigate to the Kafka bin directory. This directory houses the various CLI scripts. The exact path depends on your Kafka installation location. A common example might be /opt/kafka/bin.

Managing Kafka Topics with kafka-topics.sh

The kafka-topics.sh script is your primary tool for managing Kafka topics. This script allows you to create, list, describe, and delete topics, providing complete control over your topic structure.

Creating a Topic: To create a new Kafka topic, use the following command. Remember to replace placeholders with your desired values.

      kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic my-new-topic
  

• --create: Specifies the creation operation.

• --zookeeper localhost:2181: Indicates the ZooKeeper connection string. Adjust if your ZooKeeper is running on a different host or port.

• --replication-factor 1: Sets the replication factor to 1. This means each partition's data will be replicated only once (to ensure high availability, consider increasing this value in a production setting).

• --partitions 1: Defines the number of partitions for the topic. More partitions can improve throughput and parallelism, but also increase management complexity.

• --topic my-new-topic: Specifies the name of the topic you're creating. Choose a descriptive name relevant to the data stream.

Listing Topics: View all existing topics using this concise command:

      kafka-topics.sh --list --zookeeper localhost:2181
    

Describing a Topic: Get detailed information about a specific topic, such as its partitions and replication factor:

      kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-new-topic
    

Deleting a Topic: Remove a topic completely:

      kafka-topics.sh --delete --zookeeper localhost:2181 --topic my-new-topic
    

Important Note: Remember that topic metadata and partition data are physically stored in the directories specified by the log.dirs property within your server.properties file (e.g., /tmp/kafka-logs).

Producing Messages with kafka-console-producer.sh

The kafka-console-producer.sh script enables you to send messages to your Kafka topics interactively. This is invaluable for testing and debugging purposes.

Starting the Producer: Begin producing messages with the following command, ensuring that your topic already exists:

      kafka-console-producer.sh --bootstrap-server localhost:9092 --topic my-new-topic

• --bootstrap-server localhost:9092: Provides the address of one of your Kafka brokers. Modify this if your broker is hosted elsewhere.

• --topic my-new-topic: Specifies the topic to which messages will be sent.

Once the command is executed, you'll be presented with a prompt where you can type and send messages. Each line of text entered will be sent as a separate message.

Consuming Messages with kafka-console-consumer.sh

The kafka-console-consumer.sh script is the counterpart to the producer, allowing you to retrieve messages from your Kafka topics.

Consuming Messages from the Beginning: To view all messages in a topic, including those produced before the consumer started, use the --from-beginning flag:

      kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-new-topic --from-beginning
    

Consuming Only New Messages: To only consume messages produced after the consumer starts, omit the --from-beginning flag:

      kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic my-new-topic
    

Understanding Kafka Log Files

Messages within Kafka topics reside in partition-specific directories within the log.dirs path. For example, messages from my-new-topic, partition 0 would typically be found under /tmp/kafka-logs/my-new-topic-0/.

Inside each partition directory, you'll find three crucial files:

• .log: This file contains the actual message data.

• .index: This file maps message offsets to their physical positions within the .log file, enabling efficient retrieval.

• .timeindex: This file maps timestamps to offsets, facilitating time-based data retrieval.

A Practical Workflow: A common workflow involves these steps:

1. Topic Creation and Validation: Create a topic using kafka-topics.sh --create, then describe it using kafka-topics.sh --describe to verify the settings. Also, inspect the log.dirs directory to confirm the topic's presence.

2. Message Production: Start the producer (kafka-console-producer.sh) and send several test messages.

3. Message Consumption: Verify message delivery by consuming them using kafka-console-consumer.sh.

4. Log Monitoring: Inspect the log files within the log.dirs directory to ensure that messages have been written and stored correctly.

Tips and Best Practices

ZooKeeper Dependency: Remember that most Kafka CLI commands rely heavily on ZooKeeper for metadata management. Ensure ZooKeeper is running and accessible throughout your CLI operations.

Bootstrap Server Configuration: Producer and consumer scripts require the address of at least one Kafka broker. For a single-node setup, localhost:9092 is usually sufficient, but adjust this for multi-node clusters.

Debugging and Validation: Leverage the Kafka CLI extensively for debugging and validating your Kafka setup, particularly during development. The commands provide direct insight into the state of your topics, producers, and consumers.

Advanced Kafka Operations

Once you've mastered the fundamental CLI commands, you can explore more advanced features. Kafka Connect allows you to integrate Kafka with various external systems, while Kafka Streams provides tools for real-time stream processing. For production environments, consider automating these CLI workflows or integrating them with monitoring systems for enhanced operational efficiency.

Conclusion

The Apache Kafka command-line interface provides an indispensable set of tools for managing and troubleshooting your Kafka deployments. From creating and managing topics to producing and consuming messages, the CLI offers precise control and valuable insights into your Kafka environment. Understanding and mastering these commands is paramount for effective Kafka administration and development.