Kafka Topic Clearing after Producing Messages

[UPDATE: Check out the Kafka Web Console to more easily administer your Kafka topics]


 

This week I’ve been working with the Kafka messaging system in a project.

Basic C# Methods for Kafka Producer

To publish to Kafka I built a C# app that uses the Kafka4n libraries – it doesn’t get much simpler than this:

I was reading from various event and performance monitoring logs and pushing them through just fine.

Basic Python Kafka Consumer

For watching the realtime feed, I created a consumer on a Linux machine using the Python kafka-python package:

This worked great until I started pushing in a lot of data both in size and quantity.  Eventually I started getting an error that seems to relate to the max size my consumer could request:

So I tweaked my publisher to make sure it wasn’t putting in really large messages (which wasn’t needed for my application anyway) and then cleared the Kafka topic.

Clearing Kafka Topics with Python

After trying a few different approaches to clearing the topic, I found this python approach to be simplest, using the zc.zk module.  First I listed the contents from Zookeeper:

Which showed the topics and consumers that were of interest to me .  I found deleting topics was not enough, I had to also reset the consumer data here so my script would not try to pick up where it left off:

To delete the topic and consumers, it only takes a couple more commands:

Then I started streaming new data back into my topic which will be auto-created.

Increasing Buffer Size

The default buffer size for my Python consumer script was set to some small size that prevents it from getting too many messages.

To fix this I add one more line to my Python consumer script, setting the max buffer size.  See what the default is by getting the value for:

Here I set it to 1Mb and it streams on happily for now, setting it to zero seems to let it go infinitely:

References

  1. Kafka messaging system: https://kafka.apache.org/
  2. Kafka4n .NET client: https://github.com/miknil/Kafka4n
  3. Kafka-python module: https://pypi.python.org/pypi/kafka-python/0.9.2
  4. Python Zookeeper module: https://pypi.python.org/pypi/zc.zk

© Tyler Mitchell / Spatialguru.com