Optimizing throughput of Kafka consumers

This article is about ways to optimize the througput of Kafka consumers, specifically using Spring Kafka. With some simple techniques we can improve the throughput by a factor.

Kafka is often marketed as 'low latency' and 'real time', but actually Kafka is designed for througput. Many frameworks don't properly take advantage of this and could perform much better.

Take for example Spring Kafka. Spring Kafka allows you to processe messages from Kafka by adding annotations to methods. This is the canonical example:

@KafkaListener(topics = "demo", groupId = "demo")
    public void oneByOneListener(ConsumerRecord<String, String> record) {
            repository.save(record.key(), record.value());
    }

Assuming the defualt settings, this boils down to:

while(true) {
	var records = consumer.poll()
	for(record in records) {
		try {
			oneByOneListener(record);
		} catch (Exception e) {
			handleError(e);
		}
	}
	consumer.commit();
}

Processing one message at a time, and commits the offset of this message to Kafka when it is processed, effectively acknowledging that the message was processed:

For correctness, this is a good default. When an error occurs during processing of the message, the offset is not committed, Spring Kafka will revert the offset to ensure the message is processed again.

For performance however this is not a good approach.

Commiting the offset takes time and puts some load on the Kafka broker. Another problem is that call to databases or external services have some overhead.

By processing batches of records we can greatly reduce this overhead.

This is also in line with Kafka's design, .poll() returns a batch of records for a reason.

https://docs.spring.io/spring-kafka/reference/kafka/receiving-messages/message-listener-container.html#committing-offsets

Published: 2024-03-14

Tagged: clojure

Chris Blom

Optimizing throughput of Kafka consumers