Kafka with Spring Boot: Producers, Consumers, and DLQ

Apache Kafka fits naturally into Spring Boot via spring-kafka. This guide covers a minimal yet production-minded setup.

Producer Configuration

Enable idempotence to avoid duplicate messages during retries:

spring:
  kafka:
    bootstrap-servers: localhost:9092
    producer:
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.springframework.kafka.support.serializer.JsonSerializer
      properties:
        enable.idempotence: true
        acks: all

@Service
public class OrderEventPublisher {
  private final KafkaTemplate<String, OrderEvent> template;
 
  public void publish(OrderEvent event) {
    template.send("orders", event.orderId(), event);
  }
}

Consumer with Retry and DLQ

Route poison pills to a dead-letter topic after retries:

@KafkaListener(topics = "orders", groupId = "order-processor")
public void handle(OrderEvent event) {
  orderService.process(event);
}

spring:
  kafka:
    listener:
      ack-mode: record

Idempotency on the Consumer

Store processed message IDs in your database or Redis so redeliveries are safe:

@Transactional
public void process(OrderEvent event) {
  if (processedIds.contains(event.eventId())) return;
  // business logic
  processedIds.add(event.eventId());
}