This repository contains an optimized Kafka consumer implementation in Go designed to prevent temporary message processing stalls during consumer group rebalances.
To ensure smooth partition handoffs and prevent "stop-the-world" stalls, the following configurations have been applied:
-
Partition Assignment Strategy:
partition.assignment.strategyis set tocooperative-sticky(equivalent toorg.apache.kafka.clients.consumer.CooperativeStickyAssignorin Java). This enables Incremental Cooperative Rebalancing, allowing unaffected partitions to continue processing messages during a rebalance.
-
Consumer Stability Tuning:
max.poll.interval.msis set to300000(5 minutes) to prevent false-positive rebalances from slow processing of message batches.session.timeout.msis set to45000(45 seconds) to handle transient network hiccups without triggering immediate rebalances.heartbeat.interval.msis set to3000(3 seconds) to maintain active group membership and detect failures quickly.
-
Non-Blocking Rebalance Listener:
- The rebalance callback commits offsets asynchronously in a separate goroutine during partition revocation, preventing blocking of the main consumer poll loop.