Multiple Sentry consumers repeatedly become unhealthy due to Kafka coordinator/session timeouts

[douglasndm]

Self-Hosted Version

26.8

CPU Architecture

x86_64

Docker Version

29.7.2

Docker Compose Version

v5.5.0

Machine Specification

• My system meets the minimum system requirements of Sentry

Installation Type

Upgrade from 26.7.2

Steps to Reproduce

1. Deploy a self-hosted Sentry installation using the standard Docker Compose configuration.

2. Start all Sentry services and verify that the consumers initially become healthy.

3. Leave the Sentry installation running under normal workload.

4. Allow the Sentry consumers to process events, performance metrics, buffered segments, monitor tasks, and other Kafka topics.

5. After some time, multiple consumers begin losing communication with the Kafka consumer group coordinator.

6. The issue can be observed in the logs with messages such as:

   Consumer group session timed out (in join-state steady) after 45000 ms
   without a successful response from the group coordinator
   

7. The consumers subsequently report errors such as:

   KafkaError{code=NOT_COORDINATOR}
   KafkaError{code=_WAIT_COORD}
   KafkaError{code=COORDINATOR_LOAD_IN_PROGRESS}
   

8. The affected containers eventually become unhealthy.

9. Restarting the Sentry stack temporarily restores the consumers to a healthy state.

10. The problem recurs several times per day, requiring approximately 4 Sentry restarts per day.

Affected containers include:

sentry-snuba-subscription-consumer-events-1
sentry-process-segments-1
sentry-post-process-forwarder-errors-1
sentry-monitors-clock-tasks-1
sentry-ingest-monitors-1
sentry-generic-metrics-consumer-1
sentry-events-consumer-1

The issue also affects Snuba's Kafka consumer for:

snuba-commit-log

Expected Result

Sentry consumers should remain connected to their Kafka consumer groups and continue processing messages normally during continuous operation.

The consumer containers should remain healthy and should not require manual restarts.

Temporary Kafka coordinator changes or consumer group rebalances should recover automatically without causing the containers to become unhealthy.

Actual Result

Multiple Sentry consumers periodically lose communication with the Kafka group coordinator and experience approximately 45-second consumer session timeouts.

The logs contain messages such as:

Consumer group session timed out (in join-state steady) after 45000 ms
without a successful response from the group coordinator

followed by:

Broker: Not coordinator

Local: Waiting for coordinator

and:

Broker: Coordinator load in progress

The consumers revoke their Kafka partitions and attempt to rejoin the consumer group.

For example:

Partitions to revoke:
[Partition(topic=Topic(name='events'), index=0)]

and:

Commit failed:
KafkaError{code=_WAIT_COORD,val=-180,str="Local: Waiting for coordinator"}

Snuba exhibits the same behavior on snuba-commit-log, suggesting that the problem is not isolated to a single Sentry consumer.

Additionally, sentry-process-segments-1 frequently reports processing delays such as:

Batch process is taking longer than 5 seconds.

and:

Waited on the process pool longer than 20 seconds.
Waiting for 1 results.

After these Kafka coordinator/session problems, multiple containers become unhealthy.

Restarting the Sentry stack temporarily resolves the problem, but the same behavior returns several times per day, requiring approximately 4 manual restarts per day.

_sentry-process-segments-1_logs.txt
_sentry-post-process-forwarder-errors-1_logs.txt
_sentry-monitors-clock-tasks-1_logs.txt
_sentry-ingest-monitors-1_logs.txt
_sentry-generic-metrics-consumer-1_logs.txt
_sentry-events-consumer-1_logs.txt
_sentry-snuba-subscription-consumer-events-1_logs.txt

Event ID

No response

[aldy505]

When this happen again, can you provide the kafka container logs instead?

[douglasndm]

When this happen again, can you provide the kafka container logs instead?

Sure, At this moment I'm having the issue again, with containers "sentry-snuba-subscription-consumer-events-1" and "sentry-ingest-monitors-1" unhealthy.

_sentry-kafka-1_logs.txt

[aldy505]

Reading from your Kafka logs, I don't see any error. The only possible cause is a slow disk, maybe? Kafka stores the data on disk, therefore if you have loads of data (in GBs) coming to your Kafka with a relatively slow disk, that could be the issue.

How I deducted this:

• Your Kafka only provides INFO level of logs, no errors, no warns. Only showing data deletion/tombstone.
• Your consumer containers erroring in "Waiting for coordinator" during offset commit. This would say you have a slow Kafka instance, therefore either a slow disk or slow CPUs.