logger_olp has sync stage disabled — mailbox grows unboundedly under CPU saturation

[alco]

logger_olp has sync stage disabled — mailbox grows unboundedly under CPU saturation

Problem

The otel_metric_exporter logger handler disables the sync stage by setting sync_mode_qlen == drop_mode_qlen. This eliminates the backpressure stage between async and drop mode. Under CPU saturation, the logger_olp process cannot update its persistent_term overload mode (because it cannot get scheduled), so callers keep casting into a growing mailbox.

Observed on a downstream Electric deployment during a recent incident: peak 94K messages in the handler mailbox, drain time measured in hours.

Fix

In the OTel logger handler config defaults:

1. Set sync_mode_qlen < drop_mode_qlen (e.g. 50 / 500) so async → sync → drop transitions cleanly.
2. Enable burst_limit_enable: true with appropriate limits.
3. Consider overload_kill_enable: true with a sane overload_kill_qlen to bound the damage.
4. Raise the OTel log handler level from :info to :warning to reduce message volume under CPU saturation.