RFC: Peep-style scheduler-local storage for hot metric contention

[ruslandoga]

👋

I've been benchmarking parts of / investigating a production CPU-scaling problem in https://github.com/Logflare/logflare and potentially traced a significant part of it to same-key ETS contention in the metric write path.

Before preparing patches, I wanted to ask whether the maintainers would be interested in receiving PRs in this area, and what scope/design you would prefer.

Codex summary

What we observed

The reported production symptom is roughly 40% host CPU while BEAM schedulers appeared fully utilized. Under representative (but local, on a laptop) load, :lcnt showed heavy contention in the metric exporter's ETS table.

The hottest metric was an untagged Broadway per-message histogram:

distribution(
  "broadway.processor.message.stop.duration",
  unit: {:native, :millisecond}
)

Because it has no tags and most observations land in the same duration bucket, many schedulers synchronously call :ets.update_counter/4 on the same ordered-set record.

In a load test with eight schedulers at approximately 20000 events/second:

• The histogram produced 55057 CA-tree lock collisions and 1.09 seconds of accumulated lock wait during a two-second capture.
• Removing only that metric reduced the result to six collisions.
• Over clean ten-second runs, scheduler utilization was:
  • 7.2% without the metrics exporter
  • 18.8% with the full exporter
  • 13.5% with only the Broadway per-message histogram removed

These measurements were made against Logflare's Supabase fork at 2a6de91, not Electric's current main. The repositories have diverged, so I would port the reproducer and confirm the result against upstream before proposing production changes. However, the relevant storage topology appears materially similar: one public ordered-set and synchronous per-bucket update_counter calls.

Peep inspiration

Peep addresses this class of problem with scheduler-local aggregation.

Its current storage implementations use a few useful techniques:

• Counter and sum keys are decentralized by scheduler.
• The striped backend uses one ETS table per scheduler.
• Distributions use atomics-backed bucket arrays.
• Collection adds counters, sums, and histogram buckets across stripes.
• Last-value metrics carry timestamps so collection can select the newest value.

Relevant implementations:

• Single-table ETS storage
• Per-scheduler striped storage
• Atomics-backed histograms

I made a rough Peep-inspired experiment in supabase/elixir-otel-metric-exporter#13. Its parallel benchmark improved from about 147K to 466K events/second, but it combines several optimizations and is not production-ready. In particular, it does not correctly merge values across stripes and has metric-ID correctness problems. I would not propose merging or porting that PR as written :(

Possible contribution

I think this would be safest as incremental PRs:

1. Add a parallel same-key benchmark and multi-scheduler correctness tests.
2. Add an opt-in atomics-backed distribution implementation as the narrowest experiment for hot histograms.
3. If the results justify it, add an opt-in Peep-style striped storage backend.
4. Only consider changing the default after downstream production validation.

The correctness tests would cover:

• Exact counter and sum totals across multiple schedulers
• Histogram count, sum, and bucket merging
• Latest-timestamp selection for last-value metrics
• Concurrent writes during generation rotation/export
• Memory accounting and cleanup for atomics/striped storage
• Unchanged output for the existing default backend

I would initially keep handler compilation/compact metric IDs out of this work so the storage change can be benchmarked and reviewed independently.

This is also separate from the aggregation-temporality problem in #44; a storage PR should avoid silently changing those semantics.

Questions

Would you be open to receiving PRs for this?

If so, would you prefer:

1. A benchmark/reproducer PR first?
2. A small opt-in atomics histogram backend before generalized striping?
3. An internal Peep-inspired implementation, or a more general storage behaviour?
4. Holding this work because the longer-term plan is migration to the official OpenTelemetry metrics integration discussed in OpenTelemtry integration with Telemetry.Metrics #14?

Happy to prepare the upstream reproducer first and share the :lcnt commands and complete results.