The code defines the temporal aggregation for Sum, Counts and Histogram metrics as cumulative. Ex:
|
aggregation_temporality: :AGGREGATION_TEMPORALITY_CUMULATIVE, |
But the code actually clears the metrics from memory after each export, making them effectively Delta temporal aggregation:
With Prometheus running locally, this seems to work, but only by accident it seems. Prometheus essentially considers each export as a reset and is then able to parse the metrics, but other providers like New Relic doesn't seem to be able to do this trick and it completely drops the metric.
On the other side, Prometheus doesn't support Delta temporal aggregation natively. If we made the switch, a user of the library would have to either:
- use the
delta_to_cumulative processor in an Otel collector;
- use
otlp-deltatocumulative or otlp-native-delta-ingestion on their Prometheus instance (If they have access to it and are on a recent enough version).
I do not think those are reasonable constraints to put on the users of the library.
The solution I am proposing is to make the temporal aggregation configurable. We would default to true cumulative metrics (would require a new implementation to fix the current issue) since this is the default right now. But the user could configure Delta aggregation which would use the current implementation and could be used when exporting to providers that prefer Delta aggregation, like New Relic.
I am willing to do this work, but before I go, I want to validate the direction with you and make sure that this is something you would accept.
The code defines the temporal aggregation for Sum, Counts and Histogram metrics as cumulative. Ex:
elixir-otel-metric-exporter/lib/otel_metric_exporter/metric_store.ex
Line 300 in e81bf4d
But the code actually clears the metrics from memory after each export, making them effectively Delta temporal aggregation:
elixir-otel-metric-exporter/lib/otel_metric_exporter/metric_store.ex
Line 262 in e81bf4d
With Prometheus running locally, this seems to work, but only by accident it seems. Prometheus essentially considers each export as a reset and is then able to parse the metrics, but other providers like New Relic doesn't seem to be able to do this trick and it completely drops the metric.
On the other side, Prometheus doesn't support Delta temporal aggregation natively. If we made the switch, a user of the library would have to either:
delta_to_cumulativeprocessor in an Otel collector;otlp-deltatocumulativeorotlp-native-delta-ingestionon their Prometheus instance (If they have access to it and are on a recent enough version).I do not think those are reasonable constraints to put on the users of the library.
The solution I am proposing is to make the temporal aggregation configurable. We would default to true cumulative metrics (would require a new implementation to fix the current issue) since this is the default right now. But the user could configure Delta aggregation which would use the current implementation and could be used when exporting to providers that prefer Delta aggregation, like New Relic.
I am willing to do this work, but before I go, I want to validate the direction with you and make sure that this is something you would accept.