Recommendations for Handling Spurious Errors in redfish-logservices checks

[OoYo0uto]

We are using the redfish-logservices check for monitoring the state of our physical servers using the redfish API. We have some (Supermicro) hosts which have one correctable ECC memory error every 6 to 12 months, leading to a Icinga warning and therefore a ticket in our ticket system.

* 2026-08-27T14:49:17Z: [ OEM ] Correctable ECC / other correctable memory error @DIMMA2 

The question now would be:

• How do you deal with spurious ECC memory errors in the event log currently?
• Should we implement in the redfish-logservices check plugin the possibility of defining a "acceptable" threshold of certain error messages? An idea would be the following:
  • Introduce a parameter --limited-ignore where you define a count, an interval and a regex. All messages matching a regex specified this way are ignored, as long as the specified count is not exceed in the given interval. By moving the regex to the end, this allows to have : in the regex itself, because everything after the second : can be considered to be the regex.
  • An example for my use-case would be --limited-ignore 10:86400:Correctable ECC.*memory error to accept up to 10 ECC errors in an interval of 24 hours.
  • I am currently not sure if you already have duration unit parsing, if yes, this could also be --limited-ignore 10:24h:Correctable ECC.*memory error.

In general, this functionality could also be interesting for other log-based checks like dmesg or journald-query, but for now, this is the only check where we have a specific case where we want to apply it.

Background

I performed some research. Intel recommends taking action in an article when more than 10 errors occur during a period of 24 hours. Two research articles about Google's Server Fleet and Facebook's Server Fleet led me to the conclusion, that an correctable ECC memory error every few months is acceptable for us, because the chance of an uncorrectable memory error is still very low.

If you want to include this as default behaviour directly in the respective plugin:
According to the study, Facebook uses a policy of repairing systems with 100 correctable errors in a single week, so I plan to use a threshold of 10 errors per day.