Redfish check tuning help

[thesefer]

Hi, is anyone else here using the redfish checks? Ever since the most recent update to those I constantly run into either

URL error "The read operation timed out" (timeout 8, retries 10)
or <Timeout exceeded.><Terminated by signal 15 (Terminated).><Terminated with exit code 128 (0x80).> constantly for Redfish Sensors.

I already raised the retries for each service and I raised the timeout (check_timeout) for each command to 3 minutes. Not 2026 hardware for sure. If it matters those are Dell servers.

Regards

[markuslf]

The new redfish plugins needed an extended redfish lib, which multiplied the number of requests to the BMC. We'll try to improve.

[markuslf]

Improved (9bc527a). This commit needs the latest lib/redfish.py (Linuxfabrik/lib@9c86868).

[thesefer]

Unfortunately, even after 7.0.0 it is still flapping very very often. Redfish Sensors is still at a permanent <Timeout exceeded.><Terminated by signal 15 (Terminated).><Terminated with exit code 128 (0x80).>.

[markuslf]

Could you please test the current main branch of both lib and monitoring-plugins? We enhanced the lib, so updating the plugins alone will not do; our installer's --source mode pulls both repositories from main.

Then please:

1. Set --timeout=30 --retries=1. --retries=10 means eleven attempts per request, and a collection is tried twice, so 176 s can pass before the check even reaches the sensors.
2. Make sure --cache-expire is larger than your check interval, otherwise the Redfish checks on a host cannot share a session.
3. Enable --verbose on Redfish Sensors for a few cycles and send us /tmp/linuxfabrik-monitoring-plugins-uid<uid>/linuxfabrik-monitoring-plugins-redfish-trace.log. It survives a check that Icinga terminates, and it contains no passwords or tokens.

Please also tell us the iDRAC firmware version, and whether all your Dell servers are affected or only some.

[thesefer]

Hey, we only install via package (debian). Shouldn't lib be updated in this case due to the packaging process? I have not found a way yet to properly distribute via git (shebang venv topic), and keeping it updated is more complex due to network and policies..

I'll try and adjust 1. and 2. first. Currently retries is on 3, timeout 30, cache on 5, so it should be reasonable with 90s out of 300s. I had it on default briefly but the flapping was even worse (retries 3 and timeout 8).

Could be that the check_timeout is too short with the default 60s now which got reset with the update,