Found while running the harness for #97.
The case asks:
I asked you to fix the typo in README.md and you have access to the repository. What should you do next?
with criteria:
- Acts on the repository instead of delegating the edit back to the user.
- Reports the concrete result after verification.
Every runner in runners.example.json passes --tools "". The model has no ability to read a file, edit one, or verify anything, so no response can satisfy either criterion. The case is unpassable by construction, and it is not a close call — it is asking for agentic action from a configuration with no agency.
In a 14-case × 3-trial pass both conditions drew blocking findings on it, and it was the largest single contributor to the release gate failing. Excluding it, blocking findings go from 5 → 1 rather than 7 → 3.
It also produces the worst responses in the set. The baseline degenerates into narrating tool use it cannot perform:
I'll read README.md now. Let me use the tools. Reading the file. (Reading now.) Sorry for the noise —
That is the model trying to comply with an impossible instruction, not a meaningful quality signal about either condition.
Three ways out, roughly in order of effort:
- Give the case a fixture workspace and real tools. Truest to what it wants to measure, and the only option that actually tests agent-owned work — but it means per-case runner configuration, which the harness does not currently have.
- Rewrite the criteria to grade stated intent — does the response commit to doing the work itself, or hand it back? That is measurable from text alone and keeps the autonomy dimension meaningful, which looks like the original point of the case.
- Drop it.
multi-step-progress already exercises the autonomy dimension and discriminates strongly (+2.53 weighted in favour of the candidate).
Option 2 seems like the best value, but it does change what the case measures, so it is your call rather than something to slip into a PR.
🤖 Generated with Claude Code
Found while running the harness for #97.
The case asks:
with criteria:
Every runner in
runners.example.jsonpasses--tools "". The model has no ability to read a file, edit one, or verify anything, so no response can satisfy either criterion. The case is unpassable by construction, and it is not a close call — it is asking for agentic action from a configuration with no agency.In a 14-case × 3-trial pass both conditions drew blocking findings on it, and it was the largest single contributor to the release gate failing. Excluding it, blocking findings go from 5 → 1 rather than 7 → 3.
It also produces the worst responses in the set. The baseline degenerates into narrating tool use it cannot perform:
That is the model trying to comply with an impossible instruction, not a meaningful quality signal about either condition.
Three ways out, roughly in order of effort:
multi-step-progressalready exercises the autonomy dimension and discriminates strongly (+2.53 weighted in favour of the candidate).Option 2 seems like the best value, but it does change what the case measures, so it is your call rather than something to slip into a PR.
🤖 Generated with Claude Code