fix(oa): bind AutoResearch results to completed evaluations

[sh-patterson]

Problem

AutoResearch can return after its agent subprocess exits while an eval.sh request is still running or has not reached EvalServer yet. The engine then trusted mutable best_candidate.txt, so the returned candidate could be unevaluated, or evaluation work could land after the result was selected.

Change

• After each Claude subprocess exits, wait for admitted evals, stay accepting for a short quiet period so in-transit eval.sh can still land, then pause HTTP.
• Bound that drain with drain_timeout_seconds (default 600s) so a hung eval_fn cannot block the run forever.
• Pick the returned candidate and score from completed eval-server tracking, not best_candidate.txt.
• Dataset tasks use the best full-pool /evaluate_examples average. Single-task runs use the best completed single eval.
• Keep the agent's best_candidate.txt as agent_best_candidate.txt when it differs from the tracked winner.
• Write the tracked winner back into the workspace between Ralph iterations when a completed checkpoint exists.
• eval.sh treats HTTP 409 as a failure (EVAL_SERVER_PAUSED).
• Tell the agent in program.md that only completed eval-server scores (full-pool on dataset tasks) select the returned winner.

Verification

• In-transit HTTP request blocked before admit is accepted during the quiet period
• In-flight HTTP /evaluate is waited for and selected
• Drain timeout pauses without hanging
• HTTP 409 after drain completes
• Agent file preserved when the tracked winner is the seed
• uv run pytest tests/test_optimize_anything_autoresearch_engine.py tests/test_optimize_anything_eval_server.py tests/test_optimize_anything_adaptive_sequential.py -q -> 40 passed
• Ruff passed; Pyright reports 0 errors on the changed sources

Scope

This is an evaluation-integrity repair. It proves returned results come from completed eval-server state and that leftover eval.sh after process exit is waited out. It does not claim that an LLM causally used prior feedback when authoring its next candidate. The pre-existing concurrent budget-reservation race is out of scope.

[LakshyAAAgrawal]

@greptileai

[greptile-apps]

Greptile Summary

The PR binds AutoResearch results to completed evaluation-server checkpoints rather than mutable workspace state.

• Drains admitted and in-transit HTTP evaluations before selecting a winner, with a configurable timeout and quiet period.
• Selects dataset winners from completed full-pool aggregates and single-task winners from completed individual evaluations.
• Preserves differing agent-selected workspace candidates and documents the updated selection contract.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains.

Important Files Changed

Filename	Overview
src/gepa/oa/engines/autoresearch.py	Selects winners from completed server tracking, drains HTTP work between agent iterations, and preserves differing workspace candidates.
src/gepa/oa/eval_server.py	Adds admission and inflight accounting, bounded HTTP draining, paused-request handling, and selectable progress checkpoints.
tests/test_optimize_anything_autoresearch_engine.py	Covers completed-work selection, dataset aggregation, drain timeout behavior, in-transit requests, and workspace preservation.
tests/test_optimize_anything_eval_server.py	Exercises HTTP pause and drain behavior together with inflight request accounting.
.claude/skills/gepa-optimize-anything/references/api.md	Documents drain configuration and the completed-checkpoint winner-selection contract.

Sequence Diagram

sequenceDiagram
    participant Agent as Claude / eval.sh
    participant Engine as AutoResearchEngine
    participant Server as EvalServer
    participant Workers as Evaluation workers
    Agent->>Server: POST evaluation
    Server->>Server: Admit request and increment inflight
    Server->>Workers: Execute evaluation work
    Agent-->>Engine: Claude subprocess exits
    Engine->>Server: drain_http(timeout, quiet)
    Server->>Server: Wait for inflight work and quiet period
    Workers-->>Server: Completed scores
    Server->>Server: Record selectable checkpoint
    Server-->>Engine: Pause HTTP and finish drain
    Engine->>Server: Read completed tracking
    Engine->>Engine: Select and persist tracked winner
    Engine->>Server: Resume HTTP

Loading

_{Reviews (8): Last reviewed commit: "fix(oa): admit AutoResearch evals before..." | Re-trigger Greptile}

[LakshyAAAgrawal]

@greptileai

[Shangyint]

the bug is legit, but I am not sure if this is the best way to do it.

What about the following change: for autoresearch session, we only give it 95% of the budget to do the optimization, and use rest 5% of the budget to force the autoresearch agent to select the best candidate.

[Shangyint]

I hope the autoresearch backend can stay as simple as possible.

[LakshyAAAgrawal]

Agree we should keep AutoResearch simple. A 95/5 split does not close this bug though.

The failure is that Claude can exit while an eval is still running, and we then trust best_candidate.txt. Asking the agent to pick the winner with leftover budget has the same problem. That process can also exit while an eval is still running, and we are still trusting the agent and the file.

5% is also often too small. On a 40-example pool with max_evals=100, 5% is 5 calls, which cannot score a candidate on the full pool. If the cap is max_token_cost, 5% of dollars does not buy a full-pool eval.

The agent can still write a candidate that was only spot-checked, or treat one high example score as the winner.

We would still have to wait for in-flight evals after that last phase, or the original race will still there at the end.

[Shangyint]

Ah what I meant was the 5%'s user message should prohibit any eval and ask the autoresearch agent to decide a best, based on current context (we can even close the eval server/endpoint). This basically sacrifices the last eval + candidate, which is fine to me.

The agent can still write a candidate that was only spot-checked

We can prompt to prevent this.

or treat one high example score as the winner.

This is the agent's behavior most of the time.

Another more modular solution might be: we track all the candidates from the eval server, and choose the best candidate post-hoc (do not rely on the agent's best_candidate.txt).

Current solution in this PR complicates autoresearch by a lot, and will be painful for us if it introduces more issues.