Coding agents running GEPA via /goal don't know to set a high-enough max_metric_calls — proposed API + docs fixes #375

Summary

A user reported on X that running GEPA inside Codex's /goal loop on GPT-5.5-high ended after a single proposal step, even though the plan clearly implied iterative optimization. The coding agent went on to declare the goal "reached" because the one accepted candidate beat the baseline on external validation.

The agent's own post-mortem (screenshot below) is unusually clear about the root cause:

Because the plan said the right high-level things, but the actual run command did not encode "multiple GEPA iterations" as a hard requirement.

What happened:

• GEPA is controlled by max_metric_calls, not an explicit iterations flag.
• The earlier 12B six-row run used max_metric_calls=8.
• With row_limit=6, the baseline full validation already costs about 6 metric calls.
• One accepted candidate then costs minibatch eval plus another full validation.
• So the budget was basically enough for only one proposal, even though the plan implied iterative GEPA optimization.

The failure was operational: I treated the first successful 12B GEPA run as meaningful progress because it produced a better candidate and passed external validation, but I did not verify that it had enough optimizer trajectory to support "scores across iterations." I should have checked run_log.json / gepa-result.json immediately and noticed num_candidates=2 before presenting graphs.

This is going to happen a lot as coding agents (Claude Code, Codex, Cursor agent mode, etc.) start invoking GEPA in long-horizon loops. The agent has no built-in model for how max_metric_calls translates into actual optimization depth, so the cheapest path (a budget that just barely produces one accepted candidate) looks the same to it as a real optimization run.

Failure mode in one line

max_metric_calls is the only knob controlling iteration count, but the relationship between budget and actual proposals is implicit, model-/data-dependent, and not surfaced at run end — so an under-budgeted run can look successful from the outside.

Proposed fixes — API

These are roughly in priority order. Items 1–3 would have caught the reported case at runtime without any user education.

1. Compute a minimum-sensible budget at the start of gepa.optimize and warn loudly if max_metric_calls is below it. The formula is straightforward and the optimizer already knows everything it needs:

   baseline_eval_cost = len(valset)
   min_per_proposal   = reflection_minibatch_size + len(valset)   # mutate-and-validate
   min_floor          = baseline_eval_cost + 3 * min_per_proposal   # require ≥3 proposal attempts
   if max_metric_calls < min_floor:
       warnings.warn(f\"max_metric_calls={max_metric_calls} only supports ~{N} proposals \"
                     f\"given valset size {len(valset)} and minibatch {reflection_minibatch_size}. \"
                     f\"Recommended minimum: {min_floor}. \"
                     f\"See https://gepa-ai.github.io/gepa/guides/budget/\")
   

   This is by far the highest-leverage change: it would have stopped the reported run from looking successful at all.

2. Add an explicit min_proposals / min_iterations parameter. Independent of max_metric_calls, let callers say "don't return until you've tried at least N proposals" — even if the budget runs out, the optimizer can extend by a fraction (with a warning) or error rather than silently halt at 1. Coding-agent prompts can then encode this as a hard requirement: min_proposals=10.

3. Make num_proposals_attempted, num_proposals_accepted, and num_validations first-class fields on GEPAResult, and print a 1-line summary at the end of every gepa.optimize run:

   GEPA finished: 1 proposal attempted (1 accepted) over 8 metric calls.
                  ⚠ Only 1 proposal — likely under-budgeted. See <link>.
   

   The reported agent specifically said "I should have checked run_log.json / gepa-result.json immediately and noticed num_candidates=2 before presenting graphs" — surfacing this at the end of optimize() makes that automatic.

4. Type-narrow max_metric_calls's docstring to call out the floor explicitly. The current docstring describes it as an upper bound; in practice it is the only iteration knob, which deserves a callout.

5. (Optional) Default max_metric_calls=None and require either an explicit budget or min_proposals. This is more invasive but rules out the failure mode entirely.

Proposed fixes — docs

1. New "Choosing max_metric_calls" page in the user guide with the budget formula and a worked example for small (~10 examples) and medium (~100 examples) valsets. Linked from the quickstart and from the warning text in fix Make v0.0.1 of GEPA #1.

2. "Running GEPA inside an autonomous coding agent" page with explicit guidance the user can drop into a /goal-style prompt:

   • Set max_metric_calls to at least baseline_eval_cost + N × (minibatch + valset) for N ≥ 10 proposals
   • Verify result.num_proposals_attempted ≥ N before declaring the goal reached
   • Inspect gepa-result.json / run_log.json for num_candidates and the score trajectory shape (a 2-point trajectory means under-budgeting, not convergence)

3. Add a callout to the quickstart above the max_metric_calls=150 line that explains why 150 (not 15), with a link to the budget guide.

4. Add an FAQ entry: "My GEPA run finished with only one proposal — what happened?" pointing at the budget formula and at result.num_proposals_attempted.

Evidence from the original incident

Agent's post-mortem (screenshot from the user's session):

The correct status should have been: baseline improved candidate found, but GEPA search depth insufficient; need a larger metric-call budget or longer run before claiming the experiment is done.

GEPA validation-score trajectory the agent presented as "done" — each pane has only candidate 0 → candidate 1 (one proposal step), exactly the symptom that fix #3 above would surface automatically:

• E4B smoke: 0.250 → 0.625
• 12B six selected: 0.286 → 0.556
• 12B twelve init: 0.424 → 0.424 (no improvement)
• 12B twelve continuation: 0.537 → 0.610

The score lifts are real, but the trajectory is a 2-point curve, not optimization. That's the user-visible footprint of the bug.

Source

• X thread by @onusoz — first public report