feat: format validation as first-class metric with automatic failure penalization #265

Problem

For tasks requiring structured output (JSON, XML, specific formats), format failures should be treated as hard failures. Dropbox observed 40% malformed JSON on gemma-3-12b before optimization, and treated broken JSON as score=0. GEPA reduced this to <3% by optimizing the prompt — but users have to implement the format penalty themselves in their evaluator.

A common pattern (penalize format failures, log the parse error as feedback) should be a built-in utility.

Proposed Solution

Ship a format_validator wrapper or OutputFormatAdapter that:

1. Wraps any evaluator to catch format errors and return score=0 + error feedback automatically
2. Optionally tracks format compliance as a separate objective for multi-objective optimization

from gepa.utils import require_json_output

@require_json_output(schema={"score": int, "reasoning": str})
def evaluator(data, response):
    # Only called when response is valid JSON matching the schema
    score = compute_quality(response["score"], data["expected"])
    return score, {"Output": response, "Expected": data["expected"]}

# Format failures automatically get score=0 and
# feedback="Malformed JSON: <parse error>" without any boilerplate

Could also support: require_xml_output, require_regex_match, or a general require_format(validator_fn).

Context

• Dropbox blog: https://dropbox.tech/machine-learning/optimizing-dropbox-dash-relevance-judge-with-dspy
• Especially valuable for smaller models (gemma, Qwen, etc.) that struggle with structured output
• The format error message in feedback is critical — it gives GEPA's reflection LM the signal to add output formatting instructions to the prompt