Add first-class vLLM / Hugging Face model loading support #359

Summary

GEPA already supports local/custom model workflows through LiteLLM model strings and callable reflection_lm functions, and some examples use Ollama. However, the vLLM + Hugging Face workflow is not first-class or clearly documented.

We should add first-class support, examples, and documentation for running GEPA with a Hugging Face model served through vLLM, so users can evaluate and optimize prompts without relying only on paid API keys.

Current State

Local models are not impossible today:

• gepa.lm.LM forwards extra LiteLLM parameters such as api_base and api_key.
• reflection_lm can be a model string or any callable.
• Existing docs/examples mention local Ollama usage.
• The AnyMaths adapter README says LiteLLM-supported providers include HuggingFace, Groq, vLLM, Ollama, etc.

The gap is that there is no dedicated, tested, end-to-end vLLM + Hugging Face guide/example showing how to start a vLLM OpenAI-compatible server and use it from GEPA.

Motivation

Many users want to test GEPA with open models such as Qwen, Llama, Mistral, Gemma, or other Hugging Face models. The current central docs mainly point users toward provider model strings and API-key based examples, while local model guidance is spread across adapter-specific examples. This creates friction for:

• local experimentation with open-weight models
• reproducible benchmarking on self-hosted models
• GPU-backed evaluation using vLLM
• avoiding paid API usage during development

Because GEPA already routes LMs through gepa.lm.LM and LiteLLM, this feature should make the vLLM path explicit, tested, and documented.

Proposed Scope

Add a supported workflow for using vLLM as the base model and/or reflection model by loading a Hugging Face model and exposing it to GEPA through an OpenAI-compatible endpoint.

The expected user flow should look roughly like:

uv sync --extra dev

uv run python -m vllm.entrypoints.openai.api_server \
  --model Qwen/Qwen2.5-7B-Instruct \
  --served-model-name qwen2.5-7b-instruct \
  --host 0.0.0.0 \
  --port 8000

Then GEPA should be usable with that local endpoint:

from gepa.lm import LM

lm = LM(
    model="openai/qwen2.5-7b-instruct",
    api_base="http://localhost:8000/v1",
    api_key="EMPTY",
)

If LiteLLM's native vLLM provider is preferred, document and test that supported model string instead.

Tasks

• Decide the public API for vLLM usage.

  • Should users call LM("openai/<served-model-name>", api_base="http://localhost:8000/v1", api_key="EMPTY")?
  • Or should GEPA expose a helper such as make_vllm_lm(...)?
  • Keep compatibility with the existing make_litellm_lm and LM design.

• Add an optional dependency group for vLLM usage.

  • Add something like vllm = ["vllm", "transformers", "huggingface-hub"] if we want GEPA to help users run the server locally.
  • Avoid making vLLM a required dependency because it is heavy and GPU/environment dependent.

• Add documentation for running a Hugging Face model with vLLM.

  • Include install command.
  • Include vLLM server startup command.
  • Include GEPA configuration example.
  • Explain api_base, api_key="EMPTY", served model names, and common port defaults.
  • Mention GPU memory requirements and quantization options at a high level.

• Add at least one runnable example.

  • Example path suggestion: examples/vllm_huggingface/README.md
  • Optional script suggestion: examples/vllm_huggingface/main.py
  • Use a small or commonly available instruct model in the example.
  • Keep the example compatible with uv run.

• Ensure both GEPA model roles can use vLLM.

  • Base/task model can be served by vLLM.
  • Reflection model can be served by vLLM.
  • Mixed setups should work, for example vLLM base model plus OpenAI/Anthropic reflection model, or OpenAI/Anthropic base model plus vLLM reflection model.

• Add unit tests around LM argument forwarding.

  • Verify api_base is forwarded to LiteLLM.
  • Verify api_key is forwarded to LiteLLM.
  • Verify model names are passed through unchanged.
  • Avoid requiring a real vLLM server in normal unit tests.

• Add an optional integration test or manual verification section.

  • Mark it clearly as GPU/local-server dependent.
  • Test against a running vLLM OpenAI-compatible endpoint.
  • Confirm LM.__call__ and LM.batch_complete both work.

• Update docs/API references.

  • Update docs/docs/guides/quickstart.md or add a dedicated local models guide.
  • Update docs/docs/guides/cost-tracking.md if local vLLM calls report zero or unavailable cost.
  • Link from adapter docs where model configuration is discussed.

• Add troubleshooting notes.

  • Model name mismatch between --served-model-name and LM(model=...).
  • Missing or incorrect /v1 suffix in api_base.
  • CUDA out-of-memory errors.
  • Hugging Face gated model authentication.
  • Chat template or tokenizer issues.
  • Concurrency limits when max_workers is too high.

Acceptance Criteria

• A user can run a Hugging Face model through vLLM and call it from gepa.lm.LM.
• A user can pass a vLLM-backed model as reflection_lm in GEPAConfig.
• The docs include a complete command-line setup and Python usage example.
• Unit tests cover the GEPA-side configuration/argument forwarding.
• The feature does not require vLLM unless the user explicitly installs the optional vLLM dependency group.
• Existing API-key based provider workflows continue to work unchanged.

Out of Scope

• Managing GPU provisioning automatically.
• Downloading large Hugging Face models during unit tests.
• Making vLLM a required GEPA dependency.
• Guaranteeing all Hugging Face models work without chat-template/model-specific configuration.

Notes

The main implementation may be mostly documentation and examples because GEPA already has a LiteLLM-backed LM abstraction and accepts custom callables. The important part is making the local vLLM/Hugging Face path explicit, tested, and easy to follow.