[Refactor][Model Runner V2][Multimodal] Move the encoder-only path out of the shared runner by gty111 · Pull Request #53176 · vllm-project/vllm

gty111 · 2026-08-20T23:33:38Z

Purpose

An encoder-only instance — --mm-encoder-only, or the producer side of
encoder-cache disaggregation — runs the vision encoder and publishes the
embeddings. It runs no language model, holds no KV cache and samples no token,
so most of a step does not apply to it. Today that is expressed as
is_encoder_only branches scattered through the shared V2 model runner: one in
get_kv_cache_spec, _dummy_run, profile_run and capture_model, two inside
execute_model, and one more in warmup.py.

This moves all of them into a single MMEncoderModelRunner, selected by the
worker. Two consequences:

The shared runner shrinks by deletion. gpu/model_runner.py is +3/-33,
and the three added lines are only the un-indenting of the surviving
prepare_inputs_embeds call. Nothing in it is restructured: the encoder
subclass reaches the same state through the inherited update_requests /
gather_batch_req_state / prepare_inputs.
It fixes an illegal memory access in the encoder. UvaBufferPool recycles
its slots every max_concurrent_batches steps with no synchronization, and
the device reads the pooled host buffers in place. That is safe for a sampling
step because AsyncOutput.get_output()'s copy_event.synchronize() — ordered
after all main-stream work by copy_stream.wait_stream(main_stream) — is a
per-step device wait, and step N is popped during N+1 while its slot is not
reused until N+2. An encoder-only step returns
make_empty_encoder_model_runner_output(), built on the host with no D2H
copy, so it never waits on the device: the host runs ahead, slots recycle
underneath a live _apply_write_kernel, and it stores through a stale
row_idx. MMEncoderModelRunner is the one runner with that property, so the
barrier lives there and buffer_utils.py is untouched.

Test plan and results

All on GB200, Qwen/Qwen3-VL-4B-Instruct, VLLM_USE_V2_MODEL_RUNNER=1.
ruff-check, ruff-format and mypy-3.12 pass on every touched file, and
pytest tests/v1/ec_connector/unit/ -q gives 149 passed.

Normal multimodal serving is unaffected — the path the deletion-only diff has
to preserve:

request	output
`"The capital of France is"`, `max_tokens=8`, `temperature=0`	`" Paris. The capital of Spain is Madrid"`
336×336 image of a red circle + "What shape and color is in this image?"	`"Circle, red"`

Encoder-cache disaggregation, end to end. ECExampleConnector with
ec_role: ec_producer and ec_role: ec_consumer over a shared /dev/shm path.
Every request carries a distinct generated image, so each one is a real encode
rather than a cache hit; every tenth is replayed against the consumer to confirm
the published embeddings are usable.

metric	result
producer requests	35,000 / 35,000 → 200
consumer requests	3,500 / 3,500 → 200, answers correct
CUDA core dumps	0
tracebacks / illegal memory access / CUDA errors	0
both instances after the run	`/health` 200

That clears every observed crash point by 1.49×:

configuration	crashed at
Mooncake EC connector + V2 runner, no barrier	9,532 encodes
NIXL EC connector, no barrier	15,275 / 15,279 / 23,542 encodes
this branch	35,000 clean

Core dumps from those failures show _apply_write_kernel's tl.store with a
garbage row_idx, all 32 lanes illegal, in the last block of the grid.

AI assistance

AI assistance (Claude) was used for this change: for the investigation that
identified the missing per-step device wait as the IMA's cause, for the
refactor, and for drafting this description. Every changed line was reviewed by
me, and the tests above were run and their output inspected by me.

Signed-off-by: Tianyu Guo <guoty@inferact.ai>

…input buffers Signed-off-by: Tianyu Guo <guoty@inferact.ai>

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

njhill

Thanks @gty111 I think this could be a nice direction.

We should also reduce the verbosity of the comments.

Size the input-buffer barrier by the UVA pool depth instead of max_concurrent_batches: the pool clamps its depth to at least 2, so a slot is recycled every 2 steps even when the engine runs one batch at a time, and this runner never waits on the device. One event per slot generation also lets the host run ahead again, which a single event did not. An encoder tier has no DP peer to agree a padded shape with and no CUDA graph to dispatch, so assert DP=1 and build the batch descriptor locally. Signed-off-by: Tianyu Guo <guoty@inferact.ai>

njhill

Thanks @gty111 I think this is a really nice way to isolate things

vLLM already uses "encoder_only" for the attention type of encoder-only transformers (AttnTypeStr in config/model.py, is_encoder_only_attention in the CPU attention backend), so the old name read as a statement about the model architecture rather than about an instance that runs only the multi-modal encoder. Also drops an is_encoder_only field from a model-runner stub in the warmup tests: after the earlier commits in this series nothing reads that attribute off a runner, so renaming it would only suggest a reader that no longer exists. Signed-off-by: Tianyu Guo <guoty@inferact.ai>

Size the input-buffer event ring by max_concurrent_batches directly, so buffer_utils no longer needs a getter for the pool depth, and wrap the wait and the record in an `input_tensor_semaphore` context manager. Its docstring, rather than a comment at the call site, now carries the constraint that the guard must be released before the encoder runs. Restore the unconditional V2 branch in the worker: gating it on the runner type let an encoder-only instance fall through to the `elif` that warms up the V1 sampler. The check belongs in warmup_kernels, where it was before. Signed-off-by: Tianyu Guo <guoty@inferact.ai>

Signed-off-by: Tianyu Guo <guoty@inferact.ai>

njhill

Thanks @gty111! LGTM

Signed-off-by: Tianyu Guo <guoty@inferact.ai>

gty111 · 2026-08-22T00:09:10Z

Thanks @gty111! LGTM

Thanks for the review! All addressed

njhill · 2026-08-22T02:07:45Z

/ci run

github-actions · 2026-08-22T02:08:02Z

✅ Triggered Buildkite CI #85144 for commit 9337479a9549.

gty111 · 2026-08-22T02:26:59Z

/ci run

github-actions · 2026-08-22T02:27:17Z

✅ Triggered Buildkite CI #85146 for commit 003ee4c0c454.

gty111 added 3 commits August 20, 2026 23:12

[V2] Move encoder-only model runner logic into one module

11a0441

Signed-off-by: Tianyu Guo <guoty@inferact.ai>

[Config] Turn off prefix caching for encoder-only instances

b090a6c

Signed-off-by: Tianyu Guo <guoty@inferact.ai>

[Bugfix][V2] Wait for the prior encoder-only step before reusing its …

f6445fd

…input buffers Signed-off-by: Tianyu Guo <guoty@inferact.ai>

gty111 requested review from ProExpertProg, WoosukKwon, houseroad, mgoin, njhill, robertgshaw2-redhat, tlrmchlsmth, yewentao256 and youkaichao as code owners August 20, 2026 23:33

claude Bot reviewed Aug 20, 2026

View reviewed changes

mergify Bot added the mrv2 Model Runner V2 specific label Aug 20, 2026

njhill reviewed Aug 21, 2026

View reviewed changes

Comment thread vllm/v1/worker/mm_encoder_model_runner.py Outdated

Comment thread vllm/v1/worker/mm_encoder_model_runner.py Outdated

njhill reviewed Aug 21, 2026

View reviewed changes

Comment thread vllm/config/vllm.py Outdated

njhill reviewed Aug 21, 2026

View reviewed changes

gty111 added 2 commits August 21, 2026 21:38

gty111 requested review from ApostaC, alexm-redhat, heheda12345, ivanium, orozery and ywang96 as code owners August 21, 2026 22:01

mergify Bot added the scheduler label Aug 21, 2026

[V2] Drop two comments from the encoder-only runner

230db0d

Signed-off-by: Tianyu Guo <guoty@inferact.ai>

njhill reviewed Aug 21, 2026

View reviewed changes

Comment thread vllm/v1/worker/mm_encoder_model_runner.py Outdated

[V2] Hoist the input event into a local

9337479

Signed-off-by: Tianyu Guo <guoty@inferact.ai>

njhill approved these changes Aug 22, 2026

View reviewed changes

njhill added the ready ONLY add when PR is ready to merge/full CI is needed label Aug 22, 2026

njhill enabled auto-merge (squash) August 22, 2026 02:07

Merge branch 'main' into mm-encoder-model-runner

003ee4c

Uh oh!

Uh oh!

Conversation

gty111 commented Aug 20, 2026

Purpose

Test plan and results

AI assistance

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

njhill left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

njhill left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

njhill left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gty111 commented Aug 22, 2026

Uh oh!

njhill commented Aug 22, 2026

Uh oh!

github-actions Bot commented Aug 22, 2026

Uh oh!

gty111 commented Aug 22, 2026

Uh oh!

github-actions Bot commented Aug 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

njhill left a comment •

edited

Loading