[Bugfix] Restore multimodal support on the plain "vllm" throughput backend - #52168
Open
mganczarenko wants to merge 1 commit into
Open
[Bugfix] Restore multimodal support on the plain "vllm" throughput backend#52168mganczarenko wants to merge 1 commit into
mganczarenko wants to merge 1 commit into
Conversation
…ckend get_requests() only let multimodal content through "vllm-chat", even though the plain "vllm" backend also forwards multi_modal_data straight to LLM.generate. --backend vllm always computed an empty allowlist and self-rejected (e.g. ASR/Whisper datasets). Regression from vllm-project#50981. Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com> Signed-off-by: Michal Ganczarenko <michal.ganczarenko@intel.com>
Member
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
vllm bench throughput --backend vllmcrashes on any multimodal dataset (e.g. ASR/Whisper) with:ValueError: Multi-modal content is not supported on backend 'vllm'; use one of [].
Regression from #50981.
get_requests()computed the multimodal-backend allowlist from the backend already selected instead of from what the backend can actually handle:This is circular — it only lets the already-selected backend through the gate, and only if it's literally "vllm-chat". So --backend vllm always computes mm_backends=() and self-rejects, even though _run_vllm_requests (the function that executes --backend vllm requests) already
forwards multi_modal_data straight into LLM.generate — it was always multimodal-capable, same as vllm-chat. Only hf (asserts prompts must be str) and mii (text-only) genuinely can't carry multimodal content, so the allowlist should be static rather than derived from the chosen
backend:
Test Plan
openslr/librispeech_asr, asserts get_requests() resolves without raising.
Test Result
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.