Skip to content

[Model] Skip unused Jina V5 output layers - #52037

Merged
noooop merged 2 commits into
vllm-project:mainfrom
BabyDrangoner:perf/jina-v5-skip-lm-head
Aug 13, 2026
Merged

[Model] Skip unused Jina V5 output layers#52037
noooop merged 2 commits into
vllm-project:mainfrom
BabyDrangoner:perf/jina-v5-skip-lm-head

Conversation

@BabyDrangoner

@BabyDrangoner BabyDrangoner commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Purpose

Jina Embeddings V5 models are pooling-only, but their vLLM wrappers inherit
causal-LM classes. Because these wrappers already declare themselves as pooling
models, they bypass the generic pooling adapter that replaces generation-only
output layers. The encoder/nano variant therefore retained an unused
ParallelLMHead with shape [128256, 768].

This change applies the existing pooling-model no_init_weights pattern to
both Jina V5 wrappers, replacing ParallelLMHead and LogitsProcessor with a
StageMissingLayer. The decoder/small checkpoint ties its output head to the
input embedding (VocabParallelEmbedding), so that shared embedding remains
intact.

On an NVIDIA RTX PRO 6000 Blackwell (TP=1), the nano checkpoint's loaded model
state changed as follows:

Metric Before After
Registered parameter bytes 620,533,248 423,532,032
Unused lm_head bytes 197,001,216 0
CUDA memory delta after model load 1,283,457,024 1,086,324,736

This removes 187.875 MiB of resident parameters (31.75%) and approximately
188 MiB of post-load CUDA memory. This is a resident-memory result; it is not a
claim that the transient module-construction peak is eliminated.

I searched open vLLM PRs and issues for Jina V5/Embeddings V5 combined with
lm_head, output-layer, and pooling terms and found no direct or semantic
duplicate. #32757 is the already-merged generic pooling mechanism used as the
implementation precedent, not another Jina-specific fix.

AI assistance (OpenAI Codex) was used to help investigate, implement, and test
this change. I reviewed the complete diff and the validation results.

Test Plan

  • Load and embed two natural prompts with both the base and patched
    jinaai/jina-embeddings-v5-text-nano checkpoints; compare parameter and
    post-load CUDA memory plus float32 embedding bytes.
  • Repeat the embedding comparison with
    jinaai/jina-embeddings-v5-text-small to cover its tied embedding.
  • Run the existing Jina pooling-model, adapter, and registry tests, then all
    pre-commit hooks applicable to the changed production file.

Commands:

.venv/bin/python -m pytest -q \
  tests/models/language/pooling/test_jina_embeddings_v5.py \
  tests/models/test_adapters.py \
  'tests/models/test_registry.py::test_registry_imports[JinaEmbeddingsV5Model]'

uvx pre-commit run --files \
  vllm/model_executor/models/jina.py

git diff --check

Test Result

  • Pytest: 12 passed.
  • Pre-commit: all applicable hooks passed, including Ruff, formatting, mypy,
    SPDX, forbidden-import, and configuration checks.
  • Nano real-model A/B:
    • Both runs produced embedding SHA-256
      f355f21e03004acfc86af056b3610282290563ab563dd5f9d04e5768bca06688.
    • Both output vectors had shape [768] and L2 norm 1.0.
    • Resident parameter and CUDA memory reductions are reported above.
  • Small real-model A/B:
    • Both runs retained 1,192,099,840 parameter bytes and no independent
      lm_head, confirming the tied input embedding was not removed.
    • Both produced embedding SHA-256
      c740490981d58e61cee407638b98c0ad724674e11341bc94086a7fa19a675942.
    • Both output vectors had shape [1024] and L2 norm 1.0.

Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: kiroxu <148877251+BabyDrangoner@users.noreply.github.com>
@BabyDrangoner
BabyDrangoner marked this pull request as ready for review August 12, 2026 19:03
@BabyDrangoner
BabyDrangoner requested a review from noooop as a code owner August 12, 2026 19:03

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

Comment thread tests/models/language/pooling/test_jina_embeddings_v5.py Outdated
Co-authored-by: Codex <noreply@openai.com>
Signed-off-by: kiroxu <148877251+BabyDrangoner@users.noreply.github.com>
@BabyDrangoner
BabyDrangoner requested a review from noooop August 13, 2026 02:10

@noooop noooop left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for your contribution.

@noooop noooop added the verified Run pre-commit for new contributors without triggering other tests label Aug 13, 2026
@noooop
noooop enabled auto-merge (squash) August 13, 2026 02:32
@github-actions github-actions Bot added the ready ONLY add when PR is ready to merge/full CI is needed label Aug 13, 2026
@noooop

noooop commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #83653 for commit 2f5766a43365.

@BabyDrangoner

Copy link
Copy Markdown
Contributor Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #83673 for commit 2f5766a43365.

@noooop
noooop merged commit 89c8401 into vllm-project:main Aug 13, 2026
95 of 96 checks passed
zyp2014 pushed a commit to zyp2014/vllm that referenced this pull request Aug 21, 2026
Signed-off-by: kiroxu <148877251+BabyDrangoner@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready ONLY add when PR is ready to merge/full CI is needed verified Run pre-commit for new contributors without triggering other tests

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants