Skip to content

[Bugfix][Kimi K3] Enable deferred MoE finalization before weight loading - #53327

Merged
zyongye merged 1 commit into
vllm-project:mainfrom
zyongye:fix/k3-deferred-tail-finalize-init
Aug 22, 2026
Merged

[Bugfix][Kimi K3] Enable deferred MoE finalization before weight loading#53327
zyongye merged 1 commit into
vllm-project:mainfrom
zyongye:fix/k3-deferred-tail-finalize-init

Conversation

@zyongye

@zyongye zyongye commented Aug 21, 2026

Copy link
Copy Markdown
Member

Summary

  • Enable K3 deferred MoE finalization from the selected monolithic expert class during runner construction.
  • Keep the existing hidden-size and parallel-topology guards.
  • Log when deferred top-k finalization is active and add a regression test for the pre-weight-loading lifecycle.

Root cause

#53152 checked quant_method.moe_kernel in LatentMoERunner.__init__. Mxfp4MoEMethod initializes that field to None and only creates the kernel later in process_weights_after_loading, so the deferred path was always disabled during runner construction even when the supported FlashInfer TRTLLM MXFP4 expert class had already been selected.

The tail operator must know whether it consumes an UnfinalizedMoEOutput during construction so its CuTeDSL warmup units are registered with the correct top-k shape. This change therefore uses the already-selected expert class at that lifecycle point.

Duplicate-work check

Tests

.venv/bin/python -m pytest \
  tests/models/kimi_k3/test_latent_moe_tail.py::test_deferred_finalize_enabled_before_moe_kernel_setup \
  tests/models/kimi_k3/test_latent_moe_tail.py::test_latent_moe_tail_deferred_finalize_matches_finalized \
  -q

Result: 2 passed on 8x NVIDIA B300. The second test compares deferred and finalized outputs across TP8.

.venv/bin/pre-commit run ruff-check --files \
  vllm/models/kimi_k3/nvidia/latent_moe_runner.py \
  tests/models/kimi_k3/test_latent_moe_tail.py
.venv/bin/pre-commit run ruff-format --files \
  vllm/models/kimi_k3/nvidia/latent_moe_runner.py \
  tests/models/kimi_k3/test_latent_moe_tail.py

Result: both passed.

Serving validation: Kimi-K3 TP8 selected the FlashInfer TRTLLM MXFP4/MXFP8 backend, reported deferred top-k finalization enabled, and completed 40/40 requests with no failures.

AI assistance

OpenAI Codex was used to diagnose the initialization-order issue, draft the code and test, run validation, and prepare this PR. The human submitter must review every changed line and understand and defend the change end-to-end.

Co-authored-by: OpenAI Codex <codex@openai.com>
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
@mergify mergify Bot added kimi k3 bug Something isn't working labels Aug 21, 2026
@zyongye
zyongye marked this pull request as ready for review August 21, 2026 22:10

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@zyongye

zyongye commented Aug 21, 2026

Copy link
Copy Markdown
Member Author

/ci run

@zyongye
zyongye enabled auto-merge (squash) August 21, 2026 22:16
@github-actions github-actions Bot added the ready ONLY add when PR is ready to merge/full CI is needed label Aug 21, 2026
@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #85122 for commit bbbf7405d192.

@zyongye
zyongye merged commit e9d1398 into vllm-project:main Aug 22, 2026
72 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working k3 kimi ready ONLY add when PR is ready to merge/full CI is needed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants