Skip to content

Revert "[Bugfix][MoE] Tune FlashInfer experts to scheduler token limit" (#52989) - #53186

Merged
mgoin merged 1 commit into
vllm-project:mainfrom
vllm-agent:auto-revert/pr-52989
Aug 21, 2026
Merged

Revert "[Bugfix][MoE] Tune FlashInfer experts to scheduler token limit" (#52989)#53186
mgoin merged 1 commit into
vllm-project:mainfrom
vllm-agent:auto-revert/pr-52989

Conversation

@vllm-agent

Copy link
Copy Markdown
Contributor

Reverts #52989 ("[Bugfix][MoE] Tune FlashInfer experts to scheduler token limit", merged as bfb6c1349).

Why

Nightly build #84887 (bfb6c1349, the merge commit of #52989) turned :nvidia: (B200) LM Eval PCP red for the first time. That job was green in the previous four nightlies (84473, 84555, 84687, 84753).

evals/gsm8k/test_gsm8k_correctness.py::test_gsm8k_correctness[GLM-5.2-NVFP4-TP2-PCP2-EP] now dies with Server exited unexpectedly.; every one of the four workers raises:

MemoryError: CUDA out of memory. Tried to allocate 12.22 GiB.
GPU 0 has a total capacity of 178.35 GiB of which 9.68 GiB is free.
  File ".../fused_moe/experts/flashinfer_cutlass_moe.py", line 368, in apply
  File ".../flashinfer/fused_moe/core.py", line 636, in cutlass
  File "<unknown>", line 0, in FusedMoeRunner::getWorkspaceInfo(...)

Line 368 is the flashinfer_cutlass_fused_moe(...) call this PR modified. The job runs with max_num_batched_tokens=32768, so fi_moe_largest_bucket() returns max(32768 * dp_size, 8192) and the FlashInfer CUTLASS autotune bucket jumps 8192 -> 32768, quadrupling the workspace getWorkspaceInfo reserves.

fi_moe_largest_bucket's own docstring warns that "overestimation may be dangerous, increasing tuning-cost and memory use" and that PCP is not modelled by the estimate — which is exactly the configuration that broke. The sibling GLM-5.2-NVFP4-TP1-PCP4-EP config, which has a smaller per-rank footprint, still passes (0.9325).

Related: open issue #51071 tracks earlier OOMs in this job.

Note for reviewers

This is a draft. The underlying autotune-coverage problem #52989 set out to fix is real; a narrower fix that caps the tuned bucket (or accounts for PCP/EP memory headroom) would be preferable to this revert. Reverting is proposed only to get the B200 PCP eval green while that fix is prepared. The revert applied cleanly with no conflicts and is an exact inverse of the original diff (6 files, 1 insertion, 12 deletions).

Auto-generated by CI failure analyzer.

@mergify mergify Bot added quantization nvidia bug Something isn't working labels Aug 21, 2026
@jeejeelee

Copy link
Copy Markdown
Member

cc @mgoin

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

@mgoin

mgoin commented Aug 21, 2026

Copy link
Copy Markdown
Member

Let's just revert for now to avoid causing OOMs for users

@github-project-automation github-project-automation Bot moved this to Ready in NVIDIA Aug 21, 2026
@mgoin mgoin added the ready ONLY add when PR is ready to merge/full CI is needed label Aug 21, 2026
@github-actions

Copy link
Copy Markdown

@vllm-agent, CI is now available for this PR.

  • /ci run starts upstream CI; /amd-ci run starts AMD CI only.
  • /ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
  • /amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
  • /ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

@mgoin
mgoin merged commit ba53da6 into vllm-project:main Aug 21, 2026
16 of 23 checks passed
@github-project-automation github-project-automation Bot moved this from Ready to Done in NVIDIA Aug 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working nvidia quantization ready ONLY add when PR is ready to merge/full CI is needed

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants