[safety] nvfp4 MTP slugs run the vllm#43559 corruption pair (MTP + prefix-caching) UNPATCHED — pr48375 wiring note is stale

[noonghunna]

Found auditing #765's compose lineage. Two shipped slugs form the vllm#43559 corruption pair (DeltaNet-hybrid MTP + prefix caching) without the pr48375 mamba-drop mitigation:

• single/nvfp4/mtp.yml: entrypoint injects --speculative-config {"method":"mtp",...} by DEFAULT (SPEC_ARGS, ~L134) AND ships a literal --enable-prefix-caching (~L183 — not even the PREFIX_CACHE_ARG seam). Not in pr48375's load_bearing_when list.
• dual/nvfp4/mtp.yml: same default-on SPEC_ARGS injection (~L152) — directly contradicting patches.yml's wiring note, which exempts it as "ships no MTP arg, so the pair cannot form." The note predates the SPEC_ARGS entrypoint pattern.

Root cause of the miss: the #736 patch tiering keyed on composes shipping a literal --speculative-config line; the SPEC_ARGS indirection made these two invisible to that grep. Same failure shape as #786's signature-scoped guard: scoped to the syntax, not the property.

Proposed fix (hotfix-grade)

1. Immediate: convert both slugs' prefix-caching to the PREFIX_CACHE_ARG seam defaulted OFF until the patch is wired (loses prefix-cache TTFT on these slugs; corruption loses more).
2. Then: add both slugs to pr48375 load_bearing_when + wire the overlay; correct the stale patches.yml note; re-run the patch-attribution + status-drift gates.
3. Guard the class: test-patch-attribution (or a new check) should detect the property — any compose whose entrypoint can inject --speculative-config while prefix caching is enableable — not the literal flag.

Heads-up owed to @seanyourhighness (#765 runs the single slug with the pair active) once mitigation ships.

[seanyourhighness]

Taking the heads-up — I'm the #765 rig, and I've been running single/nvfp4/mtp.yml's shape with the pair active all day. Tested your proposed hotfix directly, and on sm_120 it does not fix the corruption — it converts it into a hard crash. Data below, because I'd rather you have it before step 1 ships.

You also warned me about #43559 in #765 §3(c) on 2026-07-26. I dismissed it then because the Ornith run had prefix caching off and MTP off. Every MTP arm below has both halves of the pair active, so the warning was applicable and I was wrong to set it aside.

Rig/stack: RTX 5090 (sm_120) 32,607 MiB, driver 610.62, CUDA 13.0, WSL2, 575 W stock. vLLM 0.25.1 (752a3a5) + my SM120 NVFP4 V-scale write fix, FlashInfer 0.6.15.post1. nvidia/Qwen3.6-27B-NVFP4, nvfp4 KV, 131,072 ctx, max-num-seqs 6, batched 4096, --speculative-config {"method":"mtp","num_speculative_tokens":3}, --language-model-only. Correctness probe = planted-codeword recall at ~32k prompt tokens; throughput = club bench.sh; acceptance = traffic-weighted from SpecDecoding metrics (idle samples excluded — tail -1 reports a working drafter as 0.0%).

The matrix

#	patches	prefix cache	cudagraph	narrative / code	acceptance	32k needle
A	stock	on	FULL_AND_PIECEWISE	98.88 / 111.83	57.8% (len 2.74)	FAIL
C	49891	on	FULL_AND_PIECEWISE	105.84 / 104.51	57.8% (len 2.73)	FAIL
G	49891+46324	on	FULL_AND_PIECEWISE	100.61 / 113.45	60.3% (len 2.81)	FAIL
I	49891+46324	on	FULL_AND_PIECEWISE (control)	100.81 / 108.60	59.2% (len 2.71)	FAIL
H	49891+46324	off	FULL_AND_PIECEWISE	—	—	CRASH
B	stock	on	PIECEWISE	—	—	CRASH
F	49891+46324	on	PIECEWISE	58.03 / 72.19	65.8% (len 2.97, peak 3.51)	PASS

Baseline for reference — MTP off, same everything else: 58–66 / 66.4 tok/s, KV pool 373,797, needle PASS at 5,042 / 20,044 / 32,030 / 64,036 prompt tokens.

Why your step 1 worries me

Arm F passes with the pair fully active — MTP on, prefix caching on. And arm H, which is your hotfix (prefix caching off), crashes with CUDA error: an illegal memory access was encountered.

Across seven arms the only consistent discriminator is cudagraph mode, not prefix caching. PIECEWISE is clean; FULL capture is corrupt or dead regardless of the prefix-cache flag.

Best reading I have: the #43559 pair may well create latent recurrent-state poisoning, but FULL cudagraph capture is what makes it observable — and FULL capture is only reached because MTP raises uniform_decode_query_len from 1 to 1 + num_speculative_tokens = 4 (cudagraph_dispatcher.py:37), changing the capture shapes. Without MTP, the identical config with prefix caching on is needle-clean to 64k.

So on sm_120, defaulting prefix caching off would cost the TTFT and leave the corruption — while removing the one arm I can show working.

What actually worked

vllm#46324 (open since 2026-06-22, one-line gate change) fixes the PIECEWISE crash. adjust_cudagraph_sizes_for_spec_decode() is gated to decode_mode() == FULL, so PIECEWISE silently skips spec-decode capture-size alignment; a partial-acceptance step then dispatches to a graph keyed on a non-multiple of uniform_decode_query_len and reads slot_mapping/positions at another graph's offsets. That is exactly my arm B. With it, arm F runs clean.

Worth noting acceptance also recovers in the working arm — 57.8% → 65.8%, accept length 2.74 → 2.97 (peak 3.51), measured over 22 samples instead of 4-5 — moving toward your ~74% / 3.2. Consistent with the target having been verifying drafts against corrupted attention.

Two more datapoints for the guard discussion:

• vllm#49891 is speed-only here: +7% narrative, acceptance identical to three significant figures with and without. It does not touch correctness.
• vllm#49757 is actively harmful on 0.25.1 — applies and compiles cleanly, then converts the silent corruption into a hard crash in the same GDN path by re-staging state from zeroed dummy block tables on every FULL replay. It targets current main. Textual apply ≠ semantic compatibility; worth a version guard if it ever enters the overlay set.

Suggestion

Your §3 instinct — "detect the property, not the syntax" — is right, but I'd widen the property. On sm_120 the pair alone is neither necessary nor sufficient for observable corruption; MTP + FULL-capable cudagraph mode is what does it. A guard keyed only on --speculative-config + prefix-caching would both miss the FULL-capture cases and flag arm F, which is the one configuration I can show working.

Happy to run anything else on this rig. Fault localization and the upstream reports are in my next comments on vllm#49010 and #46324.

[noonghunna]

This is the most useful bug report this repo has had in a while. Thank you for running it before step 1 shipped rather than after — and for the seven arms, not one.

Taking the headline straight: on sm_120 your matrix supersedes my step-1 reasoning. I derived the hotfix from the #43559 signature (MTP + prefix caching = the pair, therefore break the pair) without any way to run the config. You ran the actual hardware and the actual patch stack. Arm H is my hotfix and it crashes; arm F has both halves of the pair live and passes. That is not a close call, and I'm not going to argue the signature against your data.

Worse than that: step 1 also cost you TTFT for nothing. The prefix-off tax is real and measured elsewhere on this stack (#710: 7.4x TTFT on a 26.7K held prefix at 89.5% hit rate — a production lane, not these slugs). So on your rig #811 bought a slower engine and kept the corruption, which is the worst of both. Noted, and it's the part I most want to fix.

Three things I checked against our pinned image, since they bear on the fix

Ran these inside vllm/vllm-openai:v0.25.1 (the tag both slugs pin), not from docs:

1. Our composes ship your failing mode by omission. Neither slug sets a cudagraph flag, and the v1 default is FULL_AND_PIECEWISE (config/vllm.py:249). So every user of these two slugs, on every architecture, is in your FULL-capture column by default. Nobody opted into it.
2. Your mechanism claim holds on our exact pin. cudagraph_dispatcher.py: self.uniform_decode_query_len = 1 + self.vllm_config.num_speculative_tokens — 1 -> 4 at num_speculative_tokens=3, as you said. MTP is what moves the capture shape.
3. Flag syntax footgun on 0.25.1, in case you hit it too. -O.cudagraph_mode=PIECEWISE does not work on this version — FlexibleArgumentParser rewrites any -O<x> to --optimization-level <x>, so that string lands as --optimization-level .cudagraph_mode=PIECEWISE and cudagraph_mode stays at the default. Verified by parsing it. The form that actually binds is --compilation-config '{"cudagraph_mode":"PIECEWISE"}' (alias -cc), which the config's string validator accepts. Worth confirming which form your PIECEWISE arms used, because the silent-no-op version would read as "PIECEWISE didn't help."

What the matrix establishes, and what I think is still open

Established (I'm treating this as settled for your stack): on sm_120, with the SM120 NVFP4-KV route and your V-scale write fix in the image, cudagraph mode is the discriminator; FULL capture with MTP is corrupt or dead regardless of the prefix-cache flag; PIECEWISE + vllm#46324 is clean and recovers acceptance.

Open, and I can't close any of these from here — this rig is 2x 3090 sm_86, there is no Blackwell on it:

• KV dtype / decode route. Every arm in your matrix runs nvfp4 KV. Our shipped default on both slugs is --kv-cache-dtype fp8 on a stock pinned image. vllm#49010's own table is the reason this matters: XQA decode + FULL graphs fails on fp8 KV too (0/4), but stock decode passes 4/4. So the question that decides our default isn't FULL-vs-PIECEWISE, it's whether a stock v0.25.1 on sm_120 with fp8 KV routes decode through XQA at all, or only does so once the NVFP4-KV route is in the build. If it's the latter, the exposure is a property of patched community builds and the shipped default is a different animal.
• A counter-arm from this slug's history that your matrix doesn't have. --enable-prefix-caching was hard-on in single/nvfp4/mtp.yml from the day it landed (2026-07-06) until hotfix: prefix caching OFF on nvfp4 MTP slugs — unpatched #43559 pair (#810) #811 removed it two days ago. So the two 5090 validations in [bench] vllm/qwen-27b-single-nvfp4 OOM 5090 #613/[bug] running #608 qwen-27b nvfp4 on single 5090 - #617 ran MTP n=3 + prefix caching + fp8 KV + the FULL_AND_PIECEWISE default on a stock image — the pair fully active, FULL capture active — and came back verify-stress all-pass, NIAH-clean to 91K, MTP accept ~3.2. Different reporter, different probe (NIAH vs your planted codeword at 32k), and I can't audit their build — if either [bench] vllm/qwen-27b-single-nvfp4 OOM 5090 #613/[bug] running #608 qwen-27b nvfp4 on single 5090 - #617 run was on a patched image rather than the stock pin, say so and I'll strike this control. But it's the closest thing to an "arm A on fp8 KV, stock image" control that exists, and it passed. Read together with #49010 that points at the XQA route rather than at FULL capture as such. It does not weaken your result on your stack; it constrains how far I can carry it into the default.
• Which arms carried the NVFP4-KV enablement. Related to the above and probably a one-line answer from you: your stack line lists 0.25.1 + your V-scale fix as common to all arms, and the patches column lists 49891/46324 as the extras — but nvfp4 KV on a genuinely stock 0.25.1 should be refused/crash (vllm#43562). So does "stock" in arms A and B mean stock plus your always-on NVFP4-KV base, or does it fall back to fp8 KV? If A is fp8-KV-on-stock, that changes my read of everything above.
• Whether arm H reproduces without your patches, on another sm_120 rig. Right now the crash is one data point on one build.
• sm_86. Nobody has run this matrix on Ampere. I can partially fix that (below), with a caveat.

Proposed revision — and one correction to the obvious version of it

The obvious reading of your matrix is "pin PIECEWISE." That reading is wrong on its own, and your arm B is why: stock + PIECEWISE crashes. The delta from what we ship today to your passing arm F is not one change, it's two:

1. vendor vllm#46324, and
2. pin cudagraph_mode=PIECEWISE, and then
3. restore prefix caching (undoing hotfix: prefix caching OFF on nvfp4 MTP slugs — unpatched #43559 pair (#810) #811), riding the pr48375 overlay that [safety] nvfp4 MTP slugs run the vllm#43559 corruption pair (MTP + prefix-caching) UNPATCHED — pr48375 wiring note is stale #810 step 2 was already going to wire onto these slugs.

Shipping (2) without (1) would be actively worse than today, and not only on Blackwell: #46324's own repro is fp8 KV + DFlash on sm_120/121, i.e. the misalignment is KV-dtype-agnostic and generic to any spec-decode under PIECEWISE. Pinning PIECEWISE without it would hand every graph-capturing arch a periodic illegal-address crash, this rig included.

Concretely, per slug (seam so it stays overridable; the mode is interpolated on its own rather than the whole JSON, because a ${VAR:-{...}} default breaks on the nested brace in compose interpolation):

      # #810: MTP raises uniform_decode_query_len to 1+n=4; FULL capture on that
      # shape is where the corruption becomes observable on sm_120
      # (@seanyourhighness's 7-arm matrix). Requires the vendored vllm#46324 —
      # PIECEWISE + spec-decode without it is an illegal-address crash, not a fix.
      - --compilation-config
      - '{"cudagraph_mode":"${CUDAGRAPH_MODE:-PIECEWISE}"}'
      # restored (#811 reverted): the pair is not the discriminator here, and
      # prefix-off cost TTFT for nothing. pr48375 overlay wired per #810 step 2.
      - "${PREFIX_CACHE_ARG:---enable-prefix-caching}"

This is a proposal, not a shipped decision, and I'd like your confirmation first: does that compose shape match your arm F? Specifically — same cudagraph binding, prefix caching on, MTP n=3 untouched, #46324 as the only correctness-load-bearing overlay (you measured #49891 as speed-only with acceptance identical to 3 s.f., so I'd leave it out of the correctness path). If it does, I'll take it as step 1-revised and credit the matrix in the compose header and the UPSTREAM row.

Available today, zero patches, if you want the corruption gone before any of this lands: VLLM_ENFORCE_EAGER=1 — both slugs already have that seam in the entrypoint (it was put there as a Cliff-2 mitigation). #49010's table gives enforce-eager 4/4 needle and 100% acceptance. It costs throughput, but it's an env var, it needs no image rebuild, and it beats what #811 currently gives you.

On an sm_86 control arm: the dual slug does boot on this 2x 3090 rig (validated 2026-07-11) — but via the Marlin W4A16 weight-only fallback with fp8 KV and no XQA anywhere, a different kernel path from top to bottom. A clean run there would not clear sm_120 and a dirty one would be a separate bug, so I'm not treating an Ampere arm as load-bearing for this revision and I'm not gating the fix on producing one. If the XQA question above resolves the way #49010 suggests, the sm_86 column may simply not exist for this bug. The single slug I can't test at all: it needs >24 GB.

Upstream

Your uniform_decode_query_len observation deserves to be on vllm#43559 directly. That issue is currently framed as a pure prefix-cache x MTP KV-state bug, and the thing your matrix adds is that MTP may be gating observability by changing capture shape — which, if it holds, means the fix landscape there is wider than #48375. Same for the arm-H detail: prefix-off converting silent corruption into illegal memory access is unexplained and is exactly the kind of thing that reframes a root cause. That's your finding and your data; I'm not going to file it on your behalf or paraphrase it into a thread as secondhand. If you post it, I'll cross-reference from #810 and add UPSTREAM rows for #46324, #49891 and #49010 pointing at your comment.

Also recording your #49757 datapoint as a do-not-vendor-on-0.25.1 note with a version guard — "applies and compiles cleanly, then converts silent corruption into a hard crash because it targets current main" is precisely the failure mode our patch-vendoring process is supposed to catch and doesn't yet. That one saved us a bad week; it's going in the engine-pin checklist.

On widening the guard

Agreed, and your framing is better than mine. One consequence worth flagging: since FULL_AND_PIECEWISE is the engine default, a guard that greps composes for a cudagraph flag finds nothing — every affected compose is affected by omission. So the property check has to reason about the resolved engine default for the pinned image, not about compose text. That's a bigger change than the one I sketched in §3, and honestly the more useful lesson from this whole thread: our guards keep getting written against the syntax that's visible in the file, and the bugs keep living in what the file doesn't say.

And for the record on #765 §3(c) — you flagged your own dismissal before I could even see the thread. That's not something to apologize for; it's what made this report worth reading.

[noonghunna]

Closing with the maintainer's ruling (2026-08-02): no new patch vendoring — the pr48375 wiring this issue was tracking is not happening. The resolution decomposes as:

• The safety condition is already permanently met: prefix caching defaults OFF on both nvfp4 MTP slugs (the [safety] nvfp4 MTP slugs run the vllm#43559 corruption pair (MTP + prefix-caching) UNPATCHED — pr48375 wiring note is stale #810/hotfix: prefix caching OFF on nvfp4 MTP slugs — unpatched #43559 pair (#810) #811 hotfix), so the vllm#43559 corruption pair (MTP + prefix caching) cannot form on a default boot. That was the interim seam; it is now the standing state.
• Runtime-verified on target hardware, no rig time needed: [bench] 2x 5090 / vllm-qwen36-27b-dual-nvfp4 #849's full boot log on 2× 5090 shows enable_prefix_caching: False with MTP n=3 active — the seam holds through the launcher path in the wild, not just in the compose text.
• The stale headers this issue's title complained about are fixed (74e63db): both composes now state OFF-by-policy with the ruling and drop the 'until the wiring lands' framing (plus the dual's contradictory pre-hotfix 'ON by default' remnant).
• Clear condition: upstream vllm#48375 is still open as of today — the pair-safety un-caveats only when a pinned release contains the merged fix, tracked on the docs/UPSTREAM.md #43559/#48375 row with the same inherit-only policy as #50021. Note the two fixes are on independent merge clocks; they may or may not arrive in the same release.
• The existing pr48375 vendor on the AutoRound/W4A8 MTP composes (validated 2026-07-17, prefix-ON there) is unchanged — the ruling stops new vendoring, it doesn't unwind tested, shipped mitigations.