Commit 99c2f4a
committed
k3 B200: carry spec-decode-under-PP into the container as a patch
Upstream refuses this outright -- vllm/v1/worker/gpu/spec_decode/dspark/utils.py
on main still raises NotImplementedError when the PP group is larger than one,
and SpeculativeConfig hands the draft the target's pipeline_parallel_size, which
asks K3DSparkForCausalLM for a SupportsPP it does not declare. That is the error
all seven of our earlier PP2 arms died on.
The support lives in our vLLM fork as 2111011d33 (adopting vllm-project/vllm#50514),
540423f2c7 and 503820ebdd: the draft stops inheriting PP size and runs whole on
the last stage, aux hidden states relay between stages as IntermediateTensors
behind a Kimi-K3-only opt-in, the last stage loads the real embedding table
instead of aliasing a PPMissingLayer, and the draft loader drops fastsafetensors
under PP because its collective runs over group.WORLD.
Applied with patch(1) after a --dry-run, and verified afterwards that the refusal
is gone and the draft's PP size is pinned to 1. A mismatch fails the job rather
than running a half-patched engine; the fork's base is 38a466e7b6 and the image
is 5894fdf98, so drift will eventually break it and should be loud when it does.
Any arm using this must ship with a GSM8K run. The adopting commit records that
draft tokens previously failed to reach earlier stages and the first rank
embedded PLACEHOLDER_TOKEN_ID(-1) -- silent corruption that ignore_eos perf arms
cannot see.1 parent 5805859 commit 99c2f4a
2 files changed
Lines changed: 721 additions & 0 deletions
0 commit comments