Exp/hisparse routed experts - #49121
Open
S1ro1 wants to merge 9 commits into
Open
Conversation
…SA + GLM-5.2 indexCache plan-once & overlapped prefetch Host-resident sparse-MLA KV with per-request GPU hot-buffer decode, ported from SGLang's HiSparse to vLLM's attention-backend boundary. The full MLA KV lives in a pinned host pool (hisparse_config.host_pool_gib, per rank); decode is served from a per-request LRU working set that a CUDA swap-in kernel resolves against the indexer top-k each step, keyed by global KV slot id so prefix caching keeps working and recycled slots are invalidated once, at block-assignment time. Capture-safe under FULL_DECODE_ONLY: kernels skip graph-padding rows via a device-resident num_real_reqs scalar, and the newest-row host backup runs inline on the capture stream. NIXL: host MLA regions register as DRAM (device_id 0) and READs split into DRAM/VRAM descriptor lists so prefill->decode transfer works at TP>1. Completion notifications defer until all of a request's transfers finish; failure reporting defers until every handle is terminal (a posted RDMA READ cannot be aborted), with exactly-once semantics. Late-duplicate completions are not re-reported. Mooncake-store offload composes via MultiConnector: the store worker registers pinned-host MLA and device indexer segments side by side; store hits load directly into the host pool. Cross-layer packed registration is rejected. register_buffer failure raises; close() drains queues bounded. GLM-5.2 indexCache (index_topk_freq=4): only a group's "full" layer resolves the LRU, writing a static group-shared plan; the indexer-less "shared" layers replay it via a pure gather kernel. The full layer prefetches the shared layers' gathers on a copy stream, overlapping PCIe with the intervening compute; the fork/join captures into the decode graph. Mixed batches (llm-d shortcut routing: short-delta requests run chunked local prefill on decode) are row-split: decode rows are served from the hot buffer via swap_in's mixed mode, only prefill rows' contexts stage host->GPU. write_newest_rows clamps to scheduler-produced rows (DP/graph padding), and the swap-in kernel carries corruption tripwires (out-of-range LRU/evict slots degrade to re-misses). Top-k hardening ported from sglang upstream: tie-overflow output padding (4c5fe42be4) and a spec-decode-gated seq-len clamp (bbc537035a). Teardown is drain-then-free: the pool is pinned via cudaHostRegister on page-aligned pageable allocations (exact-size; the caching host allocator pow2-rounds per layer) and deterministically cudaHostUnregister'ed in GPUModelRunner.shutdown() before poisoned-context-sensitive cleanup; on an unusable context the pool is left for kernel exit reclaim. NIXL worker shutdown is exception-safe and drops kv-cache tensor refs so pool release never depends on connector lifetime. Local prefill on decode instances writes straight to the host pool and stages only unique referenced blocks. Enforced at startup: DSA models with index_topk, kv_consumer intent, no speculative decoding, no DBO, compiled CUDA ops required. Validated on H200: kernel fuzz vs Python reference, plan-once equivalence, capture/replay, NIXL failure suites, PD token-parity smokes, and multi-node production canaries.
Scheduler-side capture of per-layer routed expert IDs into a scheduler-owned MAP_SHARED mmap slot buffer shared with the worker, returned to the caller via ModelRunnerOutput. Only physical slot indices (block_id * block_size + offset) cross the process boundary, never the routing payload; prefix-cache hits and block reuse follow for free. Includes MLA DSA / HMA (multi KV-group) support. Replaces the single-file routed_experts_capturer with the routed_experts_capture package. No KV offload here: enable_return_routed_experts stays incompatible with any KV connector; CPU/disk offload is added in the following commits. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Follow the KV blocks' offload lifecycle: store/load per-block routing to a CPU offload buffer along each OffloadingConnector transfer job, so routing survives GPU eviction. Adds the manager's offload buffer + block-map math and the scheduler offload hooks. Relaxes the enable_return_routed_experts KV-compat check to allow OffloadingConnector + CPUOffloadingSpec (kv_role=kv_both). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Keep routed-expert CPU offload transfers scoped to the native offloading connector. NIXL P/D leaves routing local to each engine so the deployment router can splice prefill and decode rows without treating NIXL metadata as CPU-offload jobs. Co-authored-by: OpenAI Codex <codex@openai.com>
Allow the asymmetric P/D topology where prefill composes NIXL with native CPU KV offload and decode uses direct NIXL. Extract the OffloadingConnector child metadata so routed-expert rows follow CPU KV transfers, and adapt the transfer jobs to the v0.24 tuple API. Co-authored-by: OpenAI Codex <codex@openai.com>
S1ro1
requested review from
ApostaC,
Harry-Chen,
LucasWilkinson,
MatthewBonanni,
NickLucche,
ProExpertProg,
WoosukKwon,
alexm-redhat,
heheda12345,
houseroad,
mgoin,
njhill,
orozery,
pavanimajety,
robertgshaw2-redhat,
tlrmchlsmth,
xuechendi,
yewentao256,
youkaichao,
ywang96 and
zyongye
as code owners
July 19, 2026 18:46
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Extend the existing Blackwell model safety gate to GLM DSA models and online quantization. Keep explicit --moe-backend deep_gemm as an override, but prevent the global DeepGEMM enable from re-forcing unsafe FP8 MoE kernels after the model-level gate fires. The MoE selector consistency follows vllm-project#47258, extended for the online GLM deployment. Co-authored-by: Letemoin <letemoin.bug@gmail.com> Co-authored-by: OpenAI Codex <codex@openai.com>
Keep DeepGEMM available process-wide for sparse attention while deriving block-FP8 activation scale rounding from the selected MoE backend. This prevents FlashInfer, Triton, and CUTLASS modular MoE paths from inheriting DeepGEMM's UE8M0 format through the global environment. Co-authored-by: OpenAI Codex <codex@openai.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Purpose
Test Plan
Test Result
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.