Your current environment
The output of python collect_env.py
Collecting environment information...
==============================
System Info
==============================
OS : Ubuntu 24.04.4 LTS (x86_64)
GCC version : (Ubuntu 13.3.0-6ubuntu2~24.04.1) 13.3.0
CMake version : version 3.31.6
Libc version : glibc-2.39
==============================
PyTorch Info
==============================
PyTorch version : 2.13.0+cu130
Is debug build : False
CUDA used to build PyTorch : 13.0
==============================
Python Environment
==============================
Python version : 3.12.3 (main, Mar 23 2026, 19:04:32) [GCC 13.3.0] (64-bit runtime)
Python platform : Linux-6.17.0-29-generic-x86_64-with-glibc2.39
==============================
CUDA / GPU Info
==============================
Is CUDA available : True
GPU 0-7 : NVIDIA GeForce RTX 4090 (x8)
Nvidia driver version : 595.71.05
cuDNN version : Could not collect
==============================
CPU / NUMA
==============================
CPU : Intel(R) Xeon(R) Platinum 8368Q CPU @ 2.60GHz x2 (152 threads)
NUMA 节点 : 2 (node0: 0-37,76-113 / node1: 38-75,114-151)
==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.6.15.post1
[pip3] nccl4py==0.4.1
[pip3] numpy==2.2.6
[pip3] nvidia-cudnn-cu13==9.20.0.48
[pip3] nvidia-nccl-cu13==2.29.7
[pip3] tokenspeed-triton==3.8.10.post20260721
[pip3] torch==2.13.0+cu130
[pip3] torchaudio==2.11.0+cu130 / torchvision==0.28.0+cu130 / torchcodec==0.15.0
[pip3] transformers==5.15.0
[pip3] triton==3.7.1
==============================
vLLM Info
==============================
vLLM Version : 0.27.1+backport052.sm89.cu130
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; XPU: Disabled
GPU Topology: GPU0-3 ↔ NUMA0 (NODE), GPU4-7 ↔ NUMA1 (NODE), cross-socket SYS
(P/D split aligned to sockets: prefill on node0 GPUs, decode on node1 GPUs)
==============================
Environment Variables
==============================
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root
Note: the vLLM build is a custom sm89 backport of v0.27.1 (wtdcode/vllm-backport v0.5.2); the three files central to this report — scheduler.py, multi_connector.py, nixl/pull_scheduler.py — were byte-compared against current main (d626108b18) and are identical in the relevant sections.
🐛 Describe the bug
MultiConnector deduplicates async saves across sub-connectors (_extra_async_saves) but performs a bare union for async loads (finished_recving). In a P/D disaggregation setup where the decode engine also runs a second connector doing async loads (LMCache MP), a request can complete via the faster load path while the slower NIXL pull is still in flight; the late NIXL completion is then reported for a request the scheduler already removed, tripping:
File ".../vllm/v1/core/sched/scheduler.py", line 2741, in _update_from_kv_xfer_finished
assert req_id in self.requests
AssertionError
→ EngineDeadError, decode engine dies, service outage until manual restart.
Minimal reproduction
Exact decode-engine connector config (relevant flag only; rest of vllm serve is stock — TP4, fp8_ds_mla, 262K context):
vllm serve <model-path> \
--tensor-parallel-size 4 \
--kv-cache-dtype fp8_ds_mla \
--max-model-len 262144 \
--port 8200 \
--kv-transfer-config '{"kv_connector":"MultiConnector","kv_role":"kv_consumer","kv_connector_extra_config":{"connectors":[{"kv_connector":"NixlConnector","kv_role":"kv_consumer","kv_load_failure_policy":"fail"},{"kv_connector":"LMCacheMPConnector","kv_role":"kv_both","kv_connector_extra_config":{"lmcache.mp.host":"tcp://localhost","lmcache.mp.port":6556}}]}}'Prefill engine is symmetric (kv_producer roles). LMCache server is stock upstream: lmcache server --port 6556 --l1-size-gb 48 --eviction-policy LRU --chunk-size 256.
Traffic needed to trigger: multi-turn agent-style requests with long contexts (~100K tokens) where the decode-side LMCache has a prefix hit from an earlier turn while the router-initiated NIXL push of the same prompt KV is still in flight. With 262K contexts over TCP (~400 MB/s), GB-scale transfers take seconds — the race window is wide. Crashed ~20 minutes after deployment under real traffic.
An end-to-end runnable snippet is not practical here (needs the P/D router, two engines, and the LMCache server process); the config above plus the traffic pattern is the minimal reproduction.
The asymmetry in current main (d626108)
vllm/distributed/kv_transfer/kv_connector/v1/multi_connector.py, get_finished():
# Aggregate finished recving request ids.
finished_recving.update(recving or ()) # <-- bare union, no dedup
# Aggregate finished sending request ids - only include
# once we've drained the "extra" count (for cases where
# more than one connector is async-saving the same request).
for req_id in sending or ():
extra_pending = self._extra_async_saves.get(req_id) # <-- dedup for savesThe sending-path comment shows the multi-connector-same-request case was considered for saves; the symmetric load case was not.
Failure chain:
- Request R arrives at decode. The router two-phase flow means prefill unconditionally pushes R's prompt KV via NIXL (producer-initiated). If R's prefix also hits in decode-side LMCache (earlier turn),
start_load_kv — broadcast by MultiConnector to all sub-connectors — starts a second parallel async load.
- LMCache (CPU memory) load completes first → R leaves
WAITING_FOR_REMOTE_KVS → generates → finishes → removed from self.requests.
- NIXL pull of a 100K+ token context completes seconds later → NixlConnector reports R in
finished_recving.
assert req_id in self.requests → engine dies.
Aggravating factor: NixlConnector d-node returns False from request_finished() for in-flight pulls (pull_scheduler.py ~L223, if is_d_node and not self.is_bidirectional_kv_xfer_enabled: return False, None), so R's blocks are already freed when the orphaned pull completes — the UCX write can land in freed/reallocated memory. The assert crash is arguably the lucky outcome; silent KV corruption is the unlucky one.
Observed result (full traceback)
Expected behavior: a late load-completion report for an already-finished request is ignored (or the two loads are coordinated so completion is reported exactly once).
Observed (decode engine log):
(EngineCore pid=2047449) ERROR 08-19 16:30:41 [core.py:1351] EngineCore encountered a fatal error.
(EngineCore pid=2047449) ERROR 08-19 16:30:41 [core.py:1351] Traceback (most recent call last):
(EngineCore pid=2047449) ERROR 08-19 16:30:41 [core.py:1351] File ".../vllm/v1/engine/core.py", line 1342, in run_engine_core
(EngineCore pid=2047449) ERROR 08-19 16:30:41 [core.py:1351] engine_core.run_busy_loop()
(EngineCore pid=2047449) ERROR 08-19 16:30:41 [core.py:1351] File ".../vllm/v1/engine/core.py", line 1386, in run_busy_loop
(EngineCore pid=2047449) ERROR 08-19 16:30:41 [core.py:1351] self._process_engine_step()
(EngineCore pid=2047449) ERROR 08-19 16:30:41 [core.py:1351] File ".../vllm/v1/engine/core.py", line 1439, in _process_engine_step
(EngineCore pid=2047449) ERROR 08-19 16:30:41 [core.py:1351] outputs, model_executed = self.step_fn()
(EngineCore pid=2047449) ERROR 08-19 16:30:41 [core.py:1351] File ".../vllm/v1/core/sched/scheduler.py", line 1986, in update_from_output
(EngineCore pid=2047449) ERROR 08-19 16:30:41 [core.py:1351] self._update_from_kv_xfer_finished(kv_connector_output)
(EngineCore pid=2047449) ERROR 08-19 16:30:41 [core.py:1351] File ".../vllm/v1/core/sched/scheduler.py", line 2741, in _update_from_kv_xfer_finished
(EngineCore pid=2047449) ERROR 08-19 16:30:41 [core.py:1351] assert req_id in self.requests
(EngineCore pid=2047449) ERROR 08-19 16:30:41 [core.py:1351] AssertionError
Engine-death log tail additionally shows in-flight UCX get(multi) transfers not flushed at shutdown, consistent with the late-pull explanation.
Why existing tests likely don't catch it
The documented LMCache P/D example topology (docs.lmcache.ai, "disaggregated prefill") is TP=1 / 16K context on one host: NIXL transfers complete in milliseconds, so the window between "LMCache load done" and "NIXL pull done" is effectively zero. At 100K-262K contexts with GB-scale KV over TCP, the window is seconds wide and triggers within ~20 minutes of real agent traffic.
Suggested fix directions
- MultiConnector: mirror
_extra_async_saves for loads — track which sub-connectors have an in-flight load per request and emit finished_recving only when the last one completes (scheduler-side _requests_to_connector already holds the assignment; worker side needs it, e.g. via metadata as done for saves).
- NixlConnector: when a d-node request with an in-flight pull finishes, pin blocks (
return True) and report the late completion as finished_sending — the _reqs_need_send TTL machinery already exists for the bidirectional path.
- Scheduler (defense in depth): warn-and-skip instead of assert for req_ids no longer in
self.requests (state-wise a late report is harmless; memory safety must be fixed at the connector level per 1/2).
Current production workaround: LMCache on the prefill side only (the double-path there is async-save, already deduped) — stable 18+ hours under real traffic with the same model and load.
Happy to turn 1/3 into a PR if the direction sounds right.
Before submitting a new issue...
Your current environment
The output of
python collect_env.pyNote: the vLLM build is a custom sm89 backport of v0.27.1 (wtdcode/vllm-backport v0.5.2); the three files central to this report —
scheduler.py,multi_connector.py,nixl/pull_scheduler.py— were byte-compared against current main (d626108b18) and are identical in the relevant sections.🐛 Describe the bug
MultiConnectordeduplicates async saves across sub-connectors (_extra_async_saves) but performs a bare union for async loads (finished_recving). In a P/D disaggregation setup where the decode engine also runs a second connector doing async loads (LMCache MP), a request can complete via the faster load path while the slower NIXL pull is still in flight; the late NIXL completion is then reported for a request the scheduler already removed, tripping:→
EngineDeadError, decode engine dies, service outage until manual restart.Minimal reproduction
Exact decode-engine connector config (relevant flag only; rest of
vllm serveis stock — TP4, fp8_ds_mla, 262K context):Prefill engine is symmetric (
kv_producerroles). LMCache server is stock upstream:lmcache server --port 6556 --l1-size-gb 48 --eviction-policy LRU --chunk-size 256.Traffic needed to trigger: multi-turn agent-style requests with long contexts (~100K tokens) where the decode-side LMCache has a prefix hit from an earlier turn while the router-initiated NIXL push of the same prompt KV is still in flight. With 262K contexts over TCP (~400 MB/s), GB-scale transfers take seconds — the race window is wide. Crashed ~20 minutes after deployment under real traffic.
An end-to-end runnable snippet is not practical here (needs the P/D router, two engines, and the LMCache server process); the config above plus the traffic pattern is the minimal reproduction.
The asymmetry in current main (d626108)
vllm/distributed/kv_transfer/kv_connector/v1/multi_connector.py,get_finished():The sending-path comment shows the multi-connector-same-request case was considered for saves; the symmetric load case was not.
Failure chain:
start_load_kv— broadcast by MultiConnector to all sub-connectors — starts a second parallel async load.WAITING_FOR_REMOTE_KVS→ generates → finishes → removed fromself.requests.finished_recving.assert req_id in self.requests→ engine dies.Aggravating factor:
NixlConnectord-node returnsFalsefromrequest_finished()for in-flight pulls (pull_scheduler.py~L223,if is_d_node and not self.is_bidirectional_kv_xfer_enabled: return False, None), so R's blocks are already freed when the orphaned pull completes — the UCX write can land in freed/reallocated memory. The assert crash is arguably the lucky outcome; silent KV corruption is the unlucky one.Observed result (full traceback)
Expected behavior: a late load-completion report for an already-finished request is ignored (or the two loads are coordinated so completion is reported exactly once).
Observed (decode engine log):
Engine-death log tail additionally shows in-flight UCX
get(multi)transfers not flushed at shutdown, consistent with the late-pull explanation.Why existing tests likely don't catch it
The documented LMCache P/D example topology (docs.lmcache.ai, "disaggregated prefill") is TP=1 / 16K context on one host: NIXL transfers complete in milliseconds, so the window between "LMCache load done" and "NIXL pull done" is effectively zero. At 100K-262K contexts with GB-scale KV over TCP, the window is seconds wide and triggers within ~20 minutes of real agent traffic.
Suggested fix directions
_extra_async_savesfor loads — track which sub-connectors have an in-flight load per request and emitfinished_recvingonly when the last one completes (scheduler-side_requests_to_connectoralready holds the assignment; worker side needs it, e.g. via metadata as done for saves).return True) and report the late completion asfinished_sending— the_reqs_need_sendTTL machinery already exists for the bidirectional path.self.requests(state-wise a late report is harmless; memory safety must be fixed at the connector level per 1/2).Current production workaround: LMCache on the prefill side only (the double-path there is async-save, already deduped) — stable 18+ hours under real traffic with the same model and load.
Happy to turn 1/3 into a PR if the direction sounds right.
Before submitting a new issue...