[Bug]: MooncakeStoreConnector stores invalid recurrent states with mamba_cache_mode=align, causing silent output corruption

[myliu569]

Your current environment

The issue was reproduced on vLLM main at:

commit: 0ad04cff1b10267a2642becbf31f62c91018cad5
runtime vLLM version: 0.27.2rc1.dev142+g0ad04cff1
Python: 3.12.3
PyTorch: 2.13.0+cu129
CUDA runtime: 12.9
GPU: 8 x NVIDIA GeForce RTX 3090 (the reproduction uses TP=2)
NVIDIA driver: 580.173.02
mooncake-transfer-engine: 0.3.12.post1
transformers: 5.15.0
triton: 3.7.1
OS: Ubuntu 24.04.2 LTS
GCC: 13.3.0

🐛 Describe the bug

Summary

MooncakeStoreConnector can publish external prefix-cache keys for recurrent-state boundaries that do not contain a valid state.

With MambaSpec(mamba_cache_mode="align"), vLLM may intentionally leave some logical state boundaries without a materialized recurrent-state block. Such entries use NULL_BLOCK_ID = 0. The normal Mooncake Store save path currently resolves that block ID to an address and publishes the corresponding Store key without rejecting the null block.

As a result:

1. A producer publishes a key for a boundary such as token 2112 or 2960 even though no valid GDN/Mamba/ShortConv state was captured there.
2. Mooncake Store lookup sees that the key exists and reports the longer prefix as externally reusable.
3. The consumer loads the bytes successfully and skips recomputation up to that boundary.
4. Generation silently diverges from a cold baseline. There is no Put/Get error, and in some cases the generated tokens remain the same while logprobs differ significantly.

Byte-level instrumentation shows that Mooncake transfers the selected bytes exactly. The invalid all-zero bytes already exist on the producer side because the selected recurrent-state block is NULL_BLOCK_ID. This points to invalid key publication in the producer save path, rather than data corruption in Mooncake transport.

Why this is user-visible

This can affect normal serving whenever all of the following are true:

• a model has recurrent state, for example GDN, Mamba1, Mamba2, or ShortConv;
• MooncakeStoreConnector external prefix caching is enabled;
• a long producer request creates Store keys for an intermediate boundary that did not receive a valid recurrent-state snapshot under the producer's prefill/chunking history;
• a later request reuses that boundary.

The request succeeds with HTTP 200 and Mooncake reports successful Put/Get operations, but the model output can be wrong.

Generic reproduction for recurrent-state models

The same producer/consumer pattern was validated with GDN, Mamba1, Mamba2, and ShortConv caches, on both hybrid and pure recurrent-state models.

The smallest tested profile is state-spaces/mamba-130m-hf. The model-specific values used in the successful reproductions are:

Profile	MODEL	State type	Producer repeats	Consumer repeats	Common prefix	Invalid Store hit
Smallest	state-spaces/mamba-130m-hf	pure Mamba1	1000	1000	3026	3024
Pure Mamba2	mistralai/Mamba-Codestral-7B-v0.1	pure Mamba2	1000	1000	3031	3024
ShortConv hybrid	LiquidAI/LFM2.5-350M	ShortConv + Full Attention	1000	1000	3027	3024
Mamba2 hybrid	Zyphra/Zamba2-1.2B-instruct	Mamba2 + Full Attention	1000	1000	3030	2960
GDN hybrid	Qwen/Qwen3.5-9B	GDN + Full Attention	1250	700	2126	2112

Reproduction commands

All tested runs used TP=2. The commands below are parameterized so any row can be selected. For example, the smallest profile is:

For mistralai/Mamba-Codestral-7B-v0.1, also pass --load-format safetensors. Other model-specific loading flags can be added without changing the producer/consumer sequence.

export MODEL=state-spaces/mamba-130m-hf
export SERVED_MODEL=mamba-130m-hf
export VERIFICATION_CODE=PURE-MAMBA-STORE
export PRODUCER_REPEATS=1000
export CONSUMER_REPEATS=1000
export PYTHONHASHSEED=0

1. Start Mooncake Store

mooncake_master \
  --port 50061 \
  --default_kv_lease_ttl 30m

Save the following configuration as /tmp/mooncake-store.json:

{
  "mode": "embedded",
  "metadata_server": "P2PHANDSHAKE",
  "master_server_address": "127.0.0.1:50061",
  "global_segment_size": "4GB",
  "local_buffer_size": "1GB",
  "protocol": "tcp",
  "device_name": "",
  "enable_offload": false
}

export MOONCAKE_CONFIG_PATH=/tmp/mooncake-store.json

2. Generate a shared-prefix producer/consumer request pair

python3 - <<'PY'
import json
import os

from transformers import AutoTokenizer

model = os.environ["MODEL"]
served_model = os.environ["SERVED_MODEL"]
code = os.environ["VERIFICATION_CODE"]
producer_repeats = int(os.environ["PRODUCER_REPEATS"])
consumer_repeats = int(os.environ["CONSUMER_REPEATS"])

shared = (
    "Cache correctness record. The exact verification code is "
    f"{code}. Remember it for a later question. "
)
producer_prompt = (
    shared
    + "neutral cache context " * producer_repeats
    + "Producer-only task: write the cache and answer ACK:"
)
consumer_prompt = (
    shared
    + "neutral cache context " * consumer_repeats
    + "Consumer-only question: What is the exact verification code? "
      "Answer only with the code:"
)
common = {
    "model": served_model,
    "max_tokens": 16,
    "temperature": 0,
    "seed": 0,
    "logprobs": 1,
}
for path, prompt in (
    ("/tmp/producer-request.json", producer_prompt),
    ("/tmp/consumer-request.json", consumer_prompt),
):
    with open(path, "w", encoding="utf-8") as output:
        json.dump({**common, "prompt": prompt}, output)

tokenizer = AutoTokenizer.from_pretrained(model, trust_remote_code=True)
producer_ids = tokenizer.encode(producer_prompt, add_special_tokens=True)
consumer_ids = tokenizer.encode(consumer_prompt, add_special_tokens=True)
common_prefix = 0
for producer_id, consumer_id in zip(producer_ids, consumer_ids):
    if producer_id != consumer_id:
        break
    common_prefix += 1
print({
    "producer_tokens": len(producer_ids),
    "consumer_tokens": len(consumer_ids),
    "common_prefix_tokens": common_prefix,
})
PY

The important shape is that the common prefix reaches an intermediate Store boundary for which the producer did not retain a recurrent-state snapshot. The exact boundary depends on the model's state-cache page size and tokenizer, so assert the actual external hit metric rather than relying only on prompt character counts.

3. Record a cold baseline

Start vLLM without a KV connector:

CUDA_VISIBLE_DEVICES=2,3 vllm serve "$MODEL" \
  --served-model-name "$SERVED_MODEL" \
  --port 8002 \
  --tensor-parallel-size 2 \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.85 \
  --enforce-eager \
  --enable-prefix-caching \
  --no-disable-hybrid-kv-cache-manager \
  --mamba-cache-mode align

curl -sS -H 'Content-Type: application/json' \
  --data-binary @/tmp/consumer-request.json \
  http://127.0.0.1:8002/v1/completions > /tmp/cold-response.json

Stop the baseline server so that the Store consumer can use the same GPUs.

4. Populate Mooncake Store from a producer

CUDA_VISIBLE_DEVICES=0,1 vllm serve "$MODEL" \
  --served-model-name "$SERVED_MODEL" \
  --port 8000 \
  --tensor-parallel-size 2 \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.85 \
  --enforce-eager \
  --enable-prefix-caching \
  --no-disable-hybrid-kv-cache-manager \
  --mamba-cache-mode align \
  --kv-transfer-config '{
    "kv_connector":"MooncakeStoreConnector",
    "kv_role":"kv_producer",
    "kv_connector_extra_config":{
      "cache_prefix":"state-boundary-repro",
      "enable_lookup":false
    }
  }'

curl -sS -H 'Content-Type: application/json' \
  --data-binary @/tmp/producer-request.json \
  http://127.0.0.1:8000/v1/completions > /tmp/producer-response.json

Wait until the Put succeeds and becomes visible to Store lookup.

5. Restore the prefix in a separate consumer

CUDA_VISIBLE_DEVICES=2,3 vllm serve "$MODEL" \
  --served-model-name "$SERVED_MODEL" \
  --port 8002 \
  --tensor-parallel-size 2 \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.85 \
  --enforce-eager \
  --enable-prefix-caching \
  --no-disable-hybrid-kv-cache-manager \
  --mamba-cache-mode align \
  --kv-transfer-config '{
    "kv_connector":"MooncakeStoreConnector",
    "kv_role":"kv_consumer",
    "kv_connector_extra_config":{
      "cache_prefix":"state-boundary-repro",
      "enable_lookup":true,
      "load_async":true
    }
  }'

curl -sS -H 'Content-Type: application/json' \
  --data-binary @/tmp/consumer-request.json \
  http://127.0.0.1:8002/v1/completions > /tmp/store-response.json

Verify all three conditions:

1. Store metrics show a non-zero external prefix hit and successful Get operations.
2. The Store response differs from the cold response in generated token strings and/or per-token logprobs.
3. Repeating the experiment with --max-num-batched-tokens set to the invalid Store-hit boundary makes that boundary a real state-capture boundary; the restored output then exactly matches the cold output. This third step is a diagnostic control, not a general workaround.

Observed result (Qwen3.5-9B example)

Mooncake Store reports successful operations:

producer: Put 56 keys, 484442112 bytes, 0 failed keys
consumer: Get 14 keys, 121110528 bytes, 0 failed keys
consumer external prefix cache hit: 2112 tokens

However, deterministic output differs from the cold baseline:

cold baseline:
 HYBRID-528-GDN.

The exact verification code is

Mooncake Store consumer:
 HYBRID-528-GDN

The exact verification code is HY

maximum absolute per-token logprob difference: 1.4653081893920898

The mismatch is deterministic in repeated runs.

Expected result

An external prefix-cache key should only be published when every state required to resume from that boundary is valid and recoverable.

If the recurrent-state block at a candidate boundary is NULL_BLOCK_ID, the producer should not publish that boundary as reusable. Lookup should return the longest earlier boundary that is valid for all required cache groups, and the consumer should recompute the remaining tokens.

For deterministic greedy decoding, a cold run and a correctly restored Store run should produce the same text, generated tokens, and per-token logprobs.

Root-cause evidence

Instrumentation immediately before Mooncake Put and immediately after Mooncake Get at the 2112-token boundary shows:

Qwen3.5 GDN recurrent states:
  producer states using physical block 0: 6 / 6
  producer states containing only zero bytes: 6 / 6
  producer -> consumer byte-exact matches: 6 / 6
  SHA-256 for every zero state:
    6334891ad0b99640af1d1dd5dc8863447ff79bb5b3d9936f8bcfc98a5c41922d

Qwen3.5 Full Attention KV entries:
  producer entries containing non-zero bytes: 2 / 2
  producer -> consumer byte-exact matches: 2 / 2

Therefore:

• the recurrent-state entries are already invalid before transport;
• the Full Attention KV entries are valid;
• Mooncake transfers all selected entries byte-for-byte;
• the failure is caused by publishing a key whose recurrent state resolves to the null block.

As a diagnostic control, adding --max-num-batched-tokens 2112 to all three servers changes the producer's state-capture boundary:

GDN recurrent states at 2112:
  physical blocks: non-null blocks 1, 2, and 3
  non-zero states: 6 / 6
  producer -> consumer byte-exact matches: 6 / 6

Full Attention KV:
  producer -> consumer byte-exact matches: 2 / 2

output comparison:
  text: exact match
  generated tokens: exact match
  maximum absolute per-token logprob difference: 0.0

This is a diagnostic control, not a general production workaround: arbitrary request lengths can expose other intermediate boundaries.

Cross-model evidence

The same default-vs-aligned experiment was repeated on five full models covering four recurrent-state implementations. bad boundary is a boundary advertised by Store lookup under the default scheduler configuration. null state is measured on the producer before Put.

Model	Cache architecture	Page size	Bad boundary	Producer recurrent-state evidence	Mooncake transfer	User-visible result	Aligned control
Qwen3.5-9B	GDN + Full Attention	528	2112	6/6 GDN states use null block and are all zero	8/8 total entries byte-exact	Tokens differ; max logprob diff 1.465308	Exact at 2112
Zyphra/Zamba2-1.2B-instruct	Mamba2 + Full Attention	80	2960	14/14 Mamba2 states use null block and are all zero	16/16 total entries byte-exact	Tokens differ; max logprob diff 2.302462	Exact at 2960
LiquidAI/LFM2.5-350M	ShortConv + Full Attention	16	3024	4/4 ShortConv states use null block and are all zero	6/6 total entries byte-exact	Tokens happen to match; max logprob diff 0.150565	Exact at 3024
state-spaces/mamba-130m-hf	Pure Mamba1	16	3024	2/2 Mamba1 states use null block and are all zero	2/2 entries byte-exact	Tokens differ; max logprob diff 2.436541	Exact at 3024
mistralai/Mamba-Codestral-7B-v0.1	Pure Mamba2	16	3024	2/2 Mamba2 states use null block and are all zero	2/2 entries byte-exact	Tokens differ; max logprob diff 2.756008	Exact at 3024

The pure Mamba results show that hybrid Full Attention groups are not required to trigger the bug.

Suspected code path

All links below point to the tested commit.

1. NULL_BLOCK_ID is defined as block 0.
2. MambaManager.remove_skipped_blocks() replaces skipped state blocks with the null block.
3. ChunkedTokenDatabase.prepare_values() resolves logical chunks through block_ids and computes transfer addresses, but does not reject NULL_BLOCK_ID.
4. The normal save path builds keys from processed token ranges, prepares the addresses, and submits the batch Put without filtering null state blocks.
5. In contrast, the partial-tail save path already skips NULL_BLOCK_ID.

The ordinary save path appears to need equivalent validity handling while keeping token ranges, keys, group indices, addresses, and sizes aligned.

Suggested fix direction

One possible minimal fix is to reject a candidate Store boundary when any recurrent-state group resolves to NULL_BLOCK_ID. More generally, the save path should only publish a boundary after verifying that all cache groups required to resume from that boundary are recoverable.

Related issues and PRs

I searched existing vLLM issues and PRs for MooncakeStoreConnector, NULL_BLOCK_ID, recurrent/Mamba state boundaries, skipped blocks, and incorrect-output reports. I did not find an issue or PR that already contains this exact, directly reproduced normal-save-path failure.

The closest existing report is #50630:

• No capability flag declares which KV-cache kinds may be peeked past a candidate boundary — each margin implementation excludes Mamba by hand #50630 describes a broader missing-capability/boundary-invariant problem around recurrent caches. It explicitly lists write-side chunk fragmentation, where a recurrent snapshot is registered under a block hash for a different boundary, as an open class.
• However, No capability flag declares which KV-cache kinds may be peeked past a candidate boundary — each margin implementation excludes Mamba by hand #50630 focuses primarily on EAGLE/MTP cache-peek behavior and states that it had not established whether the write-side path was reachable on main.
• This report directly demonstrates reachability without EAGLE, MTP, speculative decoding, or P/D disaggregation. It isolates the ordinary MooncakeStoreConnector Put path, identifies NULL_BLOCK_ID before Put, proves byte-exact Put/Get, and reproduces silent output divergence on five models.

Other related but distinct work

Issue / PR	Relationship	Difference from this report
#49499	Mooncake Store silent recurrent-state corruption with TP > 1	Fixed TP-shard dedup/keying. The tested commit already contains this fix; here each transferred entry matches its producer bytes, but the producer bytes come from the null block.
#49502	Mooncake Store partial-tail and read-side hit-boundary handling	Addresses reliable partial-tail offload and lookup-side boundaries. This report exercises ordinary full-entry save and invalid key publication on the write side.
#44451 / #44488	Skipped/null Mamba blocks make KV-event token/hash mapping ambiguous	Concerns BlockStored event metadata. This report concerns actual Mooncake Store keys and state payloads used for inference.
#43559 / #51113 / #51351	Recurrent-cache poisoning and capability handling with EAGLE/MTP	#51113 fixes mid-block state published as an aligned state under EAGLE/MTP. The tested commit contains it; this reproduction enables neither EAGLE nor MTP and instead finds a null source block in the ordinary Store save path.
#51468	External Mamba-state/local Full Attention hit divergence	Fixes an assertion in partial-tail truncation after local eviction. This report uses ordinary prefix matching and reaches successful inference with silently incorrect state.
#49757	Stale block-table rows during dummy runs	Prevents dummy kernels from overwriting externally loaded recurrent state. This report captures invalid bytes before Put, rather than consumer state overwritten after Get.
#49360	MooncakeStoreConnector hang on a GDN model	A transport/liveness failure under EFA and offload pressure, rather than successful retrieval followed by silent incorrect output.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.