[Feature]: Performance Optimization for Deepseek V4

[yewentao256]

🚀 The feature, motivation and pitch

Performance Optimization for Deepseek V4

Tasks:

• [Perf] Optimize DSv4 prefill chunk planning, 4.0% E2E Throughput Improvement #45061 @yewentao256
• [DSv4 Perf] DSv4 flashinfer sparse index cache for metadata, 2%~4% TTFT improvement #45863 @yewentao256
• [DSv4] Pack KV caches into contiguous per-block allocations for DeepSeek V4 #44577 @tlrmchlsmth
• [Perf] Optimize fused_topk_bias for DSv4, 1.5~2x kernel performance improvement #47463 @yewentao256
• [Perf] Cache token_to_req_indices for dsv4, 5x~6x kernel performance improvement #47474 @yewentao256
• [Perf] Remove redundant repeat and copy for dsv4, 1.8% E2E TPOT improvement. #48137 @yewentao256
• [DSv4] Remove sparse-MLA q-head padding for FlashInfer >=0.6.14 #48047 @majunze2001
• [Perf] Optimize dsv4 routing using specialized kernel, 2.94% E2E TPOT improvement #48660 @yewentao256
• [Bugfix] Complete FlashInfer CUTLASS MXFP4 MoE wiring for DeepSeek-family models #48303 @ormandj
• [Bugfix] Honor DeepSeek V4 checkpoints' MTP-layer compress_ratio entry (unscaled draft rope) #48304 @ormandj
• [DSv4 Perf] Skip empty c128 kernel launch, around 2x kernel performance improvement. #48957 @yewentao256
• [Core][DSV4] Compact MXFP4 indexer KV cache and packed group overlays #48993 @GirasoleY
• [DSv4 Perf] Skip topk and router when not needed, 3.4% E2E TTFT improvement for Decode case #49486 @yewentao256
• [DSv4 Perf] Remove redundant full kernel for dsv4, 1.88x kernel performance improvement #50298 @yewentao256
• [DSv4 Perf] Fix redundant memory allocation and copy for dsv4 pp buffer, 448 MiB GPU memory saved #50312 @yewentao256
• [Perf] Adaptive budget for spec scheduled token, 55%~65% E2E TTFT Improvement #51725 + [Config] Update default _max_num_batched_tokens from 8192 to 16384 #51726 @yewentao256 (tested using kimi k3 but should also work for dsv4)
• [Perf][DSV4] Optimize global top-k index kernel with compile-time constants #51967 @chaunceyjiang
• [Perf][DSV4] Optimize sparse top-k metadata kernels for higher prefill throughput #52084 @chaunceyjiang
• [Feature] Add IndexCache support for DeepSeek V4 on Hopper #49085 @wangyicong52 / [Feature] IndexCache for DeepSeek-V4 (validated on V4-Flash-0731, including DSpark on to accelerate on top of  #51209 @DiegoCao

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

Deepseek V4 doesn't seems to work on single 80G GPU memory even if cpu offload is enabled.

Can it work by enabling that?

[yewentao256]

@vanmilleru Please take a look at https://recipes.vllm.ai/deepseek-ai/DeepSeek-V4-Pro

@yewentao256 I used --trust-remote-code --kv-cache-dtype fp8 --block-size 256 --attention_config.use_fp4_indexer_cache=True --tokenizer-mode deepseek_v4 --tool-call-parser deepseek_v4 --enable-auto-tool-choice --reasoning-parser deepseek_v4 --cpu-offload-gb 100".

It still runs out of GPU memory even with eager mode enabled. I wonder if the offload is actually working or not.

[yewentao256]

@vanmilleru I meant usually we use at least 8 cards even for H200 (141GB memory). Even if using cpu-offloading, it may not work for your 80GB GPU.

Can i run DeepSeek V4-Flash with 2*80G using one card per node?

[yewentao256]

I don't think it works as well, you might want to get more gpu resources first.

I thought V4-Flash in fp4 takes ~142GB GPU mem in total. and 2*80 gives 160GB. it will certainly need some activation and KV cache but can most of the MLP be off loaded to use cpu and cpu mem?

[yewentao256]

Thanks for the clarification.

What you describe is not the primary deployment scenarios we optimize or validate in our recipes. In practice, it may work with offloading, or even with other non-standard workarounds such as trimming parts of the model, but those are not the recommended and can easily run into various stability, compatibility, and performance issues.

For that reason, the most practical recommendation is still to use more GPU resources rather than trying to push this configuration further.