Add DeepSeek V4 Q8 fast-prefill profile

[Whamp]

Why this PR exists

This PR adds a separate incubating engine that delivers roughly 3× or greater DeepSeek V4 prefill throughput than the stock b10200 Q8 baseline, depending on context depth. The 3× claim applies to prefill only; measured output speed is about 33–38 decode tokens/s.

The speedup requires compiled CUDA/C++ changes that club-3090 cannot apply as runtime patches to stock llama.cpp. The separate engine keeps those changes isolated and leaves existing llama.cpp profiles and defaults unchanged.

Related experiment

Whamp/vllm#1 separately incubates a vLLM path for the public DeepSeek V4 WNA16 safetensors artifact on four RTX 3090 GPUs. It shares the model and hardware target, but not the engine, artifact format, quantization loader, kernels, or delivery path. This llama.cpp PR remains the Antirez GGUF fast-prefill experiment; neither replaces the other.

What this adds

• Adds the incubating catalog slug llamacpp/deepseek-flash-multi4-antirez-iq2-fast-prefill for 4× homogeneous RTX 3090-class cards.
• Registers llama-cpp-ds4-longctx as a separately named external engine. It does not replace llama-cpp-local.
• Pins the public image by OCI digest:
  ghcr.io/whamp/llama-cpp-ds4-longctx@sha256:a96bd947d63eb81d8baf9f6f5ecb26669476383976717237450fbb5727b03745.
• Adds the Antirez IQ2_XXS weights entry and the DSML chat-template override.
• Keeps the public API port closed until the mandatory full-context graph warm-up finishes. switch.sh receives a 2,400-second readiness timeout for this slug.
• Records the Q8 repair, failed approaches, measurements, caveats, and retirement trigger in models/deepseek-v4-flash-0731/INTERNALS.md and docs/UPSTREAM.md.

Engine provenance

The engine combines stable graph shapes, a resident MoE scheduler, fused kernels, and repaired Q8 cache operations. It is separately named and digest-pinned so the dependency and eventual retirement path remain explicit.

Source provenance:

• base: alesha-pro/llama.cpp@b001c8cd7
• Q8 repair: Whamp/llama.cpp@0379cf4bf
• public image build: Actions run 31322101366

The Q8 repair fixes packed concat on CUDA and CPU, keeps Q8 padding conversion on CUDA, preserves cache type in long-context gathers, and adds 45 passing Q8 concat regression cases.

Measured result

Validated on 4× RTX 3090, PCIe only, with the rig's existing 230 W/card limits unchanged:

• 430,080 tokens reserved
• startup warm-up: 429,568 input tokens at 276.28 prefill tokens/s; ready after 1,628 seconds
• deep fresh-prompt prefill: 913 prefill tokens/s with exact needle recall at 395,282 tokens
• 200K profile prefill: 1,538 prefill tokens/s at 98K depth and 1,341 prefill tokens/s at 195K depth
• output generation: about 33–38 decode tokens/s
• agent loop: about 40× context growth produced 4.2× time-to-first-token growth
• minimum observed free VRAM: 794 MiB
• medium quality: 67/75, equal to the stock b10200 Q8 baseline

Quality regression check

The full 8-pack is used here to check whether the Q8 repair and serving changes materially damage model behavior. It is not presented as an intelligence ceiling. Several misses overlap open benchlocal-cli benchmark-definition issues in DataExtract, StructOutput, and ReasonMath.

Mode / reasoning effort	Fork profile	Stock b10200, same Antirez weights and Q8 cache
Thinking off	109/150	111/150
Thinking on, low	121/150	122/150
Thinking on, high	121/150	not run
Thinking on, max	123/150*	not run

low used a 16,384-token output cap; high and max used 65,536. Neither low nor high reached its cap. ^* Four of 226 max API responses reached 65,536 tokens and ended with finish_reason=length.

The stock comparisons are within one to two scenarios of the fork, which is the relevant evidence for this PR: the quantized cache repair and fast-prefill serving path preserve the tested behavior.

Deliberate caveats

This remains 🐣 Incubating, hidden from the default list, requires --force, and has no DEFAULTS row.

• Exact validated weights: Antirez IQ2_XXS only.
• Exact validated hardware: homogeneous sm_86 4-card group.
• Matching q8_0 K and V required.
• LLAMA_ATTN_ROT_DISABLE=1 remains required; rotation-enabled Q8 is not validated.
• Every restart pays about 26 minutes of warm-up at 430K.
• The 794 MiB floor is below the repo's normal 1,024 MiB production guard, though above the experiment's accepted 750 MiB floor. CTX_SIZE=200000 is the higher-margin fallback.

Validation

Public artifact and exact catalog path:

• anonymous GHCR manifest read: pass
• pulled digest labels source revision 0379cf4bf889f3d28038a005210c4bc193fc8ba1: pass
• diagnose-profile.sh llamacpp/deepseek-flash-multi4-antirez-iq2-fast-prefill: green
• switch.sh --force llamacpp/deepseek-flash-multi4-antirez-iq2-fast-prefill: pass
  • port 8033 stayed closed during warm-up
  • exact digest launched
  • Q8 K/V confirmed in runtime logs
  • warm-up completed at 276.28 prefill tokens/s
• verify-full.sh: 9/9

Prior source-equivalent gate on the clean Q8 image:

• verify-stress.sh: 8/8 at 200K
• 430K fast stress ceiling: exact recall at 263K and 395,282 tokens
• quality-test.sh --medium: 67/75
• full thinking-off quality: 109/150; the earlier 95/150 figure included an invalid network-blocked HermesAgent 0/20 leg
• full thinking-on quality: low 121/150, high 121/150, max 123/150^*; cap details above
• soak-test.sh: 25/25, zero errors and zero silent-empty responses
• bench-agentic.sh: all 15 turns through 54K

Catalog/test gates:

• full scripts/tests/*.sh: 98/100 pass
• the two failures reproduce unchanged on base commit a6385ece:
  • test-bench-capture.sh: host-specific STREAM calibration worker-count assertion
  • test-switch-explain.sh: the 4-GPU host resolves vllm/default as multi4, where no vLLM default exists; the same test passes when the fixture is constrained to two GPUs
• targeted launcher, registry JSON, image drift, patch attribution, profile compatibility, compose mounts, readiness, and parity tests: pass
• docker compose config: pass
• CodeGraph staged cycle/signature/boundary checks: pass

aislop, ruff, ty, shellcheck, and the cockpit pytest venv were unavailable on this host. No JavaScript or TypeScript changed.

[henrykrinkle01]

full no-thinking quality: 95/150

Why so low 🤔

[Whamp]

Sorry those are inaccurate figures on quality. They include an invalid 0/20 HermesAgent leg caused
by Docker being unable to reach the endpoint. The
valid rerun scored 14/20, so the comparable
no-thinking aggregate is 109/150, not 95/150. I
also have the earlier paired full evaluation on
the same Antirez IQ2_XXS weights and Q8 cache:
111/150 no-thinking and 122/150 thinking-on on
stock b10200. I’m separating candidate results
from same-weights baseline results so the PR does
not imply the thinking-on run used the fork
image.

Rerunning the full quality suite now to get cleaner numbers but I've also run this on human eval plus and on gpqa and the quality of the quant is almost indistinguishable from the API and all my testing. The biggest consequence is about 25% more tokens generated per task than via the API, so it's token efficiency. That is the main thing that suffers from the quantization of these, not the quality of the intelligence as far as I can tell.

[noonghunna]

Review complete and approved — verification is green across the board; merge is ready the moment you flip the draft to ready (the draft marker is yours, so I'm not overriding it — if it's guarding something still in flight, say the word).

Reference-rig verification: full catalog sweep 100 pass / 0 fail (your readiness test included), diagnose-profile GREEN, the pinned digest resolves anonymously, and the readiness-gate test passes standalone.

Review notes worth recording:

• The launcher changes are the right shape — extending export_variant_engine_pin to llamacpp/* through the profile resolver keeps the 'profiles hold policy, launchers resolve' division intact, and READY_TIMEOUT as a resolver-driven export is a general mechanism, not a slug special-case. The registry-emit _current_pin generalization to *_IMAGE exports also fixes a latent fragility the old first-value read had — good catch.
• The warm-up port-gating via the TCP forwarder is a clean solution to a real problem (a 26-minute warm-up window where READY would lie), and it's honest about its own cost in the compose header.
• Topology honesty: we can't boot 4-card here, so the serving claims rest on your validated gate runs + the digest-pinned provenance chain — the same footing the multi4-q8 slug landed on, and the incubating status prices that correctly.

One pointer that may interest you: the fork's resident-MoE scheduling is conceptually adjacent to the MoE expert-cache RFC at ggml-org/llama.cpp#24528 — we posted DeepSeek measurements there today (cache +46% on decode when it engages). Your 4-card long-context workload is close to the ideal profile for that line of work if it lands upstream; the two approaches (prefill-side graphs, decode-side cache) look composable on paper.

Five for five once this lands. Thank you for another contribution that arrived pre-validated.

[Whamp]

The reasoning-effort runs are complete, and the PR body plus source docs are updated in 79aeb2dc.

• thinking off: 109/150 vs stock b10200 111/150
• thinking on, low: 121/150 vs stock b10200 122/150
• thinking on, high: 121/150
• thinking on, max: 123/150*

low used a 16,384-token output cap; high and max used 65,536. Neither low nor high reached its cap. * Four of 226 max API responses reached 65,536 tokens and ended with finish_reason=length.

The PR now frames these scores as the serving/quantization regression check they support, not as an intelligence ceiling. It also links the open benchlocal-cli benchmark-definition issues that affect several failed cases. The catalog JSON now exposes low, high, and max as profile capabilities with low as the default.

[Whamp]

I also tested this profile on DeepSWE, which uses repository-level coding tasks.

All runs used Pi 0.84.0 on the same 12 tasks:

Run	Reasoning	Full solves	Feature tests passed	Existing tests preserved
DeepSeek API	max	5.3/12 average	89.9%	99.4%
Local IQ2_XXS	low	5/12	85.5%	99.5%
Local IQ2_XXS	max	6/12	89.1%	100%

The API ran three passes and solved 6, 5, and 5 tasks. The local server ran once at each reasoning level, so this does not prove parity. Still, local max landed inside the API's observed range, while local low finished close behind.

The main difference was speed and token use:

• API max: 2.14M non-cached tokens and 23 minutes per task
• Local low: 2.55M tokens and 61 minutes per task
• Local max: 3.21M tokens and 99 minutes per task

Local max used 1.5× as many tokens and took 4.3× longer per task. One task timed out after three hours despite reaching a 0.986 partial score and passing 92% of its feature tests. I count it as unsolved, but the timeout likely reduced the final score.

My read: the IQ2_XXS model retained most of the API model's coding ability, even at low reasoning. The tradeoff is speed and token use, not a clear loss of quality.

[noonghunna]

Hi @Whamp

apologies for i've been sleeping on PRs for a while for being occupied with loads things. I'll get the ball rolling in this week.

[Whamp]

No worries at all. I'm buried too. I can't remember if I'm even done with this one.

Also I just shipped this which is a gguf DSV4F gguf on vLLM. You get speed in exchange for context length.

https://x.com/hampsonw/status/2089863035561324776