add: Qwen3.8-27B llama.cpp single-card configs (IQ4_KS/NL) - #988
Draft
ajmendez wants to merge 13 commits into
Draft
add: Qwen3.8-27B llama.cpp single-card configs (IQ4_KS/NL)#988ajmendez wants to merge 13 commits into
ajmendez wants to merge 13 commits into
Conversation
- llama-cpp/compose/single/iq4ks.yml: text-only, 200K ctx, MTP n=3 - llama-cpp/compose/single/iq4ks-vision.yml: multimodal, 150K ctx, mmproj-BF16 - models/qwen3.8-27b/README.md: model overview, quant options (bartowski + unsloth) - Register both slugs in compose_registry.py (incubating) - Add to supported models table in README Weights: bartowski/Qwen3.8-27B-GGUF (default) or unsloth/Qwen3.8-27B-GGUF (MTP Dynamic V3.0) VRAM: ~21.3 GB text / ~22.7 GB vision on RTX 3090 24GB
|
bartowski/Qwen3.8-27B-GGUF does not have Qwen3.8-27B-IQ4_KS.gguf |
- create-pr.sh: compare URL had base/head reversed; use fork:head...base - compose: pin server-cuda-b9246 (the build this rig validates against; b10236 is newer than the on-rig A/B window and unmeasured here) - README: unsloth example used a nonexistent /models/.../unsloth/ default path; show GGUF_FILE/MMPROJ_FILE override instead
- BENCHMARKS.md: first Qwen3.8-27B single-3090 row (63.2/71.9 wall TPS, n=5, CV 2.7%/1.7%, 2026-08-16) — honest about non-canonical protocol (max_tokens=200 short prompts vs bench.sh 1000/800) and rig-vs-compose drift - FACT: bartowski ships NO IQ4_KS for this arch (4-bit = IQ4_NL 16.3 GB); 'iq4ks' slug names are a quant-label placeholder, GGUF_FILE selects the file - FACT: Q4_K_M (17.8 GB, the bench rig's quant) OOMs at 200K on 24 GB (17.8 + mmproj 0.93 + q4_0 KV 3.7 GB); rig validated at 131K, ~22 GB boot - compose/README/registry/pr.md: corrected sizes, ctx ceilings, status_notes, weights table; create-pr.sh embedded body re-synced from pr.md
… table re-synced - BENCHMARKS row: [X] checked (first Qwen3.8-27B single-3090 row landed 2026-08-16) - Cross-links: self-referential to noonghunna#988, fixes the missing-IQ4_KS flag - Files table: 7 files (was 5), BENCHMARKS.md + create-pr.sh added - create-pr.sh: PR_BODY re-synced from pr.md (was stale at commit 4944652)
- README model table: IQ4_KS/NL -> IQ4_NL/Q4_K_M, 15 -> 16.3 GB - model README: document what's working (bench-validated Q4_K_M @ 131K + vision) vs unvalidated (IQ4_NL @ 200K shipped default) - composes: fix quickstart local-dir to qwen3.8-27b-gguf - compose_registry: vision weights_companions -> gguf_mmproj_f16 - add scripts/lib/profiles/models/qwen3.8-27b.yml catalog entry
# Conflicts: # scripts/lib/profiles/models/qwen3.8-27b.yml
Folded in during the upstream merge (the standalone catalog entry this supersedes was replaced by upstream's qwen3.8-27b.yml, which was re-authored 2026-08-14 in noonghunna#990 with the iq4nl/q8kxl/fp8/autoround-int4/nvfp4 slugs): - unsloth-q4km: Q4_K_M 17.8 GB — the ONLY qwen3.8-27b config with a measured TPS number on this rig (bench 2026-08-17 @ 370 W: 57.83 narr / 68.50 code wall TPS, TTFT 154/150 ms). Not yet a compose_registry slug — upstream's single-card path is unsloth-iq4nl (q8kv). - gguf_mmproj_f16: F16 projector 927,607,008 bytes — the companion the bench-validated rig mounted; enables a future -vision compose for the GGUF tier (no shipped compose mounts a projector yet). - Fixed the now-stale 'TEXT-ONLY' vision comment on the unsloth entries.
…eights refs Replace the non-canonical 2026-08-16 row (63.2/71.9, measured at the card 420W default while annotated 370W, and comparing this rig wall TPS against the reference decode TPS) with a canonical scripts/bench.sh run at a verified 370W cap: 58.05/68.38 wall, 58.57/69.78 decode, TTFT 150/148ms, PP 1265@10K 911@90K. Replicated across a swap-free repeat on the same warm engine (prior run with ~318MB paged out agreed within 1.8% on every metric). Restated the reference delta at matched metric: +17% narrative / +19% code, replacing the earlier +25% figure. Point both new registry entries at upstream unsloth-iq4nl (bartowski-iq4nl does not exist in the merged upstream profile and was failing cross-reference validation), and add the kind:mmproj weights entries the vision compose needs but upstream lacks -- gguf_mmproj_f16 (bartowski, 927,607,008 B, the projector the bench actually mounted) and gguf_mmproj_bf16 (931,145,952 B, the current compose default, unbenched). Record gate results in pr.md: verify-full PASS, verify-stress 8/8 with NIAH recall at 58,569 and 91,070 tok and a ceiling ladder filling to 120,320 tok (91% of n_ctx), soak-continuous 5x5 PASS at 0 MiB growth. Flag an upstream bug: bench.sh swap check reports "every number in this run is suspect" over a 36 kB residual VmSwap while simultaneously reporting swap_total_gib=0.0, i.e. with no swap device mounted.
Records the measured 370W arm (58.05/68.38 wall, ~360W draw at 100% util, derived ~0.163/0.194 tok/J from a spot power sample) and leaves the 420W arm explicitly blank rather than filling it with this rig 2026-08-16 numbers. Those 420W numbers are excluded because they confound three variables against the 370W run: max_tokens=200 short prompts vs canonical 1000/800, a wall-vs-decode metric mismatch in the headline, and a different engine build (server-cuda-b9246). Documents what a valid A/B needs -- same weights/ctx/KV/MTP/build and the same warm engine instance with power.limit as the only variable -- with the exact commands, and points at scripts/power-cap-sweep.sh for the fuller efficiency curve. Notes that the canonical Qwen3.6-27B reference is itself a 370W measurement and that a sibling profile records a -42% swing from 370W to 230W, so power is a first-class variable for cross-row comparison.
Retarget: drop single/iq4ks.yml (upstream master now ships an equivalent at single/unsloth-iq4nl/q8kv.yml, so it was a duplicate under a superseded flat path named iq4ks for an IQ4_NL file), and move the vision compose to upstream layout at single/bartowski-q4km/q4kv-vision.yml. The vision sibling plus a kind:mmproj weights entry is exactly what upstream own status note names as missing. Close the rig-vs-default drift instead of documenting it. Compose defaults now equal the benched config: bartowski Q4_K_M + mmproj-F16, -c 131072, -b 1024 -ub 1024, q4_0 KV, MTP n=2. Was IQ4_NL @ 200K with MTP n=3 while benching something else. Correct the weights provenance, verified by byte size: the rig runs bartowski Qwen3.8-27B-Q4_K_M.gguf (17,772,537,440 B), not unsloth (17,106,775,008 B, 666 MB smaller). Projectors also differ in name and size between the two repos. Earlier revisions credited unsloth throughout. Add the bartowski-q4km weights variant; retarget the registry entry to it with max_ctx=131072; drop the registry entry and DEFAULTS mapping for the removed compose. Replace the two-point 370/420 W comparison with the full power-cap sweep (210-450 W, 10 W steps): efficiency plateaus ~230-310 W peaking near 250-270 W, throughput saturates ~300 W (300->450 W is +5% narrative for +50% power), decode is bandwidth-bound (SM clock 840->1785 MHz while TPS plateaus, mem clock pinned at 9501 MHz), draw saturates ~428 W so caps above ~430 W are inert, and 74 C max means the curve is power-shaped not thermal. Reference box now runs 250 W; the published row remains a labelled 370 W measurement. Fix six dangling references in the model card to the removed compose, including a docker compose -f command that would have failed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Weights: bartowski/Qwen3.8-27B-GGUF (default) or unsloth/Qwen3.8-27B-GGUF (MTP Dynamic V3.0)
VRAM: ~21.3 GB text / ~22.7 GB vision on RTX 3090 24GB
Summary
Add Qwen3.8-27B support for single RTX 3090 via llama.cpp, following the same
pattern as existing models (qwen3.6-27b unsloth-q4km, tess-4-27b, etc.).
Two compose variants: text-only (200K ctx) and multimodal (150K ctx with
mmproj-BF16 vision). Both use bartowski IQ4_KS weights (~14.8 GB) with MTP n=3
speculative decoding. VRAM budget: ~21.3 GB text / ~22.7 GB vision on 24 GB.
Type of change
models/<new-model>/)Verification
# Profile (at-a-glance):blocks with
Status: incubatingand VRAM budget comments.N/A justifications
add
bash scripts/report.sh --fulloutput after first validation run.measured.
Cross-links
Files changed (5 files, +313 lines)
models/qwen3.8-27b/README.mdmodels/qwen3.8-27b/llama-cpp/compose/single/iq4ks.ymlmodels/qwen3.8-27b/llama-cpp/compose/single/iq4ks-vision.ymlscripts/lib/profiles/compose_registry.pyREADME.mdWeights sources
mmproj:
mmproj-Qwen3.8-27B-bf16.gguf(bartowski) ormmproj-BF16.gguf(unsloth)Review
This PR was reviewed by three sub-agents (correctness, simplicity/Andrej,
conventions). All checks passed. See the PR comments for the full review.
Key findings:
registry entries, vision compose all correct.
honest about incubating status, no speculative code.
incubating, noted for follow-up).