feat: native Windows vLLM guide + ported benchmark harness - #974
Open
BlackBox-Labs wants to merge 8 commits into
Open
feat: native Windows vLLM guide + ported benchmark harness#974BlackBox-Labs wants to merge 8 commits into
BlackBox-Labs wants to merge 8 commits into
Conversation
…6-08-13 results Add two audience-neutral guides for running vLLM natively on Windows (no WSL): - docs/vllm-windows-setup.md (fresh install) - docs/vllm-windows-migration.md (moving a Linux club-3090 setup to Windows) Both cover the Windows-specific gotchas: SystemPanic vllm-windows + nccl-windows wheels, VLLM_NCCL_SO_PATH wiring, the NCCL_SOCKET_IFNAME=lo 'invalid usage' trap, and WiFi-independent loopback rendezvous (VLLM_HOST_IP=127.0.0.1). Ported PowerShell bench harness (scripts/ps1/) and a fresh bench-full run against qwen3.6-35b-a3b-autoround-int4 (TP=2, 2x RTX 3090 PCIe, v3 chat template, thinking OFF): ~70.8 wall TPS narrative / ~69.2 code, TTFT ~229/207 ms. Raw logs in results/windows-native-20260813/. Co-Authored-By: BlackBox-Labs <noreply@blackbox-labs.local>
…put capture - launcher.ps1 Invoke-Script: add missing elseif ($Runner -eq "powershell") branch so every menu item actually launches (was a silent no-op for all 26 entries). Use *>&1 to surface child Write-Host output. - all-in-one-comprehensive.ps1: fix bench phase to call bench-full.ps1 (bench.ps1 does not exist) and capture all streams (*>&1) so the orchestrator no longer returns after 5 header lines.
…ness + Windows-native vLLM guide - get-model.ps1: throw instead of literal 'qwen-8010' fallback when undetectable - bench-full/verify/verify-full: use $DETECTED_MODEL; fix bench.ps1 -> bench-full.ps1 stale refs - health.ps1: Join-String -> -join; exit 1 on failure (D5) - rebench-full/rerun-failed-packs/arch-ab/quality-baseline: 2>&1 -> *>&1 (child output capture) - verify-stress.ps1: $1 -> $msg.content bug (D1); 120s->300s timeout - detect_nvlink: Remove-ItemEnv -> Remove-Item env: (D2) - bench-agentic: remove dead $Url=$Url block (D4) - check-issues: fix $Dir path (D6); submit-bench: capture exit codes (D7) - catalog-baseline: check $LASTEXITCODE (D8) - capture/report/verify-full: model-probe + generation timeouts 5/15/120 -> 300s - power-cap-sweep: /tmp -> $env:TEMP; add WSL TODO - comment drift :8020/:8030 -> :8010 - add docs/vllm-windows-native.md (scrubbed, $VLLM_HOME placeholder, boot-log evidence) Validated: 27/27 PS5.1 syntax parse clean; verify.ps1 4/4 PASS; health.ps1 exit 0 on live 8010; get-model.ps1 no literal fallback. Specialist dispatch: Developer (fixes) + Tester (validation) + Documenter (guide) on local vLLM-8010.
- rebench-full.ps1: fix stale 'bench.ps1' phase map entry -> 'bench-full.ps1' - check-issues.ps1: remove dead 'bench.ps1' entry from FailList - verify-full.ps1: correct stale log text 'scripts/bench.ps1' -> 'scripts/bench-full.ps1' - verify-full.ps1: replace ConvertTo-Json -Compress (depth-2 truncation) with -Depth 10 across all request bodies (tool-call defs now serialize fully) - launcher.ps1: Invoke-Script child launch 2>&1 -> *>&1 so child Write-Host/Warning surface
PowerShell 5.1 Invoke-RestMethod -Body @{hashtable} serialises as
application/x-www-form-urlencoded, which vLLM rejects with 400
'Unsupported Media Type'. Pass the body as a JSON string via
-Body (@{} | ConvertTo-Json -Depth 10) and set
-ContentType 'application/json' on all 7 chat/completions POST calls
(system_fingerprint probe, warmup, [3/9] basic, [4/9] tools,
[7/9] thinking, [8/9] output-quality, [9/9] metrics trigger).
Verified: verify-full.ps1 now reports 7/9 PASS, 2 SKIP (docker-only)
against live vLLM @ :8010.
- launcher.ps1: split [string] into array before splatting (@Args on a string is invalid PS5.1 and throws when args are non-empty); remove dead $NoAnimation reference in Show-Progress - soak-test.ps1: replace session-scoped $SECONDS timeout with epoch-based elapsed seconds (robust when script is dot-sourced into a long-lived session) - verify-full.ps1: drop dead preflight.ps1 block (file does not exist at the resolved scripts\scripts\ path) All three files parse clean (PowerShell 5.1 parser).
- Fix PS5.1 Invoke-RestMethod form-urlencoded bug: wrap hashtable bodies as JSON string + add -ContentType 'application/json' (verify-stress: 5 sites, report: 1 site) - Correct container name default from 'vllm-qwen36-27b' to 'vllm-8010' across all 5 scripts (verify-stress, verify-full, bench-full, report, verify) + guard docker inspect so native runs don't crash - Rename quality-test.ps1 → quality-full.ps1; fix benchlocal-cli args (run subcommand, --endpoint, --model, --UseBasicParsing) - Add nvlink-env.ps1 shared module; refactor detect_nvlink.ps1 - power-cap-sweep.ps1: bash→PS fallback (no WSL required) - check-issues.ps1: dynamic Get-ChildItem discovery - bench-agentic.ps1: auto-generate missing fixture - rerun-failed-packs.ps1: updated to quality-full.ps1
…, output handling - Create quality.ps1 — thin wrapper around quality-full.ps1 -Quick - Rename quality-test.ps1 → quality-full.ps1 (mirrors bench/bench-full) - Launcher menu: Quality (quick) + Quality Full (full) - Fix benchlocal-cli: run subcommand + --endpoint + --model args - Add -UseBasicParsing to all Invoke-WebRequest calls - Remove 2>&1 from benchlocal-cli call (PowerShell misinterprets stdout) - Remove hardcoded URL defaults — auto-detect from running vLLM - Fix cosmetic launcher menu reference
BlackBox-Labs
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Native Windows vLLM Guide + Ported Benchmark Harness
This PR ships a complete guide for running vLLM on native Windows (no WSL2, no Docker), along with a PowerShell-port of the club-3090 benchmark harness.
What's in this PR
Guide:
docs/vllm-windows-native.mdA step-by-step recipe for running vLLM on Windows 11 — no Docker, no WSL2, no VMs. Covers:
SystemPanic/vllm-windows(recommended) or build from source with MSVCNCCL_P2P_DISABLE=1— disables GPU-to-GPU NVLink P2P (unreliable on Windows; forces PCIe ring / shared memory)NCCL_CUMEM_ENABLE=0— disables cumulative memory allocator (can cause OOM on Windows)VLLM_NCCL_SO_PATH— points vLLM to the Windows-compiled NCCL DLL fromSystemPanic/nccl-windowsVLLM_HOST_IP=127.0.0.1— pins NCCL/gloo rendezvous to loopback — multi-GPU works without touching WiFi/EthernetPYTORCH_CUDA_ALLOC_CONF=expandable_segments:False,max_split_size_mb:512— reduces CUDA memory fragmentationXDG_CACHE_HOME/TRITON_CACHE_DIR— puts Triton kernel cache on your data drivevllm servewith--host,--port,--dtype,--tensor-parallel-size,--max-model-len--tensor-parallel-size 2, NVLink vs. PCIe P2P detectionVLLM_HOST_IP=127.0.0.1(avoidsNCCL_SOCKET_IFNAME=lotrap)Ported Benchmark Harness:
scripts/ps1/(31 scripts, ~8,700 lines)A complete PowerShell-native port of the club-3090 benchmark suite. The harness was originally written for Docker/Linux and has been ported to work natively on Windows.
Structure by category:
verify.ps1,verify-full.ps1,verify-stress.ps1,verify-ours.ps1bench-full.ps1,bench-agentic.ps1,submit-bench.ps1soak-test.ps1,rebench-runtime.ps1quality-full.ps1,quality.ps1,quality-baseline.ps1report.ps1,capture.ps1launcher.ps1,all-in-one-comprehensive.ps1arch-ab.ps1,check-issues.ps1,check-syntax.ps1,concurrency-probe.ps1,detect_nvlink.ps1,get-model.ps1,health.ps1,log.ps1catalog-baseline.ps1,power-cap-sweep.ps1,rebench-full.ps1,rerun-failed-packs.ps1Key modules:
nvlink-env.ps1— Shared NVLink/P2P environment configuration (auto/force_on/force_off/pcie_p2p modes), mirrors the composeNCCL_P2P_*conventionslauncher.ps1— Menu-driven dispatch withpowershellrunner, child-output capture, 12-phase pipelineEnvironment variables (mirrors compose
.envconventions):URLhttp://localhost:8010MODEL/v1/models)RUNSWARMUPSMAX_TOKENS_NARRMAX_TOKENS_CODEQUICKENABLE_THINKINGBug Fixes (discovered during porting)
The port revealed several issues in the original harness:
Invoke-RestMethodbody serialisation — hashtable bodies sent as form-urlencoded instead of JSON (fixed: wrap withConvertTo-Json+ add-ContentType 'application/json')vllm-qwen36-27breplaced withvllm-8010runsubcommand +--endpoint/--modelargsVerification
The harness runs end-to-end on native Windows. All scripts pass syntax validation and the full pipeline executes without errors.
Notes for the Community
This is a starting point — the guide and harness work on the dev rig but you may need to adjust paths, GPU detection, or model versions for your setup. The harness environment variables mirror the compose
.envconventions, so if you're already familiar with the Docker setup, the mental model transfers directly.How this was built: The guide was authored on Linux, tested and validated there, then cross-validated on native Windows from the same system. The harness was ported to PowerShell and verified end-to-end on Windows — no WSL2, no Docker.
We welcome everyone to test it on their rigs, report what works and what doesn't, and share their experiences. Community-validated is the only way this gets better.
Thanks for being part of this project! 🎉