feat(model-switch): self-healing control plane — watchdog, rollback, /pull

[titan550]

Summary

tools/model-switch (added in #549) wraps switch.sh for one-shot swaps. An unattended harness needs more: only one model fits in VRAM on a 1–2 GPU rig, much of the catalog is experimental, and a wedged model strands the rig until a human notices. This adds a watchdog that re-launches a crashed/wedged model, rollback to the previously-healthy model on a failed switch, a crash-loop budget that degrades instead of thrashing, POST /pull for missing weights, and a ROUTES-driven GET / manifest so an agent can learn the API in one call. No TPS impact — the serving path, composes and KV config are untouched; this is host-side control plane only. Trade-off: ~1.6k lines of new host code and a background thread, in exchange for the rig staying up unattended. Compared against the existing tools/model-switch on master (the only prior art) and tools/serve-cockpit, which does the same job as a TUI.

Type of change

• New compose variant
• New patch / sidecar
• New script or tool (scripts/, tools/)
• New model
• Doc-only / typo
• Other

Verification

• verify-full.sh PASS — 8 checks pass, Genesis skipped (not in this compose).
• verify-stress.sh — boundary 5/5, recall ladder 3/3, all 6 ceiling rungs to 240,633 tok (91% of n_ctx=262144).
• Tool's own suite — test-model-switch.sh (live HTTP) + test-model-switch-unit.py, green with and without weights on disk.
• test-locale-utf8.sh — 106 scripts (this PR adds control-api.sh, which carries the PYTHONUTF8 export per [bug] half the script layer breaks on non-UTF-8 locales — 42/84 tests fail under LC_ALL=C #779).
• Full report.sh --full — see N/A below.
• Profile header / BENCHMARKS row / CHANGELOG — see N/A below.

verify-full.sh (click)

[1/9] Server reachable on /v1/models ...      ✓ server is serving
[2/9] Genesis patches applied ...             ⊘ no Genesis marker (skipped)
[3/9] Basic completion — capital of France ✓ reply contains 'Paris'
[4/9] Tool calling ...                        ✓ tool_calls[] populated with get_weather
[5/9] Streaming (SSE) ...                     ✓ streamed 15 chunks, 113 chars
[6/9] Streaming tool-calls (thinking-on) ...  ✓ delta.tool_calls + finish_reason=tool_calls, no <tool_call> leak
[7/9] Thinking / reasoning mode ...           ✓ reasoning 636 chars, content 3 chars (finish=stop)
[8/9] Output quality / cascade detection ...  ✓ 9968 chars, variety=0.665, max_line_repeat=0
[9/9] MTP acceptance length threshold ...     ✓ AL = 2.79 (>=2.0 — spec-decode contributing)

All checks passed. Stack is ready for full-functionality use.

verify-stress.sh (click)

[7/8] Long-context needle large rungs (60K / 90K — Cliff 2 territory) ...
    ✓  58569 tokens: recalled 'silver platypus 64'   prefill=1831.8 t/s (32s)
    ✓  91070 tokens: recalled 'crimson capybara 13'  prefill=1907.1 t/s (48s)
[8/8] Context ceiling ladder (~95000 → 0.92 × n_ctx) ...
    ✓ rung 1/6: 94K tok (36%)   prefill=2360.9 t/s  VRAM_free=588MB
    ✓ rung 2/6: 124K tok (47%)  prefill=1822.5 t/s  VRAM_free=588MB
    ✓ rung 3/6: 154K tok (59%)  prefill=1749.8 t/s  VRAM_free=588MB
    ✓ rung 4/6: 184K tok (70%)  prefill=1667.2 t/s  VRAM_free=588MB
    ✓ rung 5/6: 214K tok (81%)  prefill=1504.0 t/s  VRAM_free=588MB
    ✓ rung 6/6: 240K tok (91%)  prefill=1505.2 t/s  VRAM_free=588MB

  boundary checks  5/5 passed
  recall ladder    3/3 passed

⚠️ The run prints 1 stress check(s) failed despite 5/5 and 3/3. That counter is the
VRAM-margin advisory at verify-stress.sh:1515 — 588 MB free at 91% fill, under the
1024 MB threshold — which prints a ⚠ but increments FAILED. It is a property of the
vllm/dual compose that was already running, not a functional failure, and this PR
touches no compose, serving path or KV config. Flagging rather than hiding it.

Note: CONTRIBUTING says "verify-stress 7/7"; since #1018 the script reports split
boundary/recall verdicts instead. The above is the equivalent full pass.

N/A justifications

• report.sh --full (~35 min: soak-continuous + bench) — N/A. That pass exists to validate a serving config. This PR adds a host-side control plane and changes no compose, engine flag, or KV setting, so soak-continuous and bench.sh would measure the unmodified vllm/dual config. verify-full + verify-stress were still run (above) to prove the tool doesn't perturb the running stack.
• Profile header — N/A, no compose added or changed.
• BENCHMARKS row — N/A, no TPS claim. Nothing here can move TPS.
• CHANGELOG entry — N/A. models/<model>/CHANGELOG.md is the per-model gate and no model is touched; consistent with recent scripts/+tools/ changes on master, none of which carry a CHANGELOG entry.

Cross-links

• Closes # — none; no tracking issue exists. Filed as a draft to scope it, per CONTRIBUTING's "open an issue first" — happy to convert to an issue instead if you'd prefer to align on shape before reviewing code.
• Related: Launch should catch a missing weight shard before the engine does #1042 (Launch should catch a missing weight shard before the engine does). This PR's pre-check is adjacent but does not close it. _weights_present() verifies the subdir exists and that some file matches verify_glob, so it catches "weights never downloaded" but not Launch should catch a missing weight shard before the engine does #1042's case. Tested explicitly: with 65 of 66 shards on disk it returns True. A shard-level index.json check is the separate fix Launch should catch a missing weight shard before the engine does #1042 describes.
• Builds on tools: add HTTP model-switch service (thin wrapper over switch.sh) #549 (the original thin wrapper).
• control-api.sh carries the PYTHONUTF8 export required by [bug] half the script layer breaks on non-UTF-8 locales — 42/84 tests fail under LC_ALL=C #779 / fix(scripts): guarantee Python UTF-8 mode + gate it (#779) #782.