Skip to content

VALIDATION_REPRODUCIBILITY.md

Latest commit

 

History

History
137 lines (104 loc) · 4.63 KB

VALIDATION_REPRODUCIBILITY.md

File metadata and controls

137 lines (104 loc) · 4.63 KB
Raw

Validation Reproducibility

ODS's full release-grade validation uses public CI, distro labs, and a private real-hardware fleet. Not every fork can reproduce every lane, but every operator should be able to understand what was tested and run an appropriate subset for their own hardware.

Use this guide with RELEASE_VALIDATION.md, VALIDATION-MATRIX.md, and HIGH_RISK_CHANGE_MAP.md.

Validation Layers

Layer Publicly reproducible Purpose
Static checks Yes Syntax, contracts, docs, manifest health, config drift
Dashboard tests Yes UI/API unit and build confidence
Compose matrix Yes Validates stack file combinations without full hardware
Distro containers Mostly Package-manager and bootstrap behavior across distro families
Incus/systemd VMs Lab-dependent Docker daemon and systemd behavior
Real hardware fleet Hardware-dependent GPU/backend, model, lifecycle, and full-product behavior
Full-model capabilities Hardware/model-dependent Proves agent, search, files, code, model identity, and context after model swap

Minimum Local Audit Set

From ods/:

git diff --check
python scripts/audit-extensions.py --project-dir .
python scripts/validate-generated-configs.py
python scripts/validate-golden-paths.py
python scripts/check-dependency-pins.py

Add dashboard checks when dashboard code or dashboard-api behavior changes.

Reproducing Installer Confidence

For a local fork or appliance candidate:

  1. Start from a clean checkout.
  2. Record the commit.
  3. Run a fresh install on representative hardware.
  4. Confirm service health.
  5. Exercise dashboard model and extension flows.
  6. Confirm Hermes can answer a seed prompt.
  7. Wait for full model download and hot-swap.
  8. Run capability probes or an equivalent manual checklist.
  9. Run idempotent reinstall, ods restart, and ods doctor.
  10. Record skipped surfaces honestly.

If you cannot run the full fleet, say which hardware and distro classes were not tested.

For model-management changes, also name the tier:

  • release tier: six distinct planned test models per reachable host, each running the full discover, download, load, app-use, restore, and delete chain;
  • smoke tier: one planned test model per host running that same full chain.

Smoke evidence is useful during iteration, but it is not a substitute for the release six-model matrix when a change can affect model routing, app probes, or agent viability gates.

Capability Deferrals

Capability checks should not fail just because a large model is still downloading. A valid report distinguishes:

  • bootstrap model active;
  • full model downloading;
  • full model downloaded but not served yet;
  • full model served and capability probes passed;
  • full model served and a probe failed.

Do not mark a release as fully validated until deferred full-model checks have resolved or are explicitly excluded from that release.

Model Probe Deferrals

Enabled LLM apps should be discovered from manifests and known routing config, then probed after each model swap. A valid deferral names the app, the blocker, and the owner or release decision.

Open WebUI needs special care because a login page can look superficially healthy. A release receipt should say which admin/API credential was provisioned for the probe. If the harness only reaches HTTP 401 or an auth wall, record the probe as failed or deferred; do not count it as pass.

Owner-Card And Talk Probes

ODS Talk owner-card probes only gate releases when the owner-card surface is enabled and ods-proxy is healthy. A default non-LAN install should skip those probes rather than failing a surface the user did not expose.

If your fork depends on owner-card access, add a dedicated validation lane for the LAN/proxy path.

Recording A Validation Receipt

Use a concise receipt:

Project ref:
Downstream ref:
Date:
Hardware:
OS/distro:
Install command:
Services enabled:
Model selected:
Gates passed:
Gates skipped/deferred:
Known limitations:
Logs/report path:

Validation receipts are most valuable when they are boring, specific, and easy to compare with the next run.

How To Read Upstream Claims

Upstream validation proves the upstream candidate on the tested hardware and software surface. It is a strong signal for forks, but it does not automatically validate:

  • private extensions;
  • changed model catalogs;
  • changed compose overlays;
  • custom LAN exposure;
  • different Docker Desktop versions;
  • different GPU drivers;
  • unsupported distros or hardware.

Forks should cite upstream receipts and add their own downstream receipt for local changes.