Skip to content

MODEL-MANAGEMENT.md

Latest commit

 

History

History
466 lines (359 loc) · 17.9 KB

MODEL-MANAGEMENT.md

File metadata and controls

466 lines (359 loc) · 17.9 KB
Raw

Model Management

ODS runs local language models as GGUF files from data/models/. The recommended path is the Dashboard Models page. Manual model swaps are also available for headless maintenance and advanced operator workflows.

Recommended: Dashboard Models Page

Open the Dashboard and go to Models.

From there you can:

  • Separate models already installed from the curated ODS catalog.
  • Search compatible GGUF repositories on Hugging Face without leaving ODS.
  • Check approximate model size, VRAM requirement, context length, and specialty.
  • Download a catalog model into data/models/.
  • Import an integrity-qualified Hugging Face GGUF into data/models/.
  • Load a downloaded model.
  • Load a manually copied single-file GGUF discovered in data/models/.
  • Delete a downloaded catalog model.

The expected user flow is a six-verb chain:

  1. Discover a viable model in the catalog.
  2. Download it.
  3. Load it.
  4. Use it through every enabled LLM app.
  5. Restore the original model when validating a temporary swap.
  6. Delete the temporary model when cleanup is part of the workflow.

The Models page should keep compatibility gates visible before a user commits to a load. Agent viability gates are especially important: if an enabled agent declares a context floor such as 65536, models below that floor should be shown as gated or warned before the swap. Badges should distinguish downloaded, loaded, swap-safe, not-swap-safe, gated, and probe-failed states so a model cannot look ready while an enabled app is known to be incompatible.

Hugging Face imports

The Hugging Face source searches the live Hub and only offers complete GGUF artifacts with exact byte-size and SHA-256 metadata. Before download, ODS re-reads the selected repository, pins its immutable revision, rejects projectors, adapters, incomplete split files, and repositories intended for a different runtime, then asks the host agent to download and verify every file. Community imports are labelled as unvalidated until they have been benchmarked on the local machine; they are not added to the ODS recommended catalog.

ODS requests the Hub's parsed GGUF metadata together with the repository and uses its declared context window when available. After download, the context stored in the local GGUF header takes precedence over Hub and catalog values. Some community repositories do not publish parseable context metadata. ODS labels that limit as unknown instead of presenting a guessed maximum, starts from a conservative 8K runtime default, and still permits an explicit context choice through the normal verified activation transaction.

Public repositories require no configuration. For a private or gated repository, accept its upstream license first and set HF_TOKEN in .env or in the Dashboard environment editor. The token is read at request time, is never returned to the browser, and is not written into the import registry. In Settings → Environment Editor, the field is under Provider and Hub Credentials. A stored token is shown only as a masked placeholder; leaving the field blank preserves the existing value. Saving a replacement does not restart the stack because dashboard-api and the host agent read the mounted .env when each Hub request or fallback download starts.

Cancelling a download stops the active transfer and removes job-only temporary files. Any previously verified shards remain available for a later retry. A retry re-reads the immutable Hub revision and verifies every retained or newly downloaded file before the model can become installed. An incomplete or cancelled transfer is never eligible for activation.

Imported metadata is stored separately in data/model-imports.json, so a source update or installer rerun does not modify config/model-library.json or discard community imports. Deleting a downloaded file keeps the import record, allowing the same pinned artifact to be downloaded again. The registry and completed model files are also independent of dashboard-api and host-agent process lifetime: restarting either service reloads the same pinned records and on-disk artifacts.

When a catalog model is loaded, ODS updates the active GGUF settings and restarts the local inference service so OpenAI-compatible clients use the new model. After the switch settles, verify it from the host:

ods model current
curl http://localhost:11434/v1/models

On macOS native Metal and Windows native/Lemonade installs, use http://localhost:8080/v1/models unless you changed the port.

Dashboard activation, Unix ods model swap <tier>, and Windows .\ods.ps1 model swap <tier> use the same authenticated host-agent transaction. The transaction updates .env, models.ini, the native or container inference runtime, LiteLLM, Hermes, OpenClaw, OpenCode, and Perplexica when those consumers are installed. It verifies the new runtime and downstream routes before reporting success. A late failure restores the prior files, runtime, and persisted app routes and then proves the previous model is serving again.

Choosing the runtime context

Before loading a model, the Dashboard offers context presets derived from the catalog and a custom context input for any safe whole number from 1024 tokens. The catalog's recommended context is the conservative default. Its declared maximum is shown separately and is used to build presets, not as a hard block for advanced operators.

A custom value above the declared model context is allowed with a warning. This does not claim that the model, runtime, available VRAM, or enabled apps can serve that value. ODS writes the requested context through the same activation transaction, starts the platform-specific runtime, and requires runtime identity and context proof before committing the swap. If NVIDIA Compose, macOS Metal, Windows native llama-server, or Lemonade cannot establish the requested context, activation fails and the previous model configuration is restored.

The committed value is propagated to CTX_SIZE, MAX_CONTEXT, llama-server or Lemonade runtime configuration, stable model routes, and installed context-aware consumers such as Hermes. Optional stopped services remain stopped while their persisted configuration is updated. Use the Models page to change this value; the corresponding Environment Editor fields remain read-only so direct .env edits cannot bypass activation verification and rollback.

For Hugging Face imports, repository GGUF metadata is preferred over generic model config. After download, the context embedded in the local GGUF header is authoritative and replaces stale Hub metadata. If neither source publishes a usable value, the Dashboard reports the declared limit as unknown instead of inventing one; the operator may still choose an explicit context and let the runtime verification decide whether it can be served.

This selector applies to local inference in local, hybrid, and lemonade modes. In cloud mode, the remote provider owns its context policy, so ODS does not rewrite local runtime or application context from the Models page.

Multi-GPU assignment replanning

On Linux NVIDIA and managed Linux AMD/ROCm installations with more than one GPU, the same transaction also checks the target model's declared or size-derived VRAM envelope against the persisted llama-server assignment. If the existing subset is too small, ODS reruns the topology planner, expands llama-server to a sufficient GPU subset, and updates GPU_ASSIGNMENT_JSON_B64, LLAMA_SERVER_GPU_UUIDS, LLAMA_SERVER_GPU_INDICES, LLAMA_ARG_SPLIT_MODE, and LLAMA_ARG_TENSOR_SPLIT atomically with the model route. Pipeline assignments clear a stale explicit tensor split so llama.cpp can fit layers to the live free memory of heterogeneous cards. If activation fails, the previous model and GPU assignment are both restored and health-proven before rollback is reported as successful.

For AMD, ODS also updates ROCR_VISIBLE_DEVICES. The runtime accepts the expanded assignment only when every selected GPU has a detected gfx architecture. A homogeneous gfx1151 set receives the ODS-managed HSA override and custom llama.cpp binary; ODS refuses to combine gfx1151 with another architecture in one llama-server process because that override is not device-scoped. Custom operator-provided HSA or llama.cpp override values are preserved; only the exact values managed by ODS are removed when they no longer match the selected GPU architecture. Single-GPU, Apple, Windows-native AMD/Lemonade, externally managed inference, and unpersisted all-GPU fallback configurations keep their existing behavior.

To inspect or intentionally override a multi-GPU plan, use the supported CLI instead of editing the encoded assignment in .env:

ods gpu assignment
ods gpu reassign --dry-run --auto
ods gpu reassign --manual
ods gpu validate

Manual reassignment accepts GPU indices for llama-server and optional accelerated services, validates them against the live topology, and persists the readable runtime variables together with GPU_ASSIGNMENT_JSON_B64. Leaving an auxiliary prompt blank preserves its current placement; on a legacy install without prior placement, an enabled service defaults to the first live GPU while a disabled service remains omitted. Applying the change recreates the affected stack without first tearing it down. If Compose rejects the new contract, ODS restores the previous .env and recreates the previous stack; declining the prompt leaves the validated plan saved until the next ods restart. Model activation never expands an assignment marked as manual; if that set is too small for a target model, choose a larger set with ods gpu reassign --manual first. Explicit NVIDIA visibility controls such as all, none, and void are also left unchanged.

Open WebUI, Token Spy, Privacy Shield, and OpenAI-compatible SDK clients follow the stable ODS endpoint and do not persist a separate model route. Optional apps that are not installed are skipped. Optional services that were stopped remain stopped: persisted Hermes/OpenCode state and Compose environment are updated without starting them, and Perplexica reconciles its app-owned state from that environment the next time ODS starts it.

Direct edits to .env, models.ini, or app-owned settings bypass this transaction. The Dashboard Settings editor therefore treats active model, tier, artifact integrity, Lemonade identity, and runtime-profile fields as read-only; use Model Manager instead. Use the manual procedure below only for recovery or unsupported custom models, and verify every affected consumer afterward.

New or updated LLM apps should avoid direct model coupling. The swap-safe extension contract is documented in SWAP-SAFE-EXTENSIONS.md: route through http://litellm:4000/v1, use model ods/current, and declare service.llm when the app needs a context floor, dynamic refresh, or post-swap probe.

Where Models Live

Default model directory:

~/ods/data/models/

On Windows installs:

$env:USERPROFILE\ods\data\models\

Each model is normally a single .gguf file:

ls -lh ~/ods/data/models/*.gguf

The active model is recorded in .env:

grep -E "^(LLM_MODEL|GGUF_FILE|CTX_SIZE|MAX_CONTEXT)=" ~/ods/.env

GGUF_FILE is the filename ODS should load from data/models/. LLM_MODEL is the friendly logical model name used by scripts and config. CTX_SIZE and MAX_CONTEXT control context length.

Hermes requires at least a 64K context window. Installer bootstrap mode uses 65536 for the fast-start model, then switches .env, llama-server, and Hermes config to the model selector's chosen full-model context when the background download completes. Larger tiers may use 131072; constrained tiers can remain at a smaller selected context.

Manual: Download a Catalog Model

For most users, use the Dashboard. If you are debugging a failed download or preloading a machine, download the exact catalog GGUF URL from config/model-library.json into data/models/.

Example:

cd ~/ods
mkdir -p data/models

curl -L \
  -o data/models/Qwen3.5-9B-Q4_K_M.gguf \
  https://huggingface.co/unsloth/Qwen3.5-9B-GGUF/resolve/main/Qwen3.5-9B-Q4_K_M.gguf

Then open Dashboard -> Models. If the filename matches a catalog entry, the model should appear as downloaded and you can load it from the Dashboard.

Bring Your Own GGUF

For a single local .gguf, the normal flow is:

  1. Copy the file into data/models/.
  2. Open Dashboard -> Models.
  3. Load the local entry.

The Dashboard updates .env, config/llama-server/models.ini, and the active runtime routing before restarting the inference service.

On Lemonade installs, loading a model directly inside the Lemonade app only changes Lemonade's current runtime state. It does not update ODS's .env or LiteLLM routing. Open WebUI talks through ODS/LiteLLM, so its next chat can ask for the persisted ODS model and Lemonade may unload the model you opened manually. Use Dashboard -> Models -> Load when you want Open WebUI and other ODS clients to keep using the local GGUF.

Use the manual procedure below only if you cannot access the Dashboard or need to repair an install by hand.

  1. Download the GGUF into data/models/.
cd ~/ods
mkdir -p data/models
cp /path/to/MyModel-Q4_K_M.gguf data/models/
  1. Update .env.
ods config edit

Set:

LLM_MODEL=my-model
GGUF_FILE=MyModel-Q4_K_M.gguf
CTX_SIZE=8192
MAX_CONTEXT=8192
  1. Update config/llama-server/models.ini.
[my-model]
filename = MyModel-Q4_K_M.gguf
load-on-startup = true
n-ctx = 8192
  1. If Hermes is enabled, update data/hermes/config.yaml.
model:
  default: "MyModel-Q4_K_M.gguf"
  context_length: 65536

For Lemonade/AMD backends, use:

model:
  default: "extra.MyModel-Q4_K_M.gguf"
  context_length: 65536

Also keep auxiliary.compression.context_length at the same value and use compression.threshold: 0.50; older absolute-token thresholds can leave Hermes waiting too long to compact.

  1. For AMD/Lemonade installs, verify config/litellm/lemonade.yaml.

Each local model alias should use the extra.<GGUF_FILE> form and should keep Qwen3 thinking disabled for clients that do not pass that flag themselves:

extra_body:
  chat_template_kwargs:
    enable_thinking: false
  1. If Perplexica is enabled, reseed or verify its model setting.
LLM_MODEL="$(grep -E '^LLM_MODEL=' .env | tail -n1 | cut -d= -f2 | tr -d '"')"
PERPLEXICA_PORT="$(grep -E '^PERPLEXICA_PORT=' .env | tail -n1 | cut -d= -f2 | tr -d '"')"
scripts/repair/repair-perplexica.sh "http://127.0.0.1:${PERPLEXICA_PORT:-3004}" "$LLM_MODEL"

Dashboard activation and ods model swap handle this automatically. Raw GGUF or .env edits still require verification because Perplexica stores its own app settings in its volume.

  1. Restart the affected services.
ods restart llama-server
ods restart litellm
docker restart ods-hermes 2>/dev/null || true

If your install uses direct Docker Compose commands instead of the ods CLI, recreate llama-server so it rereads .env.

Verify a Switch

Use these checks after Dashboard or manual model changes:

ods model current
curl http://localhost:11434/v1/models

For LiteLLM installs that require an API key, use the key from .env:

LITELLM_KEY=$(grep '^LITELLM_KEY=' .env | cut -d= -f2-)
curl -H "Authorization: Bearer $LITELLM_KEY" http://localhost:4000/v1/models

From inside a Docker container, the inference endpoint is:

http://llama-server:8080/v1

For release or harness validation, do not stop at server identity. A valid model-management pass proves the full verb chain for the selected tier:

  • release tier: a six-model matrix per host, with download, load, app use, restore, and cleanup evidence for each planned target;
  • smoke tier: one complete verb chain through one planned test model;
  • app probes: every enabled LLM consumer discovered from manifests or known routing config is probed after the swap;
  • Open WebUI: an auth wall, missing admin credential, or HTTP 401 is not a passing probe. Provision an admin/API credential for the lane, or mark the probe red/deferred with the reason visible in the report;
  • agent gates: context and capability floors for Hermes-style agents remain visible before selection and are rechecked after load.

Troubleshooting

The download finished, but the model is not visible

Check the file is present and non-empty:

ls -lh data/models/*.gguf

If it is a catalog model, confirm the filename exactly matches config/model-library.json. The Dashboard only marks catalog models as downloaded when the on-disk filename matches the catalog entry.

The model file exists, but loading fails

Check service logs:

ods logs llm

Common causes:

  • The model needs more VRAM or unified memory than the machine has.
  • Context length is too high; lower CTX_SIZE / MAX_CONTEXT.
  • The GGUF is not compatible with the active backend.
  • On AMD/Lemonade, a service is still asking for the raw filename instead of extra.<GGUF_FILE>.

Open WebUI or another app still shows the old model

Verify the server first:

curl http://localhost:11434/v1/models

If the server is correct, refresh the app. If the server is wrong, restart llama-server and verify .env / models.ini.

Hermes still asks for the old model

Hermes has its own config:

grep -n "default:\|context_length:" data/hermes/config.yaml
docker restart ods-hermes

For AMD/Lemonade, use extra.<GGUF_FILE>.

Current Limitations

  • Dashboard download and load support the ODS catalog and integrity-qualified GGUF artifacts discovered through the Hugging Face source.
  • Import from an arbitrary URL is not a first-class Dashboard workflow. Local single-file GGUFs are discovered after they are copied into data/models/.
  • ods model swap switches ODS tiers, not arbitrary GGUF files.
  • scripts/upgrade-model.sh is a legacy helper for model-directory layouts and should not be used as the primary GGUF switch path on current installs.