Skip to content

run.md

Latest commit

 

History

History
242 lines (164 loc) · 15 KB

run.md

File metadata and controls

242 lines (164 loc) · 15 KB
Raw

Run

Terminology used throughout this doc:

  • Runtime — llama_cpp or qairt (the plugin_id).
  • Compute unit — NPU, GPU, CPU, or hybrid (mapped from --device / device_id).
  • Chipset — the SoC, e.g. Snapdragon X Elite (SM8750, SM8850, …).

Two runtimes ship with geniex and both can drive the Snapdragon NPU, but through separate user-space stacks that consume different model formats:

  • llama_cpp — GGUF models, targets Hexagon NPU (via ggml-hexagon), Adreno GPU (via OpenCL), or CPU.
  • qairt — QAIRT .bin shards, targets Hexagon NPU via Qualcomm's QNN runtime.

They are not interchangeable; the runtime is chosen per model.

For the CI/S3 signing pipeline that backs HTP releases, see release.md § Hexagon HTP signing.

Compute-unit aliases

The alias table lives in the SDK, not in the bindings: sdk/src/device.cpp exposes geniex_resolve_device via sdk/include/geniex.h. The Go wrapper (bindings/go/device.go), Python wrapper (resolve_device in bindings/python/geniex/_ffi/_api.py), and Android/JNI wrapper (resolve_device in bindings/android/app/src/main/cpp/jniutils.cpp) are all thin FFI shims over that one function. Editing alias semantics means editing sdk/src/device.cpp, rebuilding the SDK bridge (/build), and possibly updating all three FFI stubs if the struct shape changes (see CONTRIBUTING.md for the FFI-sync rule).

Alias device_id sent to SDK n_gpu_layers override Use case
cpu empty 0 Pure CPU.
gpu GPUOpenCL --ngl (default -1) Adreno via OpenCL.
npu HTP0 --ngl (default -1) Pinned single-session HTP. Deterministic, slower on LLMs — see § NPU compute-unit selection (llama_cpp).
hybrid empty --ngl (default -1) llama_cpp per-tensor HTP+CPU scheduler.

--ngl defaults to -1, which llama.cpp reads as "all layers", so gpu / npu / hybrid offload everything unless --ngl is set. The value passes through the SDK unchanged. qairt ignores --ngl (forced to 0).

Defaults when the user passes nothing (--device "" / device_map="auto"): npu for both llama_cpp and qairt. QAIRT exposes only one device, so cpu / gpu / hybrid against a qairt model get coerced to NPU with a warning on stderr — the CLI does not exit early.

Beyond the aliases, --compute also accepts an explicit device list of concrete ids (HTP0,HTP1,HTP2,HTP3, GPUOpenCL) — llama_cpp passes it through to llama.cpp verbatim (handy for multi-DSP recipes that need more than the single HTP0 the npu alias pins); --ngl still applies. qairt is NPU-only, so a device list gets coerced to NPU with a warning.

Compute-unit selection (llama_cpp)

llama_cpp supports OpenCL and Hexagon on Windows ARM64. The compute unit is driven by two inputs on geniex_LlmCreateInput:

  • device_id — string, runtime-specific (HTP0, GPUOpenCL, CPU, …).
  • config.n_gpu_layers — int; how many layers to offload. -1 = all.

NPU compute-unit selection (llama_cpp)

sdk/plugins/llama_cpp/src/llm.cpp:73-114 branches on whether device_id is non-null, producing two runtime paths with very different performance:

  1. device_id null + n_gpu_layers=-1 (the hybrid alias) → llama.cpp's per-tensor scheduler. It inspects each tensor and assigns it to whichever registered backend supports the op (HTP for computable ops, CPU for fallbacks), using CPU-resident buffers for the fallback tensors. Fast path. On X1E80100 + Qwen3-1.7B-Q8_0: ~90 tok/s prefill, ~27 tok/s decode, ~200 ms TTFT. Task Manager shows NPU pegged.

  2. device_id="HTP0" + n_gpu_layers=-1 (the npu alias) → runtime calls ggml_backend_dev_by_name("HTP0") and sets mpar.devices = {HTP0}. This pins the model to a single compute-unit layout and disables per-tensor hybrid assignment. Any op HTP doesn't support gets handled less efficiently. On the same model: ~60 tok/s prefill, ~22 tok/s decode, ~350 ms TTFT. Task Manager shows CPU pegged (the host thread driving HTP busy-waits, plus all fallbacks run there). Useful when you want deterministic layout / all weights on a known compute unit. Note: a non-zero n_gpu_layers is required even with the compute unit pinned — device_id="HTP0" with ngl=0 opens an HTP session and then runs every layer on CPU, so the default ngl=-1 (all layers) applies for this alias (sdk/src/device.cpp).

Bonus: when the device_id string starts with "HTP0", the runtime also flips KV cache to Q8_0 and enables flash-attn (llm.cpp:136-140). Orthogonal to perf — path (2) is slower than (1) even with those enabled.

Rule of thumb: use --device hybrid (or leave --device empty) for fastest throughput; use --device npu when you need determinism or when debugging placement.

History: the fb98467 commit ("add device parameter") originally made --device npu synthesize device_id="HTP0", collapsing the fast path. That was reverted (hybrid became the implicit default), then the two semantics were split into explicit npu / hybrid aliases to let callers pick.

Running from the CLI

Use --device (-d):

geniex infer Qwen/Qwen3-1.7B-GGUF                 # hybrid (default) for llama.cpp
geniex infer Qwen/Qwen3-1.7B-GGUF --device npu    # pinned HTP0
geniex infer Qwen/Qwen3-1.7B-GGUF --device hybrid # explicit hybrid
geniex infer Qwen/Qwen3-1.7B-GGUF --device gpu
geniex infer Qwen/Qwen3-1.7B-GGUF --device cpu

Sanity-checking which path actually ran

The SDK's default log handler is a no-op in release builds (sdk/src/ml.cpp:36-60), so stdout/stderr stays silent and "did it actually use HTP?" is easy to guess wrong. Ways to check:

  • Python: set GENIEX_LOG=INFO. The Python binding installs a geniex_set_log callback that routes SDK messages (Found device: HTP0, Using N device(s), etc.) to stderr. If you see Found device: … lines you're on the pinned-HTP0 path (the npu alias); absence = hybrid path.
  • Windows: Task Manager's NPU graph. Hybrid lights it up; pinned-HTP0 pegs the CPU (host thread busy-waits HTP the whole inference).
  • Signature: on Snapdragon X1E80100 + a 1.7B Q8_0 model, hybrid gives prefill ≳ 80 tok/s and TTFT ≲ 250 ms; pinned-HTP0 gives prefill ≲ 65 tok/s and TTFT ≳ 340 ms. Prefill and TTFT separate the two paths more cleanly than decode.
  • Threadpool: any offloaded path (all aliases except cpu) logs threadpool tuned for offload: 6 threads pinned to cores [2, 8), strict, poll=1000 — the SDK mirrors upstream's fixed -t 6 --cpu-mask 0xfc (sdk/plugins/llama_cpp/src/threadpool.cpp); pass n_threads in geniex_ModelConfig to override. Absent on --device cpu.

If you see Device '…' not found, skipping, the runtime loaded but the GGML backend DLL did not — verify test-signing is still on (for HTP) or that ggml-opencl.dll is present in sdk/pkg-geniex/lib/llama_cpp/.

Q4_K_M is a suboptimal quant for HTP — it prefers Q4_0 / Q8_0, so some tensors fall back to CPU. Use Q4_0 for a clean NPU run.

Running QAIRT models

QAIRT exposes only its Hexagon NPU compute unit (plugin_id="qairt", device_id="NPU"). The SDK's geniex_resolve_device coerces --device cpu / gpu / hybrid to npu with a stderr warning so existing shell pipelines don't break — expect a line like:

Warning: qairt plugin only supports NPU inference; ignoring device='cpu' and running on NPU

QAIRT models need a geniex.json to work. See the granite4_micro example.

Build and run locally

hf download yichqian/geniex-qairt-models --local-dir=geniex-qairt-models
bazelisk run //cli -- pull local/granite4_micro \
  --model-hub localfs \
  --local-path /absolute/path/to/geniex-qairt-models/granite4_micro
bazelisk run //cli -- infer local/granite4_micro

Hand off a build to another machine

Builder:

bazelisk build //cli:artifact
# Export bazel-bin/cli/artifact.zip and ggml-htp-v1.cer

Recipient:

# unzip artifact.zip
hf download yichqian/geniex-qairt-models --local-dir=geniex-qairt-models
./geniex.exe pull local/granite4_micro --model-hub localfs \
  --local-path /absolute/path/to/geniex-qairt-models/granite4_micro
./geniex.exe infer local/granite4_micro

Running a prebuilt CI release (Windows on Snapdragon)

Every v* tag publishes the Windows ARM64 installer on the Releases page. Download both:

  • geniex-cli-setup.exe — the installer
  • geniex-sdk-windows-arm64-<tag>.zip — the SDK

The SDK filename encodes the HTP signing flavor:

Filename HTP signing Extra setup
geniex-sdk-windows-arm64-<tag>.zip Microsoft-signed None — skip to "Run" below.
geniex-sdk-windows-arm64-<tag>-selfsigned.zip Self-signed (test) See Self-signed fallback.

If the release also attaches ggml-htp-v1.cer, you're on the self-signed flavor.

Run:

  1. Install with geniex-cli-setup.exe.
  2. hf download yichqian/geniex-qairt-models --local-dir=geniex-qairt-models
  3. geniex.exe pull local/granite4_micro --model-hub localfs --local-path <abs-path>\geniex-qairt-models\granite4_micro
  4. geniex.exe infer local/granite4_micro

Self-signed fallback

Only needed when the release ships the -selfsigned SDK plus ggml-htp-v1.cer. Windows refuses to load libggml-htp.cat until you both enable test signing and trust the cert.

Pre-built users already have ggml-htp-v1.cer from the release page — skip ahead to step 2 below.

Builders (generating their own cert for a local build): run these in elevated cmd.exe. The .pfx is what HEXAGON_HTP_CERT needs at build time; the .cer is what's imported into the trust stores.

set "PATH=C:\Program Files (x86)\Windows Kits\10\bin\10.0.26100.0\arm64;%PATH%"
mkdir C:\Users\%USERNAME%\Certs
cd C:\Users\%USERNAME%\Certs
makecert -r -pe -ss PrivateCertStore -n CN=GGML.HTP.v1 -eku 1.3.6.1.5.5.7.3.3 -sv ggml-htp-v1.pvk ggml-htp-v1.cer
pvk2pfx -pvk ggml-htp-v1.pvk -spc ggml-htp-v1.cer -pfx ggml-htp-v1.pfx
setx /M HEXAGON_HTP_CERT "C:\Users\%USERNAME%\Certs\ggml-htp-v1.pfx"

makecert prompts twice for a password — leave blank for a throwaway dev cert. Do not reuse a .cer extracted from someone else's signed binary: it has no private key (so it's unusable for signing) and importing a random third-party root is a security risk.

Then, for both builders and pre-built users:

  1. Enable test signing (elevated PowerShell, then reboot):

    bcdedit /set TESTSIGNING ON

    If this fails with a Secure Boot error, disable Secure Boot in UEFI first, then retry.

  2. Import ggml-htp-v1.cer into two stores via certlm.msc (must be launched elevated, else imports fail with "store was read only"):

    • Trusted Root Certification Authorities → Certificates → right-click → All Tasks → Import… → select ggml-htp-v1.cer.
    • Repeat into Trusted Publishers → Certificates.

    Both stores are required: Root makes the chain valid; Trusted Publishers suppresses the driver-load prompt.

  3. Reboot if you haven't yet. Verify:

    bcdedit /enum | Select-String testsigning   # should show "testsigning   Yes"

Upstream background: third-party/llama.cpp/docs/backend/snapdragon/windows.md.

Update checks

geniex consults a cached "latest release" entry and prints a one-line notice if a newer version exists, at most once per 8 h. A background refresh re-fetches at most every 24 h.

The version data comes from a public S3 index (qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-geniex/index.json). Failures are silent (logged at debug only, no stdout spam).

Run geniex update to upgrade:

  • Windows — downloads and launches the signed installer once it's published; otherwise reports "up-to-date".
  • Linux — prints the install-script one-liner to re-run (curl -fsSL … | bash); auto-update is not wired up yet.

Pass --skip-update on any command to skip the probe (and the notify banner) entirely for that invocation.

Performance metrics

--verbose (and the Go/Python APIs' ProfileData, and geniex-bench) report one number per inference phase. Which metric covers which phase:

Metric Field Phase it measures
ttft ttft Start of generate → first sampled token. For a VLM this includes the media encoder, so it is not comparable to a pure prefill number.
media time media_time The vision/audio encoder only — turning pixels/audio into decoder-space embeddings. 0 on text-only runs.
prompt / prefill time prompt_time Prefill — running the prompt tokens through the model. On a VLM this includes prefilling the media (soft) tokens; it excludes the encoder.
prompt / prefill speed prefill_speed prompt_tokens / prompt_time.
decode time / speed decode_time / decoding_speed Generation phase (first token → last token).

Key points:

  • media_time is the encoder only. Everything downstream of the encoder (prefilling the media tokens through the model) lives in prompt_time, same as text.
  • prompt_tokens counts text + media tokens on a VLM run, so prefill_speed reflects the full prefill the model actually did.
  • ttft spans encoder + prefill, so ttft ≈ media_time + prompt_time for a VLM.

Both runtimes measure media_time at the same boundary (encoder wall time only), so the numbers are comparable across plugins:

  • llama.cpp — timed per-chunk: the encode (mtmd_encode_chunk) is media_time; prefilling the embeddings (mtmd_helper_decode_image_chunk → llama_decode) goes to prompt_time, same as text chunks. Bitmap loading and tokenization are timed by neither, so they fall only in ttft; ttft ≈ media_time + prompt_time up to that overhead.
  • QAIRT — media_time is the encoder wall time (encodeVision); the media-token NPU prefill stays in prompt_time (= ttft − media_time). QAIRT's prompt_time is derived from ttft, so treat it as indicative, not exact. The encoder number itself is directly measured.