GitHub - EricLBuehler/candle-vllm: Efficent platform for inference and serving local LLMs including an OpenAI compatible API server.

Efficient, easy-to-use platform for inference and serving local LLMs including an OpenAI compatible API server.
English | 简体中文

✨ Why Candle-vLLM?

	Feature	Details
⚡	High Performance	Native Flash Attention, FlashInfer, CUDA Graphs, continuous batching, prefix caching.
🗜️	Aggressive KV Compression	TurboQuant (`2–4 bit` KV cache) extends context up to 4.7× with minimal quality loss
🌍	Cross-platform	CUDA (Linux), Metal (macOS). Same codebase, same API
🏭	Production-ready	OpenAI-compatible API server, built-in ChatGPT-style Web UI, MCP tool calling, streaming
📦	Easy to deploy	One-line install script, Docker images, or build from source
🔧	Extensible	Trait-based architecture for rapid implementation of new model pipelines
🖥️	Multi-GPU & Multi-Node	Multi-process and multi-threaded tensor parallelism, TCP-based multi-node inference

🚀 Quick Start

📦 Install

Option 1 — One-line install (DEB or binary)

curl -sSL https://ericlbuehler.github.io/candle-vllm/install.sh | bash

Option 2 — Build from source

git clone git@github.com:EricLBuehler/candle-vllm.git
cd candle-vllm

# CUDA (11+, 12+, 13.0) — remove flashinfer,cutlass for sm_70/sm_75
cargo install --features cuda,nccl,flashinfer,cutlass --path .

# macOS/Metal
cargo install --features metal --path .

Option 3 — Docker

# Pass custom SM version and CUDA version: ./build_docker.sh "cuda,nccl,flashinfer,cutlass" sm_90 13.0.0
./build_docker.sh "cuda,nccl,flashinfer,cutlass"

▶️ Run

Using HuggingFace Model ID:

candle-vllm --m Qwen/Qwen3.6-27B-FP8 --ui-server

candle-vllm --m unsloth/Qwen3.5-122B-A10B-GGUF --f Q3_K_S --d 0,1 --ui-server

candle-vllm --m zai-org/GLM-5.2-FP8 --d 0,1,2,3,4,5,6,7 --ui-server

Using local model path:

# Local safetensors directory
candle-vllm --d 0,1,2,3,4,5,6,7 --m /home/data/GLM-5.2-FP8/ --ui-server

# Local GGUF file (single or split-shard)
candle-vllm --d 0,1 --m /home/data/model-Q4_K_M.gguf --ui-server

# Local directory containing GGUF files (auto-detected)
candle-vllm --d 0,1 --m /home/data/Qwen3.5-35B-A3B-GGUF/ --ui-server

Tip: Add --ui-server to launch the built-in ChatGPT-style Web UI. The UI server uses the API port minus one (e.g., API on 2000, UI on 1999).

📈 Performance

Single-request decode speed (input 4k, output 1k, on Hopper 80G)

#	Model	BF16 (Decode Speed / req)	Quantized
1	LLAMA	119 tks/s (8B)	163 tks/s (8B, Q4K), 171 tks/s (8B, Marlin)
2	Mistral	122 tks/s (7B)	181 tks/s (7B, Q4K), 190 tks/s (7B, Marlin)
3	Phi3/Phi4	153 tks/s (3.8B)	196 tks/s (3.8B, Q4K)
4	QWen2/Qwen3 Dense	127 tks/s (8B)	154 tks/s (8B, Q4K)
5	QWen3 MoE	102 tks/s (30B)	124 tks/s (30B, Q4K)
6	QWen3-Next MoE	80 tks/s (80B, BF16, tp=2)	89 tks/s (AWQ, tp=1)
7	QWen3.5/3.6/3.8 Dense	36 tks/s (27B, BF16)	49/58 tks/s (27B, Q4K / FP8)
8	QWen3.5/3.6 MoE	90 tks/s (35B)	118 tks/s (35B, FP8)
9	Yi	168 tks/s (6B)	199 tks/s (6B, Q4K)
10	StableLM	251 tks/s (3B)	-
11	Gemma-2/Gemma-3	103 tks/s (9B)	130 tks/s (9B, Marlin)
12	DeepSeek V2/V3/V3.2/R1	TBD	~20 tks (AWQ 671B, tp=8, offloading)
13	QwQ-32B	51 tks/s (32B, tp=2)	70 tks/s (32B, Q4K)
14	GLM4	96 tks/s (9B)	139 tks/s (9B, Q4K)
15	GLM4.7 Flash	TBD	82 tks/s (31B, Software NVFP4)
16	LLama4	TBD	47 tks/s (107B, Software NVFP4)
17	Gemma4	(26B) 83 tks/s	82 tks/s (26B, Software NVFP4)
18	MiniMax-M2.5/M2.7	TBD	72 tks/s (229B, Software NVFP4, TP=2)
19	GLM-5.2	TBD	Supported (FP8, tp=8)

Demo Video — GPU & Apple Silicon

Chat demo on GPU (A100, BF16, QWen3-8B Reasoning Model)

Chat demo on Apple Silicon (M4, 16GB unified memory, Q2K, QWen3-8B)

🧠 Features

OpenAI compatible API server for serving LLMs
Streaming support in generation
Efficient KV cache management with PagedAttention
Continuous batching (batched decoding for incoming requests over time)
In-situ quantization (and In-situ Marlin format conversion)
GPTQ/Marlin format quantization (4-bit)
Support Mac/Metal devices
Support Multi-GPU inference (both multi-process and multi-threaded mode)
Support Multi-node inference via TCP-based coordination
Support Chunked Prefilling (default chunk size 8K)
Support CUDA Graph
Support Qwen3.5 MTP speculative decoding with CUDA Graph via --mtp
Support Model Context Protocol (MCP) and OpenAI-compatible tool calling
Support Prefix Caching
Support Block-wise FP8 Models (SM90+, Qwen3 Series)
Support FP8 KV Cache on all CUDA and Metal platforms
Support TurboQuant KV Cache (turbo8/turbo4/turbo3) with native flash attention kernels
Support Flashinfer Backend
Support manual YaRN RoPE scaling override via --yarn-scaling-factor
Support MXFP4/NVFP4 models
Support GPTQ/AWQ/Marlin models, including pack-quantized AWQ MoE
Support DeepSeek V3.2 and GLM-5.2 FP8 models

📘 Usage

Running Models

Tip: By default, candle-vllm starts an OpenAI-compatible API server at http://localhost:2000. Add --ui-server to also launch the built-in ChatGPT-style Web UI.

# FP8 model + Web UI
candle-vllm --m Qwen/Qwen3.8-27B-FP8 --ui-server

# GLM-5.2 FP8 model
candle-vllm --d 0,1,2,3,4,5,6,7 --m zai-org/GLM-5.2-FP8 --ui-server

# Unquantized Safetensors (multi-GPU)
candle-vllm --d 0,1 --w /home/Qwen3-30B-A3B-Instruct-2507/

# ISQ on-the-fly quantization
candle-vllm --m Qwen/Qwen3.6-27B --isq q4k

# FP4 Model
candle-vllm --m GadflyII/GLM-4.7-Flash-NVFP4 --ui-server

# GGUF model
candle-vllm --m unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF --f Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf --ui-server

# Manual YaRN scaling
candle-vllm --m Qwen/Qwen3.6-35B-A3B --yarn-scaling-factor 4.0 --ui-server

FP8 / FP4 models

# FP8 Model (block-wise quant, build with cutlass feature)
candle-vllm --m Qwen/Qwen3.8-27B-FP8 --ui-server

# Faster GDN prefill on Hopper with slight precision loss
SM90_LOWER_PRECISION_GDN_PREFILL=1 candle-vllm --m Qwen/Qwen3.5-35B-A3B-FP8

# Qwen3.5 MTP speculative decoding (2 draft tokens per step)
candle-vllm --w /data/Qwen3.5-35B-A3B-FP8/ --mtp 2 --ui-server

# GLM-5.2 FP8 Model
candle-vllm --d 0,1,2,3,4,5,6,7 --m zai-org/GLM-5.2-FP8 --ui-server

# FP8 on MacOS/Metal (Dense)
candle-vllm --m Qwen/Qwen3-4B-Instruct-2507-FP8 --ui-server

# FP4 Model (MXFP4/NVFP4, MLX quantized format not supported)
candle-vllm --m GadflyII/GLM-4.7-Flash-NVFP4 --ui-server

# MXFP4
candle-vllm --m nm-testing/Qwen3-30B-A3B-MXFP4A16 --ui-server

GGUF models

# Local GGUF file via --m (recommended)
candle-vllm --m /home/data/Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf --ui-server

# Local GGUF file via --f (legacy)
candle-vllm --f /home/data/Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf --ui-server

# Local directory containing GGUF (auto-detected, mmproj loaded on demand)
candle-vllm --m /home/data/Qwen3.5-35B-A3B-GGUF/ --ui-server

# From HuggingFace (exact file)
candle-vllm --m unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF --f Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf --ui-server

# From HuggingFace (subfolder — downloads all GGUF files in the remote path)
candle-vllm --m unsloth/Qwen3.5-122B-A10B-GGUF --f Q3_K_S --d 0,1 --ui-server

# Local multi-shard GGUF (split files auto-discovered from local path)
candle-vllm --m /home/data/model-00001-of-00003.gguf --d 0,1 --ui-server

# GGUF on Apple Silicon
candle-vllm --m /home/qwq-32b-q4_k_m.gguf --ui-server
candle-vllm --m Qwen/QwQ-32B-GGUF --f qwq-32b-q4_k_m.gguf --ui-server

Multi-shard GGUF: Split GGUF files (e.g., model-00001-of-00005.gguf) are automatically discovered — both locally (from the same directory) and remotely (from the HuggingFace repo). When --f is a subfolder name (not ending in .gguf), all GGUF files in that remote subfolder are downloaded. Vision tower auxiliary files (mmproj*.gguf) are loaded on demand for multimodal models.

ISQ (In-situ quantization)

Simply add --isq parameter when running unquantized models:

candle-vllm --m Qwen/Qwen3.6-27B --isq q4k

Options: q4_0, q4_1, q5_0, q5_1, q8_0, q2k, q3k, q4k, q5k, q6k

GPTQ / AWQ / Marlin models

# Marlin-compatible GPTQ (4-bit, 128-group, desc_act=False)
candle-vllm --m thesven/Llama-3-8B-GPTQ-4bit

# Convert uncompressed model to Marlin-compatible format
python3 examples/convert_marlin.py --src /home/DeepSeek-R1-Distill-Qwen-14B/ --dst /home/DeepSeek-R1-Distill-Qwen-14B-GPTQ_4bit-128g
candle-vllm --w /home/DeepSeek-R1-Distill-Qwen-14B-GPTQ_4bit-128g

# Convert AWQ to Marlin-compatible format
python3 examples/convert_awq_marlin.py --src /home/Meta-Llama-3.1-8B-Instruct-AWQ-INT4/ --dst /home/Meta-Llama-3.1-8B-Instruct-AWQ-INT4-Marlin/ --bits 4 --method awq --group 128 --nk False
candle-vllm --d 0 --w /home/Meta-Llama-3.1-8B-Instruct-AWQ-INT4-Marlin/

# AWQ MoE (pack-quantized), e.g. Qwen3-Coder-Next
candle-vllm --m cyankiwi/Qwen3-Coder-Next-AWQ-4bit

# Direct Marlin-format model
candle-vllm --w /home/DeepSeek-R1-Distill-Qwen-14B-GPTQ-Marlin/

🗜️ TurboQuant KV Cache

TurboQuant compresses the KV cache using Walsh-Hadamard transform for higher throughput and longer context:

Mode	Description	KV Cache Compression	Recommended Use
`turbo8`	FP8 K + 4-bit V	~2.6x	Best quality-compression trade-off
`turbo4`	4-bit K + 4-bit V	~3.7x	Balanced quality and memory savings
`turbo3`	3-bit K + 4-bit V	~4.7x	Maximum memory savings

# Turbo4 (4-bit KV cache, ~3.7x compression)
candle-vllm --w /data/Qwen3.5-27B-FP8/ --kvcache-dtype turbo4

# Turbo8 (FP8 K + 4-bit V, ~2.6x compression)
candle-vllm --w /data/Qwen3.5-27B-FP8/ --kvcache-dtype turbo8

# Turbo3 (3-bit K + 4-bit V, ~4.7x compression)
candle-vllm --w /data/Qwen3.5-27B-FP8/ --kvcache-dtype turbo3

# FP8 KV Cache
candle-vllm --w /data/Qwen3.5-35B-A3B-FP8/ --kvcache-dtype fp8

Note: TurboQuant uses native flash attention kernels (flashinfer is automatically disabled). Supported on both CUDA (SM70+) and Metal (Apple Silicon) platforms. MLA models (DeepSeek, GLM4/GLM-5.2) auto-fallback to standard KV cache as TurboQuant is incompatible with their compressed KV layout.

🖥️ Multi-GPU Inference

Multi-process mode (recommended)

# QwQ-32B BF16 on two GPUs
candle-vllm --d 0,1 --w /home/QwQ-32B/

# QwQ-32B 4-bit AWQ on two GPUs
python3 examples/convert_awq_marlin.py --src /home/QwQ-32B-AWQ/ --dst /home/QwQ-32B-AWQ-Marlin/ --bits 4 --method awq --group 128 --nk False
candle-vllm --d 0,1 --w /home/QwQ-32B-AWQ-Marlin/

Note: Number of GPUs (--d) must be a power of 2 (e.g., 2, 4, or 8).

Multi-threaded mode (debug)

# Add --multithread parameter
candle-vllm --multithread --d 0,1 --w /home/QwQ-32B/

# Troubleshooting
export NCCL_P2P_DISABLE=1  # disable P2P if encountering illegal memory access

🌐 Multi-Node Inference

Distribute inference across multiple machines using TCP-based NCCL bootstrap. No MPI required.

# On master node (192.168.1.100):
candle-vllm --d 0,1,2,3,4,5,6,7 --w /data/DeepSeek-R1-AWQ-Marlin/ \
  --num-nodes 2 --node-rank 0 --master-addr 192.168.1.100 --master-port 29500

# On worker node (192.168.1.101):
candle-vllm --d 0,1,2,3,4,5,6,7 --w /data/DeepSeek-R1-AWQ-Marlin/ \
  --num-nodes 2 --node-rank 1 --master-addr 192.168.1.100 --master-port 29500

All nodes must have model weights locally and be TCP-reachable on --master-port (default 29500).

Flag	Description
`--num-nodes N`	Total number of nodes in the cluster
`--node-rank R`	This node's rank (0 = master)
`--master-addr ADDR`	IP address of the master node
`--master-port PORT`	Port for NCCL ID exchange (default: 29500)

📐 NUMA Binding

Show command

sudo apt-get install numactl

# 8 GPUs, 2 NUMA nodes
MAP_NUMA_NODE=0,0,0,0,1,1,1,1 numactl --cpunodebind=0 --membind=0 candle-vllm --d 0,1,2,3,4,5,6,7 --w /home/data/DeepSeek-V2-Chat-AWQ-Marlin

# 4 GPUs
MAP_NUMA_NODE=0,0,0,0 numactl --cpunodebind=0 --membind=0 candle-vllm --d 0,1,2,3 --w /home/data/DeepSeek-V2-Chat-AWQ-Marlin

numactl --cpunodebind=0 --membind=0 specifies the master rank's NUMA binding and must match MAP_NUMA_NODE.

⚙️ CLI Reference

Flag	Description
`--h`	Bind address (default `0.0.0.0`). Supports `host`, `host:port`, `[ipv6]:port`, `tcp://host[:port]`, `file:///path`, `socket:///path`, `unix:///path`
`--p`	TCP server port when `--h` does not include a port (default `2000`)
`--d`	Device IDs (e.g. `--d 0,1`)
`--m`	Model source: HuggingFace model ID, local directory, or local `.gguf` file. Auto-detects GGUF vs safetensors in directories
`--w`	Local weight directory (safetensors or GGUF). Prefer `--m <local_dir>` for new commands
`--f`	GGUF file or subfolder: `--m repo --f file.gguf` (exact file), `--m repo --f subfolder` (all GGUFs in path), or local GGUF path
`--dtype`	Data type (`bf16`, `f16`)
`--isq`	In-situ quantization: `q4_0`, `q4_1`, `q5_0`, `q5_1`, `q8_0`, `q2k`, `q3k`, `q4k`, `q5k`, `q6k`
`--kvcache-dtype`	KV cache quantization: `auto`, `fp8`, `turbo8`, `turbo4`, `turbo3`
`--kv-fraction`	Auto-size KV cache as fraction of remaining GPU memory (default `0.6`)
`--mem`	Fixed KV cache budget in MB
`--prefill-chunk-size`	Prefill chunk size (default 8K, `0` to disable)
`--max-gen-tokens`	Max output tokens per response (default: 1/5 of max_sequence_len)
`--frequency-penalty`	Frequency penalty (−2.0 to 2.0)
`--presence-penalty`	Presence penalty (−2.0 to 2.0)
`--yarn-scaling-factor`	YaRN RoPE context extension factor
`--enforce-parser`	Force tool parser backend: `qwen_coder`, `qwen`, `json`, `mistral`
`--ui-server`	Start with built-in ChatGPT-like Web UI
`--multithread`	Use multi-threaded mode (debug)
`--num-nodes`	Total nodes in cluster (multi-node)
`--node-rank`	This node's rank (0 = master)
`--master-addr`	Master node IP address
`--master-port`	NCCL ID exchange port (default `29500`)
`--disable-prefix-cache`	Disable prefix caching (enabled by default)
`--prefix-cache-max-tokens`	Cap prefix cache size
`--disable-cuda-graph`	Disable CUDA graph capture (enabled by default on CUDA)

Binding examples:

candle-vllm --h 127.0.0.1 --p 8000 --m Qwen/Qwen3.6-27B-FP8
candle-vllm --h 127.0.0.1:8000 --m Qwen/Qwen3.6-27B-FP8
candle-vllm --h '[::1]:8000' --m Qwen/Qwen3.6-27B-FP8
candle-vllm --h unix:///tmp/candle-vllm.sock --m Qwen/Qwen3.6-27B-FP8

📚 Documentation

Guide	Description
Rust Crate Usage	Use as a Rust library
Embedding Models	Text embedding API
MCP & Tool Calling	Model Context Protocol integration
Tool Call Parsing	Tool call detection and parsing
Prefix Cache	Automatic KV cache reuse
Multimodal Models	Vision-language models

Using Agents under Candle-vLLM backend: xbot · OpenCode · Kilo Code

🛠️ Roadmap

📚 References

Python implementation: vllm-project
vllm paper

Report Issue

If you encounter any problems, please create an issue.

Name		Name	Last commit message	Last commit date
.devcontainer		.devcontainer
.github		.github
.vscode		.vscode
docs		docs
examples		examples
paged		paged
res		res
src		src
.dockerignore		.dockerignore
.gitignore		.gitignore
.typos.toml		.typos.toml
Cargo.toml		Cargo.toml
Dockerfile		Dockerfile
LICENSE		LICENSE
README-CN.md		README-CN.md
README.md		README.md
build.rs		build.rs
build_docker.sh		build_docker.sh
setup.py		setup.py

Navigation Menu

Repository files navigation

✨ Why Candle-vLLM?

🚀 Quick Start

📦 Install

▶️ Run

📈 Performance

🧠 Features

📘 Usage

Running Models

🗜️ TurboQuant KV Cache

🖥️ Multi-GPU Inference

🌐 Multi-Node Inference

📐 NUMA Binding

⚙️ CLI Reference

📚 Documentation

🛠️ Roadmap

📚 References

Report Issue

About

Releases

Packages

Used by

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

✨ Why Candle-vLLM?

🚀 Quick Start

📦 Install

▶️ Run

📈 Performance

🧠 Features

📘 Usage

Running Models

🗜️ TurboQuant KV Cache

🖥️ Multi-GPU Inference

🌐 Multi-Node Inference

📐 NUMA Binding

⚙️ CLI Reference

📚 Documentation

🛠️ Roadmap

📚 References

Report Issue

About

Resources

Stars

Watchers

Forks

Releases

Packages

Used by

Contributors

Languages