[Bug] Anthropic endpoint: tool_reference parts in tool_result break chat templates without deferred-reference support (HTTP 500)

[jiaqianjing]

Describe the bug

The Anthropic-compatible endpoint (/v1/messages) converts tool_reference blocks found inside tool_result content into structured parts {"type": "tool_reference", "name": ...} (deferred tool loading, #22702). That part type is only understood by chat templates that implement deferred-reference expansion (GLM family). For any other model, the template hits the unknown item type and the whole request fails with HTTP 500:

jinja2.exceptions.TemplateError: Unexpected item type in content.
... [ts] Anthropic error response api_error (exception=ValueError): Internal server error

With Claude Code this is especially disruptive: once a session uses deferred tool loading (ToolSearch) and a tool_reference block enters the conversation history, every subsequent request in that session fails — the history carries the block forward, so retries cannot recover and only clearing/compacting the context fixes it. We observed ~26 consecutive 500s on a single session until it was abandoned.

Reproduction

Server: SGLang 0.5.17 (same code on main), any model whose chat template does not handle tool_reference (reproduced with the stock Qwen3.8-27B template):

curl -s http://127.0.0.1:30000/v1/messages \
  -H 'content-type: application/json' -H 'x-api-key: dummy' -d '{
    "model": "qwen3.8-27b",
    "max_tokens": 16,
    "stream": false,
    "tools": [{"name": "DemoTool", "description": "demo", "input_schema": {"type": "object", "properties": {}}}],
    "messages": [
      {"role": "user", "content": "list the deferred tools"},
      {"role": "assistant", "content": [
        {"type": "tool_use", "id": "toolu_1", "name": "ToolSearch", "input": {"query": "select:Foo"}}
      ]},
      {"role": "user", "content": [
        {"type": "tool_result", "tool_use_id": "toolu_1", "content": [
          {"type": "text", "text": "Found 1 tool:"},
          {"type": "tool_reference", "tool_name": "DemoTool"}
        ]}
      ]},
      {"role": "user", "content": "now use it"}
    ]
  }'

Actual: HTTP 500 {"type":"error","error":{"type":"api_error","message":"Internal server error"}} with TemplateError: Unexpected item type in content in the server log.
Same request with the tool_reference item removed: HTTP 200. (A thinking block in assistant history does not trigger this; the minimal trigger is the tool_reference item in a tool_result content list.)

Root cause

python/sglang/srt/entrypoints/anthropic/serving.py, _convert_to_chat_completion_request → _convert_tool_result_content:

elif item_type == "tool_reference":
    ref_name = item.get("tool_name") or item.get("name")
    if ref_name:
        tool_content_parts.append(
            {"type": "tool_reference", "name": ref_name}
        )

The structured part is emitted unconditionally. Templates without deferred-reference expansion (e.g. Qwen's render_content macro, which only accepts text/image/video items) raise TemplateError, which is surfaced as a 500.

Suggested fix: detect whether the loaded chat template handles tool_reference (e.g. substring check on the template source, mirroring how the unit tests fake tokenizer_manager.tokenizer.chat_template) and, when it does not, degrade the reference to a text part such as [tool reference: <name>] so history echoes keep working for all models while GLM behavior is preserved. A PR implementing this is coming.

Environment

Python: 3.12.12 | packaged by conda-forge
CUDA available: True
GPU: 8x NVIDIA H100 80GB HBM3 (CC 9.0), Driver 580.95.05, CUDA 12.9
PyTorch: 2.11.0+cu130
sglang: 0.5.17
sglang-kernel: 0.4.5
flashinfer_python: 0.6.15.post1
triton: 3.6.0
transformers: 5.12.1
numpy: 2.3.5
fastapi: 0.141.1
pydantic: 2.14.0b1
anthropic: 0.125.0

Full python3 -m sglang.check_env output:

Python: 3.12.12 | packaged by conda-forge | (main, Oct 22 2025, 23:25:55) [GCC 14.3.0]
CUDA available: True
GPU 0,1,2,3,4,5,6,7: NVIDIA H100 80GB HBM3
GPU 0,1,2,3,4,5,6,7 Compute Capability: 9.0
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 12.9, V12.9.86
CUDA Driver Version: 580.95.05
PyTorch: 2.11.0+cu130
sglang: 0.5.17
sglang-kernel: 0.4.5
flashinfer_python: 0.6.15.post1
flashinfer_cubin: Module Not Found
flashinfer_jit_cache: Module Not Found
triton: 3.6.0
transformers: 5.12.1
torchao: 0.17.0
numpy: 2.3.5
aiohttp: 3.14.3
fastapi: 0.141.1
huggingface_hub: 1.28.0
interegular: 0.3.3
modelscope: 1.39.1
orjson: 3.12.0
outlines: 0.1.11
packaging: 26.3
psutil: 7.2.2
pydantic: 2.14.0b1
python-multipart: 0.0.32
pyzmq: 27.1.0
uvicorn: 0.52.4
uvloop: 0.22.1
vllm: Module Not Found
xgrammar: 0.2.1
openai: 2.6.1
tiktoken: 0.14.0
anthropic: 0.125.0
litellm: Module Not Found
torchcodec: 0.11.1
NVIDIA Topology: 
	�[4mGPU0	GPU1	GPU2	GPU3	GPU4	GPU5	GPU6	GPU7	CPU Affinity	NUMA Affinity	GPU NUMA ID�[0m
GPU0	 X 	NV18	NV18	NV18	NV18	NV18	NV18	NV18	0-47,96-143	0		N/A
GPU1	NV18	 X 	NV18	NV18	NV18	NV18	NV18	NV18	0-47,96-143	0		N/A
GPU2	NV18	NV18	 X 	NV18	NV18	NV18	NV18	NV18	0-47,96-143	0		N/A
GPU3	NV18	NV18	NV18	 X 	NV18	NV18	NV18	NV18	0-47,96-143	0		N/A
GPU4	NV18	NV18	NV18	NV18	 X 	NV18	NV18	NV18	48-95,144-191	1		N/A
GPU5	NV18	NV18	NV18	NV18	NV18	 X 	NV18	NV18	48-95,144-191	1		N/A
GPU6	NV18	NV18	NV18	NV18	NV18	NV18	 X 	NV18	48-95,144-191	1		N/A
GPU7	NV18	NV18	NV18	NV18	NV18	NV18	NV18	 X 	48-95,144-191	1		N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

Hypervisor vendor:: KVM
ulimit soft: 1048576