[RFC][NPU] Ascend Cross-Generation Device Adaptation

[wangyao-i]

Summary

As Ascend A5 support is being added to SGLang, generation differences are no longer limited to kernel availability. They increasingly affect operator providers, signatures, tensor dtypes/layouts, KV representations, and quantization metadata.

If these differences continue to be handled directly in Attention, MoE, Quantization, Model, and Memory components:

if is_a5():
    ...
else:
    ...

generation-specific logic will keep leaking into feature code.

This RFC proposes a two-layer adaptation model:

Model / Attention / MoE / Quant / Memory
                    │
                    ▼
          SGLang DeviceOperator
       runtime-visible device contract
                    │
                    ▼
       sgl-kernel-npu Stable Operator
                    │
                    ▼
       target-specific provider
          Ascend910 / Ascend950
                    │
                    ▼
     torch_npu / ACLNN / AscendC / Triton

The core rule is:

Hardware differences that can be fully hidden behind a stable standalone operator contract should be handled in sgl-kernel-npu. Differences that change SGLang runtime-visible dtype, layout, metadata, or cross-operator state should be handled by SGLang DeviceOperator.

---

Motivation

Current A5 support already exposes two fundamentally different kinds of generation differences.

Kernel-local differences

For example, Gemma RMSNorm uses different implementations across generations:

A2/A3:
torch_npu.npu_gemma_rms_norm

A5:
torch_npu.npu_rms_norm(input, 1 + weight)

However, both can expose the same semantic API:

npu_gemma_rms_norm(input, weight, eps)

sgl-kernel-npu#638 demonstrates this model: provider selection happens while building a target-specific wheel, while SGLang only consumes a stable operator API and does not select providers at runtime.

Runtime-visible differences

Other generation differences cannot be hidden inside a standalone operator.

For example, an A5 KV/attention path may change:

INT8 KV       -> FP8 KV
scale format  -> different metadata representation
KV layout     -> packed representation
attention API -> different operator contract

These differences affect Memory Pool, Attention, and possibly other runtime components before a kernel is invoked.

Therefore, we need an explicit ownership boundary between kernel implementation differences and runtime device contract differences.

---

Goals

This RFC aims to:

• reduce scattered Ascend generation checks in SGLang;
• keep kernel/provider differences inside sgl-kernel-npu whenever possible;
• centralize runtime-visible generation contracts in SGLang DeviceOperator;
• preserve existing ownership of Attention, MoE, Memory, and Quantization components;
• provide a consistent extension model for future Ascend generations;
• avoid introducing a large device abstraction that wraps every NPU operator.

---

Ownership Model

The default decision rule is:

Generation difference
        │
        ▼
Can it be hidden behind one stable
standalone operator API?
        │
   ┌────┴────┐
   │ Yes     │ No
   ▼         ▼
sgl-kernel   Does runtime require different
-npu         dtype/layout/metadata/state?
                  │
             ┌────┴────┐
             │ Yes     │ No
             ▼         ▼
         SGLang      Feature /
        DeviceOp     capability

Typical ownership:

Difference	Owner
torch_npu / ACLNN / AscendC / Triton provider difference	sgl-kernel-npu
Same semantic operator with different kernel implementations	sgl-kernel-npu
Operator-local reshape/cast	sgl-kernel-npu
Runtime KV dtype/layout	SGLang DeviceOperator
Runtime metadata contract	SGLang DeviceOperator
Representation shared across multiple operators	SGLang DeviceOperator
Enable MXFP4 / FP8 KV	Feature / Config
KV allocation and lifetime	Memory Pool
DSA/MLA or TP/EP policy	Feature

An operator signature difference alone does not require SGLang DeviceOperator.

If the difference can be cleanly normalized behind a stable standalone API, it should remain in sgl-kernel-npu.

---

sgl-kernel-npu: Stable Operator + Target Provider

We do not propose another large abstraction such as:

class BaseDeviceOps:
    ...

class A5DeviceOps(BaseDeviceOps):
    ...

inside sgl-kernel-npu.

Instead, the kernel layer follows:

Stable Operator Contract + Target-specific Provider

For example:

        npu_gemma_rms_norm()
                  │
         ┌────────┴────────┐
         ▼                 ▼
 Ascend910 provider   Ascend950 provider
      native              ACLNN

SGLang only depends on:

npu_gemma_rms_norm(...)

and does not need to know which provider is used.

A stable operator should remain standalone:

Tensor / scalar / attributes
            │
            ▼
       Stable Operator
            │
            ▼
      Tensor / outputs

It should not depend on SGLang runtime objects such as:

ForwardBatch
MemoryPool
Scheduler
ModelRunner
AttentionBackend

---

Target-specific Provider Staging

sgl-kernel-npu #734 generalizes the Gemma-specific mechanism introduced in#638 into generic target-provider staging.

Its core principle is:

The build system knows targets, but does not know operators.

Providers are organized by target and stable module path:

target_providers/
├── Ascend910/
│   └── norm/
│       └── gemma_rmsnorm.py
│
└── Ascend950/
    └── norm/
        └── gemma_rmsnorm.py

During wheel build:

target_providers/Ascend950/norm/gemma_rmsnorm.py
                         │
                         ▼
              sgl_kernel_npu/
                  norm/
                  gemma_rmsnorm.py

The installed public API therefore remains:

sgl_kernel_npu.norm.gemma_rmsnorm

regardless of the target implementation.

Directory as Registry

There is intentionally no global operator registry such as:

PROVIDERS = {
    "norm.gemma_rmsnorm": ...,
    "quant.dynamic_mx": ...,
}

Instead, the directory layout itself acts as the registration mechanism.

Adding another portable operator only requires matching module paths:

target_providers/
├── Ascend910/
│   └── quant/dynamic_mx.py
└── Ascend950/
    └── quant/dynamic_mx.py

The generic build framework does not need to know whether the module implements Gemma RMSNorm, MXFP, MoE, or Attention.

#734 reduces provider staging to:

validate target
      ↓
scan provider tree
      ↓
stage provider modules

No operator-specific build logic is required.

---

Build-time, Not Runtime Dispatch

Provider selection should happen while building the target-specific wheel:

Concrete SoC
    │
    ▼
normalize provider target
    │
    ├── Ascend910
    └── Ascend950
            │
            ▼
    stage provider tree
            │
            ▼
     target-specific wheel

The runtime does not contain:

if is_a5():
    ...

or:

provider = registry[current_device]

There is also no silent runtime fallback.

Unknown targets, missing provider directories, and conflicts with common modules should fail during the build.

This keeps provider compatibility out of the inference hot path and makes target mismatches explicit.

---

Provider Contract

A target-specific provider is appropriate when all of the following are true:

same semantic operation
+
different target implementation
+
same logical input/output contract

For example:

Ascend910:
npu_gemma_rms_norm(input, weight, eps)

Ascend950:
npu_rms_norm(input, 1 + weight, eps)

Both expose:

npu_gemma_rms_norm(input, weight, eps)
    -> (output, rstd)

#734 also enforces that target provider trees expose the same module set, keeping target differences at the implementation layer rather than the public API layer.

---

SGLang DeviceOperator

When generation differences cross the standalone operator boundary, they should be handled in SGLang.

For example:

A3:
INT8 KV
FP16 scale
legacy sparse-attention contract

A5:
FP8 packed KV
different scale metadata
different sparse-attention contract

In this case, the Memory Pool must allocate a different representation before the kernel executes. A kernel provider alone cannot hide the difference.

The proposed responsibility is:

SGLang DeviceOperator
    │
    ├── runtime dtype contract
    ├── layout contract
    ├── metadata contract
    └── semantic operator adaptation
              │
              ▼
      sgl-kernel-npu stable ops

DeviceOperator should not own:

kernel provider selection
feature enable/disable
KV lifetime
algorithm selection

In short:

DeviceOperator defines how an already-selected feature runs on the current device generation. It does not decide whether the feature should be enabled.

---

Representative Cases

Gemma RMSNorm

different implementation
same standalone semantic contract

→ sgl-kernel-npu target provider

#638 + #734provide the reference implementation.

MXFP scale layout

If normalization is operator-local:

kernel output
    ↓
local reshape / normalization
    ↓
stable public representation

→ sgl-kernel-npu

If the representation is stored and consumed across multiple runtime components:

producer
   ↓
runtime state
   ↓
multiple consumers

→ SGLang DeviceOperator

KV / Sparse Attention

If generation differences affect:

KV dtype
physical KV layout
scale metadata
memory allocation contract

→ SGLang DeviceOperator

The concrete standalone kernels used by that path remain in:

sgl-kernel-npu

---

Rollout

The two repositories can evolve independently under the same ownership model.

sgl-kernel-npu

#638(https://github.com/sgl-project/sgl-kernel-npu/pull/638)
Gemma stable operator
        ↓
#734(https://github.com/sgl-project/sgl-kernel-npu/pull/734)
generic target-provider staging
        ↓
additional portable operators

After #734, adding a new cross-generation operator only requires:

1. confirm that it has the same semantic and logical I/O contract;
2. add the same module path under the required target_providers/<target>/ directories;
3. add contract tests.

No generic build-system changes should be required.

SGLang

Introduce a minimal DeviceOperator and migrate only existing runtime-visible differences, such as:

KV representation
Sparse Attention contract
Indexer dtype/layout
cross-operator quantization metadata

This RFC does not propose wrapping every NPU operator.

---

Acceptance Criteria

The intended long-term rules are:

1. SGLang feature code does not directly select kernel providers.
2. Standalone generation differences are handled in sgl-kernel-npu whenever possible.
3. sgl-kernel-npu exposes stable semantic/module contracts.
4. Target-specific providers are selected during wheel build.
5. The build framework knows targets, not individual operators.
6. Provider dispatch is kept out of the inference hot path.
7. SGLang DeviceOperator only handles runtime-visible device contracts.
8. Feature, Memory, and Scheduler ownership remains unchanged.
9. Neither repository introduces a God Object covering all NPU operators.

---

Open Questions

1. Do we agree on the following default ownership rule?

standalone kernel/provider difference
        → sgl-kernel-npu

runtime-visible representation/contract difference
        → SGLang DeviceOperator

2. Which existing runtime-visible A5 difference should be the first minimal SGLang DeviceOperator migration to validate the abstraction?

---

References

• sgl-kernel-npu #638 — Add portable Gemma RMSNorm API
• sgl-kernel-npu #734 — Generalize target-specific operator provider staging
• SGLang #32745 — Fix Qwen3.5 GemmaRMSNorm on Ascend 950
• SGLang #33030 — Add Ascend 950 backend paths for DeepSeek-V4
• vLLM-Ascend device_op.py — prior art for runtime device adaptation