[Online quantization] Support activation quant key override by fxmarty-amd · Pull Request #51808 · vllm-project/vllm

fxmarty-amd · 2026-08-11T11:08:20Z

Purpose

Support --quantization online --quantization-config.moe.activation xyz API.

Currently, activation override for online quantization raises a NotImplementedError, but overriding the activation quant key is truly needed to dispatch on preferred MOE backends (e.g. AITER_MXFP4_BF16, AITER_MXFP4_MXFP4, AITER_MXFP4_FP8) that can not be selected in any other way.

More context at #52167

Test Plan

pytest tests/quantization/test_quantization_config_args.py -s -vvvvv
pytest tests/quantization/test_online.py -s -vvvvv

Test Result

Passing on MI350

mergify · 2026-08-11T15:55:26Z

Documentation preview: https://vllm--51808.org.readthedocs.build/en/51808/

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

BowenBao

Nice. Left some questions / comments. Overall looks good

BowenBao · 2026-08-17T17:35:50Z

+checkpoint-specific; today this is wired up for MXFP4 MoE checkpoints
+(gpt-oss) where you can opt into FP8 activations:


Probably offtopic, does this only support gpt-oss for now? if not let's extend the description here with other mxfp4 models (kimi k3, deepseek v4) and other quantizers (quark, etc)

Updated, this should read as "weight quantization dtype specific" (weight quant key), this previous doc is outdated.

Note however that some linear/MOE oracles do NOT support activation quant key, see e.g.

vllm/vllm/model_executor/layers/fused_moe/oracle/mxfp8.py

Line 107 in 402547d

def select_mxfp8_moe_backend(

Fixed in 7846c20

BowenBao · 2026-08-17T17:38:55Z

+) -> Mxfp8LinearKernel:
    """Select and instantiate the best MXFP8 linear kernel for the
    current platform."""
+    if activation_quant_key not in (None, kMxfp8Dynamic):


Is it sound to use None as no change to default? Not sure if other places use None to represent BF16.

also how is the default activation quant key for each quant method documented / decided?

This is a good point. Historically, None represents "no quantization"/BF16, see:

vllm/vllm/config/quantization.py

Lines 25 to 36 in 75dde08

# User-facing names addressable from quantization_config.

QUANT_KEY_NAMES: dict[str, QuantKey] = {

"fp8_per_tensor_static": kFp8StaticTensorSym,

"fp8_per_tensor_dynamic": kFp8DynamicTensorSym,

"fp8_per_token": kFp8DynamicTokenSym,

"fp8_per_channel_static": kFp8StaticChannelSym,

"fp8_per_block_static": kFp8Static128BlockSym,

"fp8_per_block_dynamic": kFp8Dynamic128Sym,

"mxfp8": kMxfp8Dynamic,

"mxfp4": kMxfp4Dynamic,

"int8_per_channel_static": kInt8StaticChannelSym,

}

However we do need to distinguish between a default None (currently on main):

vllm/vllm/config/quantization.py

Lines 65 to 73 in 75dde08

@config

class QuantSpec:

"""Quantization spec for one layer kind (linear or MoE).

`None` on either side means the method class falls back to its own default

(typically inherited from the checkpoint, or unquantized for online).

"""

weight: QuantKeyField = None

and user-set --quantization-config {"linear": {"activation": "null"}}, this is why weight: QuantKeyField = _UNSET, activation: QuantKeyField = _UNSET is introduced in this PR, and used in https://github.com/fxmarty-amd/vllm/blob/7846c2061c808d9757b550d3d2358ff9694fb40e/vllm/config/quantization.py#L97-L110

any reason not to create a dedicated enum for unset?

mergify · 2026-08-21T15:26:27Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @fxmarty-amd.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Signed-off-by: Felix Marty <Felix.Marty@amd.com>

github-actions · 2026-08-21T20:08:00Z

✅ @fxmarty-amd, CI is now available for this PR.

/ci run starts upstream CI; /amd-ci run starts AMD CI only.
/ci retry retries failed jobs in the CI build for the current PR head. If the current head has no CI build, it starts a new CI build for the current head containing only jobs that failed in the latest earlier CI build for this PR.
/amd-ci retry retries failed jobs in AMD CI for the current PR head. Use /amd-ci run when the current head has no AMD CI build.
/ci cancel cancels scheduled or running CI builds for this PR branch; /amd-ci cancel does the same for AMD CI only.

BowenBao

LG overall. I have some small comments and a question about UNSET and default.

BowenBao · 2026-08-21T22:28:37Z

+) -> Mxfp8LinearKernel:
    """Select and instantiate the best MXFP8 linear kernel for the
    current platform."""
+    if activation_quant_key not in (None, kMxfp8Dynamic):


any reason not to create a dedicated enum for unset?

BowenBao · 2026-08-21T22:33:20Z

merge issue? unrelated changes and many other files with log2_lse_to_ln function

BowenBao · 2026-08-21T22:56:25Z

-        weight_key: "QuantKey | None" = None,
-        activation_key: "QuantKey | None" = None,
+        activation_key: "QuantKey | None",
+        weight_key: "QuantKey",


why order swap?

mergify · 2026-08-22T02:02:18Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @fxmarty-amd.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

mergify Bot added the quantization label Aug 11, 2026

mergify Bot added the documentation Improvements or additions to documentation label Aug 11, 2026

fxmarty-amd marked this pull request as ready for review August 11, 2026 15:56

fxmarty-amd requested review from AndreasKaratzas, ProExpertProg, WoosukKwon, houseroad, mgoin, pavanimajety, robertgshaw2-redhat, tlrmchlsmth, yewentao256, youkaichao and zyongye as code owners August 11, 2026 15:56

claude Bot reviewed Aug 11, 2026

View reviewed changes

fxmarty-amd mentioned this pull request Aug 13, 2026

[RFC]: Extended online quantization roadmap #52167

Open

1 task

BowenBao reviewed Aug 17, 2026

View reviewed changes

mergify Bot added the needs-rebase label Aug 21, 2026

allow act quant key override in online quantization

6e18cbc

Signed-off-by: Felix Marty <Felix.Marty@amd.com>

fxmarty-amd force-pushed the relax-online-quant-act-key-override branch from 2531578 to 6e18cbc Compare August 21, 2026 15:35

fxmarty-amd requested review from LucasWilkinson, MatthewBonanni and vadiklyutiy as code owners August 21, 2026 15:35

mergify Bot added the nvidia label Aug 21, 2026

github-project-automation Bot added this to NVIDIA Aug 21, 2026

AndreasKaratzas added the ready ONLY add when PR is ready to merge/full CI is needed label Aug 21, 2026

mergify Bot removed the needs-rebase label Aug 21, 2026

BowenBao reviewed Aug 21, 2026

View reviewed changes

mergify Bot added the needs-rebase label Aug 22, 2026

		checkpoint-specific; today this is wired up for MXFP4 MoE checkpoints
		(gpt-oss) where you can opt into FP8 activations:

	# User-facing names addressable from quantization_config.
	QUANT_KEY_NAMES: dict[str, QuantKey] = {
	"fp8_per_tensor_static": kFp8StaticTensorSym,
	"fp8_per_tensor_dynamic": kFp8DynamicTensorSym,
	"fp8_per_token": kFp8DynamicTokenSym,
	"fp8_per_channel_static": kFp8StaticChannelSym,
	"fp8_per_block_static": kFp8Static128BlockSym,
	"fp8_per_block_dynamic": kFp8Dynamic128Sym,
	"mxfp8": kMxfp8Dynamic,
	"mxfp4": kMxfp4Dynamic,
	"int8_per_channel_static": kInt8StaticChannelSym,
	}

	@config
	class QuantSpec:
	"""Quantization spec for one layer kind (linear or MoE).

	`None` on either side means the method class falls back to its own default
	(typically inherited from the checkpoint, or unquantized for online).
	"""

	weight: QuantKeyField = None

Uh oh!

Uh oh!

Conversation

fxmarty-amd commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Uh oh!

mergify Bot commented Aug 11, 2026

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

BowenBao left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

fxmarty-amd Aug 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mergify Bot commented Aug 21, 2026

Uh oh!

github-actions Bot commented Aug 21, 2026

Uh oh!

BowenBao left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mergify Bot commented Aug 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

fxmarty-amd commented Aug 11, 2026 •

edited

Loading

fxmarty-amd Aug 17, 2026 •

edited

Loading