Skip to content

[Bugfix][Quantization] Don't claim the GPU lacks FP4 support when the checkpoint is weight-only #53328

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
filipsajdak wants to merge 1 commit into
base: main
Choose a base branch
from
+17 −6
Open

[Bugfix][Quantization] Don't claim the GPU lacks FP4 support when the checkpoint is weight-only #53328

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 17 additions & 6 deletions vllm/model_executor/layers/quantization/utils/marlin_utils_fp4.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,9 @@
marlin_unpad_output,
should_use_atomic_add_reduce,
)
from vllm.model_executor.layers.quantization.utils.nvfp4_utils import (
cutlass_fp4_supported,
)
from vllm.platforms import current_platform
from vllm.scalar_type import scalar_types
from vllm.utils.math_utils import round_up
Expand Down Expand Up @@ -351,12 +354,20 @@ def prepare_nvfp4_moe_layer_for_marlin(
) -> tuple[
torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor, torch.Tensor
]:
logger.warning_once(
"Your GPU does not have native support for FP4 computation but "
"FP4 quantization is being used. Weight-only FP4 compression will "
"be used leveraging the Marlin kernel. This may degrade "
"performance for compute-heavy workloads."
)
if not cutlass_fp4_supported():
logger.warning_once(
"Your GPU does not have native support for FP4 computation but "
"FP4 quantization is being used. Weight-only FP4 compression will "
"be used leveraging the Marlin kernel. This may degrade "
"performance for compute-heavy workloads."
)
else:
logger.info_once(
"Using Marlin weight-only FP4 MoE kernels. This GPU supports native "
"FP4 computation, but the checkpoint is weight-only quantized (no FP4 "
"activation scales), so native FP4 GEMM kernels do not apply. A W4A4 "
"NVFP4 checkpoint would use the cutlass path on this device."
)

input_dtype = get_marlin_input_dtype(prefix="")
if input_dtype is not None and input_dtype.itemsize == 1:
Expand Down
Loading