[Do not merge!] [Build] Migrate vendored DeepGEMM from pybind to TORCH_LIBRARY (abi3) by cleonard530 · Pull Request #48962 · vllm-project/vllm

cleonard530 · 2026-07-17T16:19:47Z

Purpose

This PR continues the libtorch stable ABI migration (see #26946) for vLLM. Here, we are not migrating deep_gemm to the stable abi yet, but are moving it off of pybind and onto TORCH_LIBRARY. Once we make these updates, we will open a seperate PR to make deep_gemm torch ABI stable. This points to the DeepGEMM updates in the PR deepseek-ai/DeepGEMM#393

Pin DeepGEMM to cleonard530/DeepGEMM (migrate_pybind_to_torch_library) so vLLM vendors the TORCH_LIBRARY + abi3 layout instead of pybind11. If this works, we will need to discuss how we want to mergecleonard530/DeepGEMM. vLLM has a fork of DeepGEMM, but currently the build still points towards upstream.
Replace the per-Python g++ build of _C.cpython-*.so with a single _C_extension.abi3.so built via DeepGEMM’s setup.py, plus deep_gemm/_C.py as the torch.ops.deep_gemm shim.
Update cmake / wheel checks / install script for the new artifacts (_C.py + _C_extension*.so) and drop DEEPGEMM_PYTHON_INTERPRETERS / TARGET_PY.
This closes the per-interpreter DeepGEMM binding path tracked after the failed single-abi3 attempt ([Build] Make bundled DeepGEMM wheel portable across Python versions #41476 / Revert "[Build] Make bundled DeepGEMM wheel portable across Python versions" #41512) and implements the TORCH_LIBRARY direction noted in [Build] DeepGEMM wheel integration: planned cleanups #42431.

@janeyx99 @Harry-Chen

Test Plan

  python tools/check_wheel_deepgemm.py
  pytest -v -s tests/kernels/quantization/test_block_fp8.py
  pytest -v -s tests/kernels/moe/test_deepgemm.py
  pytest -v -s tests/kernels/moe/test_batched_deepgemm.py
  pytest -v -s tests/kernels/attention/test_deepgemm_attention.py

Test Result

tools/check_wheel_deepgemm.py - PASS
tests/kernels/quantization/test_block_fp8.py - 100 failed, 322 passed, all failures are fatal error: sanitizer/asan_interface.h: No such file or directory. Not related to TORCH_LIBRARY migration
tests/kernels/moe/test_deepgemm.py - PASS
tests/kernels/moe/test_batched_deepgemm.py - PASS
tests/kernels/attention/test_deepgemm_attention.py - PASS

Essential Elements of an Effective PR Description Checklist

The purpose of the PR, such as "Fix some issue (link existing issues this PR will resolve)".
The test plan, such as providing test command.
The test results, such as pasting the results comparison before and after, or e2e results
(Optional) The necessary documentation update, such as updating supported_models.md and examples for a new model.

Migration progress using the Audit Python extension torch-abi-audit:
Note, deep_gemm used to have uses-private-api instead of abi3-ok

 -- extensions --
    [STABLE  ] [abi3-ok               ] _C_stable_libtorch.abi3.so  (stable_shim=87, unstable=0)
    [STABLE  ] [abi3-ok               ] _flashmla_C.abi3.so  (stable_shim=65, unstable=0)
    [STABLE  ] [abi3-ok               ] _flashmla_extension_C.abi3.so  (stable_shim=63, unstable=0)
    [STABLE  ] [abi3-ok               ] _moe_C_stable_libtorch.abi3.so  (stable_shim=72, unstable=0)
    [NO-TORCH] [abi3-ok               ] cumem_allocator.abi3.so
    [NO-TORCH] [abi3-ok               ] fs_io_C.abi3.so
    [NO-TORCH] [abi3-ok               ] spinloop.abi3.so
--> [UNSTABLE] [abi3-ok               ] third_party/deep_gemm/_C_extension.abi3.so  (stable_shim=0, unstable=71)
    [UNSTABLE] [abi3-ok               ] vllm_flash_attn/_vllm_fa2_C.abi3.so  (stable_shim=0, unstable=85)
    [UNSTABLE] [abi3-ok               ] vllm_flash_attn/_vllm_fa3_C.abi3.so  (stable_shim=0, unstable=81)

claude

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

cleonard530 · 2026-07-17T16:34:25Z

-    set(_dg_pythons "${Python_EXECUTABLE}")
-  endif()
-  message(STATUS "DeepGEMM _C will be built for: ${_dg_pythons}")
+  message(STATUS "DeepGEMM extension will be built with: ${Python_EXECUTABLE}")


This used to link private CPython symbols, so each python needed it's own C.cpython-3XY-….so.

Now, it builds against the CPython Limited/Stable ABI (Py_LIMITED_API), so one _C_extension.abi3.so works across supported CPythons.

cleonard530 · 2026-07-17T16:44:07Z

-      FILES_MATCHING PATTERN "_C.cpython-*.so")
-  endforeach()
-  add_custom_target(_deep_gemm_C ALL DEPENDS ${_dg_markers})
+  set(_dg_dir "${CMAKE_CURRENT_BINARY_DIR}/deepgemm_C")


This block replaced the per-Python build loop with a single abi3 build

cleonard530 · 2026-07-17T17:08:17Z

+env.pop("DG_SKIP_CUDA_BUILD", None)
+
+print(f"[build_deepgemm_C] building in {src} with {sys.executable}", flush=True)
+subprocess.check_call(


Use deep_gemm's setup.py to build deep_gemm

This call also gets Python_EXECUTABLE so building DeepGEMM's abi3 extension always tracks whatever Python vLLM is currently being built for.

cleonard530 · 2026-07-17T17:21:03Z

-required = required_pythons()
-missing = [v for v in required if v not in found]
-print(f"deepgemm _C: found {sorted(found)}, required {required}, missing {missing}")
-sys.exit(1 if missing else 0)


This file used to check each required python version in pyproject.toml and failed if matching *.so python version weren't there.

cleonard530 · 2026-07-17T17:22:24Z

+        f"deepgemm vendored binding: shim={shim.is_file()}, "
+        f"extensions={[p.name for p in so_files]}"
+    )
+    if missing:


Now, it checks vllm.third_party.deep_gemm to make sure the _C.py shim and _C_extension*.so are there.

Harry-Chen · 2026-07-18T12:43:39Z

Marked as ready to automatically trigger CI run.

cleonard530 · 2026-07-20T13:22:49Z

@Harry-Chen, the failure for these three test comes from this check
DG_DEVICE_ASSERT((values[j] & 0x807fffffu) == 0);

which was added between the previously tagged commit and my commit, but not directly due to the migration. We will need to figure out how to handle this before anything is merged (I don't think it will be difficult to fix), but I am planning on putting that off until we decide if we will be using upstream or the vllm-project/DeepGEMM, which doesn't have the check.

Harry-Chen · 2026-07-20T15:28:31Z

@Harry-Chen, the failure for these three test comes from this check DG_DEVICE_ASSERT((values[j] & 0x807fffffu) == 0);

which was added between the previously tagged commit and my commit, but not directly due to the migration. We will need to figure out how to handle this before anything is merged (I don't think it will be difficult to fix), but I am planning on putting that off until we decide if we will be using upstream or the vllm-project/DeepGEMM, which doesn't have the check.

I see. I think currently we are using https://github.com/deepseek-ai/DeepGEMM/tree/nv_dev, which adds sm120 support (from NVIDIA) on top of upstream master.

cleonard530 · 2026-07-20T16:52:12Z

I see. I think currently we are using https://github.com/deepseek-ai/DeepGEMM/tree/nv_dev, which adds sm120 support (from NVIDIA) on top of upstream master.

Oh yes, you're right @Harry-Chen. Once we agree we are in a good spot, should we try to merge into this branch? Do you know how receptive the maintainers for this repo will be to our migration efforts?

CC @janeyx99

Harry-Chen · 2026-07-23T02:50:37Z

I see. I think currently we are using https://github.com/deepseek-ai/DeepGEMM/tree/nv_dev, which adds sm120 support (from NVIDIA) on top of upstream master.

Oh yes, you're right @Harry-Chen. Once we agree we are in a good spot, should we try to merge into this branch? Do you know how receptive the maintainers for this repo will be to our migration efforts?

I have an issue there: deepseek-ai/DeepGEMM#333. But given the situation of the repo, I do not think we should put too much hope in upstream PRs. But since NVIDIA has their own branch on master, I do not know whether maintaining a fork would be a good idea.

CC @janeyx99

cleonard530 · 2026-07-24T14:07:16Z

@@ -234,7 +224,8 @@ def _deepgemm_fp8_gemm_nt_warmup(

    device = w.device
    a1q = torch.empty((max_tokens, k), device=device, dtype=torch.float8_e4m3fn)
-    a1q_scales = torch.empty(
+    # Must be a power of two (UE8M0 packing asserts zero sign/mantissa bits).
+    a1q_scales = torch.ones(


Not related to the TORCH_LIBRARY migration — a separate pre-existing bug. torch.empty left a1q_scales uninitialized, occasionally tripping DeepGEMM's UE8M0 assert (scales must be exact powers of two). Switched to torch.ones; warmup output is discarded, so this has no effect on real inference.

cleonard530 · 2026-07-28T20:25:04Z

@Harry-Chen, I opened a PR upstream on the nv_dev branch, deepseek-ai/DeepGEMM#393. Please let me know if you think this is the right move and if you know anyone who might be able to review it to help get it through. If there is no response there, then the only other option I can think of is using the https://github.com/vllm-project/DeepGEMM fork, which I know we don't want to have to maintain.

mergify · 2026-07-30T12:59:25Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @cleonard530.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Signed-off-by: Chris Leonard <chleonar@redhat.com>

…py only needs SRC and OUT; update the cmake call to match. Signed-off-by: Chris Leonard <chleonar@redhat.com>

Signed-off-by: Chris Leonard <chleonar@redhat.com>

…epgemm.sh git tag Signed-off-by: Chris Leonard <chleonar@redhat.com>

Signed-off-by: Chris Leonard <chleonar@redhat.com>

… parameter to a mutable parameter Signed-off-by: Chris Leonard <chleonar@redhat.com>

torch.empty leaves a1q_scales as garbage, which fails DeepGEMM's UE8M0 assert that scales be exact powers of two. Use torch.ones instead; warmup output is discarded, so this is a no-op for real inference. Unrelated to the TORCH_LIBRARY migration — a pre-existing bug in the branch beyond the last pinned git tag. Signed-off-by: Chris Leonard <chleonar@redhat.com>

mergify · 2026-08-04T20:30:26Z

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @cleonard530.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

mergify Bot added the ci/build label Jul 17, 2026

cleonard530 marked this pull request as ready for review July 17, 2026 16:24

cleonard530 requested review from Harry-Chen, LucasWilkinson and tlrmchlsmth as code owners July 17, 2026 16:24

claude Bot reviewed Jul 17, 2026

View reviewed changes

cleonard530 commented Jul 17, 2026

View reviewed changes

Harry-Chen added the ready ONLY add when PR is ready to merge/full CI is needed label Jul 18, 2026

cleonard530 force-pushed the deep_gemm_migration_to_torch_library branch from 255e87e to bd62abb Compare July 24, 2026 14:05

cleonard530 commented Jul 24, 2026

View reviewed changes

cleonard530 requested review from mgoin, pavanimajety and zyongye as code owners July 24, 2026 16:39

cleonard530 force-pushed the deep_gemm_migration_to_torch_library branch 2 times, most recently from 1c2538e to e98f4cf Compare July 24, 2026 17:56

cleonard530 mentioned this pull request Jul 27, 2026

Migrate pybind11 to TORCH_LIBRARY deepseek-ai/DeepGEMM#393

Open

mergify Bot added the needs-rebase label Jul 30, 2026

cleonard530 added 2 commits July 31, 2026 18:21

using torch_library branch of deep_gemm instead of pybind

0e9a056

Signed-off-by: Chris Leonard <chleonar@redhat.com>

The TORCH_LIBRARY build produces a single abi3 , so build_deepgemm_C.…

731f517

…py only needs SRC and OUT; update the cmake call to match. Signed-off-by: Chris Leonard <chleonar@redhat.com>

cleonard530 added 6 commits July 31, 2026 18:21

update git_tag for deep_gemm

d512c90

Signed-off-by: Chris Leonard <chleonar@redhat.com>

fix pre-commit linting issues

f794bc6

Signed-off-by: Chris Leonard <chleonar@redhat.com>

removed unnecessary flag from build and resynced the tools/install_de…

90708f5

…epgemm.sh git tag Signed-off-by: Chris Leonard <chleonar@redhat.com>

Bump pinned DeepGEMM commit git_tag

8f35d85

Signed-off-by: Chris Leonard <chleonar@redhat.com>

update git commit tag. changed the cumulative_local_expert_recv_stats…

0bdc235

… parameter to a mutable parameter Signed-off-by: Chris Leonard <chleonar@redhat.com>

cleonard530 force-pushed the deep_gemm_migration_to_torch_library branch from e98f4cf to 2e77b11 Compare July 31, 2026 18:29

mergify Bot removed the needs-rebase label Jul 31, 2026

mergify Bot added the needs-rebase label Aug 4, 2026

BabyDrangoner mentioned this pull request Aug 7, 2026

[Test] Use UE8M0 weight scales in block-FP8 DeepGEMM matmul test when the E8M0 oracle is active #51372

Closed

5 tasks

Uh oh!

Uh oh!

Conversation

cleonard530 commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Test Plan

Test Result

Uh oh!

claude Bot left a comment

Choose a reason for hiding this comment

Claude Code Review

Uh oh!

cleonard530 Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

cleonard530 Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

cleonard530 Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

cleonard530 Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

cleonard530 Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

cleonard530 Jul 17, 2026

Choose a reason for hiding this comment

Uh oh!

Harry-Chen commented Jul 18, 2026

Uh oh!

cleonard530 commented Jul 20, 2026

Uh oh!

Harry-Chen commented Jul 20, 2026

Uh oh!

cleonard530 commented Jul 20, 2026

Uh oh!

Harry-Chen commented Jul 23, 2026

Uh oh!

cleonard530 Jul 24, 2026

Choose a reason for hiding this comment

Uh oh!

cleonard530 commented Jul 28, 2026

Uh oh!

mergify Bot commented Jul 30, 2026

Uh oh!

mergify Bot commented Aug 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

cleonard530 commented Jul 17, 2026 •

edited

Loading