Skip to content

[ROCm][CI] Stabilize MI355 FusedMoE test group - #53025

Merged
njhill merged 5 commits into
mainfrom
akaratza_fix_moe
Aug 21, 2026
Merged

[ROCm][CI] Stabilize MI355 FusedMoE test group#53025
njhill merged 5 commits into
mainfrom
akaratza_fix_moe

Conversation

@AndreasKaratzas

Copy link
Copy Markdown
Member

This PR addresses the three MI355 FusedMoE failures reported by AMD CI build 12250. The MI355 group was introduced by #41100 and exposed a B200-only DeepEP cleanup that #35077 had broadened to every platform, causing rocSHMEM destroy/reinitialize failures on MI355. The EPLB case separately failed after repeatedly registering NIXL memory while iterating the broad layer matrix. This change restores the CUDA SM100-only DeepEP cleanup and selects Gloo only for gfx950 EPLB in this test, while preserving the layer coverage and dedicated NIXL coverage.

  • DeepEP fix: Destroy the cached DeepEP all-to-all manager only on CUDA devices with compute capability 10, restoring the original B200 scope and preventing unsupported rocSHMEM reinitialization on MI355.
  • EPLB fix: Use torch_gloo only when this test has EPLB enabled on gfx950; no MI355 layer configurations are skipped, and dedicated EPLB execution tests retain NIXL coverage.
  • Repeated validation: The formerly failing allgather/EPLB node passed 10/10 independent runs on MI355, exercising all 96 internal configurations per run for 960/960 total passes.

Signed-off-by: Andreas Karatzas <akaratza@amd.com>
@AndreasKaratzas
AndreasKaratzas marked this pull request as ready for review August 20, 2026 00:51

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review for a one-time review, or @claude review always to subscribe this PR to a review on every future push.

Tip: disable this comment in your organization's Code Review settings.

@mergify mergify Bot added the rocm Related to AMD ROCm label Aug 20, 2026
@github-project-automation github-project-automation Bot moved this to Todo in AMD Aug 20, 2026
@AndreasKaratzas

Copy link
Copy Markdown
Member Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #84717 for commit 6926d6254455.

Signed-off-by: Andreas Karatzas <akaratza@amd.com>
@AndreasKaratzas

Copy link
Copy Markdown
Member Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #84754 for commit 841b5f8716dc.

@stefankoncarevic

stefankoncarevic commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Heads up: when the MI355 FusedMoE Layer group is run in CI, this branch still does not come back clean. Worth another look before merge.
https://buildkite.com/vllm/ci/builds/84754/list?jid=01a01ee2-ff54-4c6f-a14b-2b11dccecbfa&tab=output

Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
@AndreasKaratzas

Copy link
Copy Markdown
Member Author

/ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite CI #84875 for commit 0fe663c30ad1.

@AndreasKaratzas

Copy link
Copy Markdown
Member Author

/amd-ci run

@github-actions

Copy link
Copy Markdown

✅ Triggered Buildkite AMD CI #12291 for commit 0fe663c30ad1.

@AndreasKaratzas

Copy link
Copy Markdown
Member Author

the remaining flydsl failure is fixed in
#53024

@njhill
njhill merged commit 88eb946 into main Aug 21, 2026
60 checks passed
@njhill
njhill deleted the akaratza_fix_moe branch August 21, 2026 20:15
@github-project-automation github-project-automation Bot moved this from Todo to Done in AMD Aug 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/build rocm Related to AMD ROCm

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants