[Bug]: AriaForConditionalGeneration produces incoherent garbage output at tensor-parallel size > 1 (reproduces on CPU, no GPU/accelerator needed)

[wagnerpatriota]

Your current environment

The output of python collect_env.py

==============================
        System Info
==============================
OS                           : Ubuntu 22.04.5 LTS (x86_64)
GCC version                  : (Ubuntu 11.4.0-1ubuntu1~22.04.3) 11.4.0
Clang version                : Could not collect
CMake version                : version 3.22.1
Libc version                 : glibc-2.35

==============================
       PyTorch Info
==============================
PyTorch version              : 2.11.0+cu130
Is debug build               : False
CUDA used to build PyTorch   : 13.0
ROCM used to build PyTorch   : N/A
XPU used to build PyTorch    : N/A

==============================
      Python Environment
==============================
Python version               : 3.12.13 (main, Jun 29 2026, 16:03:16) [GCC 11.4.0] (64-bit runtime)
Python platform              : Linux-6.12.58-82.121.amzn2023.x86_64-x86_64-with-glibc2.35

==============================
       CUDA / GPU Info
==============================
Is CUDA available            : False
CUDA runtime version         : No CUDA
CUDA_MODULE_LOADING set to   : N/A
GPU models and configuration : No CUDA
Nvidia driver version        : No CUDA
cuDNN version                : No CUDA
HIP runtime version          : N/A
MIOpen runtime version       : N/A
Is XNNPACK available         : True

==============================
          CPU Info
==============================
Architecture:                            x86_64
CPU op-mode(s):                          32-bit, 64-bit
Address sizes:                           46 bits physical, 48 bits virtual
Byte Order:                              Little Endian
CPU(s):                                  192
On-line CPU(s) list:                     0-191
Vendor ID:                               GenuineIntel
Model name:                              Intel(R) Xeon(R) Platinum 8488C
CPU family:                              6
Model:                                   143
Thread(s) per core:                      2
Core(s) per socket:                      48
Socket(s):                               2
Stepping:                                8
BogoMIPS:                                4800.00
Flags:                                   fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology nonstop_tsc cpuid aperfmperf tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves avx_vnni avx512_bf16 wbnoinvd ida arat avx512vbmi umip pku ospke waitpkg avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq rdpid cldemote movdiri movdir64b md_clear serialize amx_bf16 avx512_fp16 amx_tile amx_int8 flush_l1d arch_capabilities
Hypervisor vendor:                       KVM
Virtualization type:                     full
L1d cache:                               4.5 MiB (96 instances)
L1i cache:                               3 MiB (96 instances)
L2 cache:                                192 MiB (96 instances)
L3 cache:                                210 MiB (2 instances)
NUMA node(s):                            2
NUMA node0 CPU(s):                       0-47,96-143
NUMA node1 CPU(s):                       48-95,144-191
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; Enhanced / Automatic IBRS; IBPB conditional; PBRSB-eIBRS SW sequence; BHI BHI_DIS_S
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

==============================
Versions of relevant libraries
==============================
[pip3] flashinfer-python==0.6.12
[pip3] mypy_extensions==1.1.0
[pip3] numpy==2.3.5
[pip3] nvidia-cublas==13.1.0.3
[pip3] nvidia-cuda-cccl==13.3.3.4.1
[pip3] nvidia-cuda-crt==13.3.73
[pip3] nvidia-cuda-cupti==13.0.85
[pip3] nvidia-cuda-nvcc==13.2.78
[pip3] nvidia-cuda-nvrtc==13.0.88
[pip3] nvidia-cuda-runtime==13.0.96
[pip3] nvidia-cuda-tileiras==13.2.78
[pip3] nvidia-cudnn-cu13==9.19.0.56
[pip3] nvidia-cudnn-frontend==1.26.0
[pip3] nvidia-cufft==12.0.0.61
[pip3] nvidia-cufile==1.15.1.6
[pip3] nvidia-curand==10.4.0.35
[pip3] nvidia-cusolver==12.0.4.66
[pip3] nvidia-cusparse==12.6.3.3
[pip3] nvidia-cusparselt-cu13==0.8.0
[pip3] nvidia-cutlass-dsl==4.5.2
[pip3] nvidia-cutlass-dsl-libs-base==4.5.2
[pip3] nvidia-cutlass-dsl-libs-cu13==4.5.2
[pip3] nvidia-ml-py==13.610.43
[pip3] nvidia-nccl-cu13==2.28.9
[pip3] nvidia-nvjitlink==13.0.88
[pip3] nvidia-nvshmem-cu13==3.4.5
[pip3] nvidia-nvtx==13.0.85
[pip3] nvidia-nvvm==13.2.78
[pip3] pyzmq==27.1.0
[pip3] sentence-transformers==5.6.0
[pip3] tokenspeed-triton==3.8.10.post20260709
[pip3] torch==2.11.0+cu130
[pip3] torch_c_dlpack_ext==0.1.5
[pip3] torch-neuronx==2.12.3.0.23413+341d13c.dev
[pip3] torchaudio==2.11.0+cu130
[pip3] torchdata==0.11.0
[pip3] torchtitan==0.2.2
[pip3] torchvision==0.26.0+cu130
[pip3] transformers==5.13.1
[pip3] transformers-stream-generator==0.0.5
[pip3] triton==3.6.0
[conda] Could not collect

==============================
         vLLM Info
==============================
ROCM Version                 : Could not collect
vLLM Version                 : 0.24.0
vLLM Build Flags:
  CUDA Archs: Not Set; ROCm: Disabled; XPU: Disabled
GPU Topology:
  Could not collect

==============================
     Environment Variables
==============================
TORCH_NEURONX_FALLBACK_ONLY_FOR_UNIMPLEMENTED_OPS=0
TORCH_NEURONX_DISABLE_FALLBACK_EXECUTION=1
TORCH_NEURONX_MLIR_ATEN_OPS=1
TORCH_NEURONX_RETAIN_DEVICE_MODE=0
TORCH_NEURONX_ENABLE_HOST_CC=0
TORCH_NEURONX_CPP_PROCESS_GROUP=1
TORCH_NEURONX_ENABLE_STACK_TRACE=0
TORCH_NEURONX_DEBUG_DIR=/tmp/neuron_backend
TORCH_NEURONX_PRESERVE_COMPILATION_ARTIFACTS=0
TORCH_NEURONX_ENABLE_LAZY_ALLOC=1
TORCHINDUCTOR_CACHE_DIR=/tmp/torchinductor_root
TORCH_NEURONX_HLO_CACHE=1
CUDA_VISIBLE_DEVICES=
CUDA_VISIBLE_DEVICES=
VLLM_USE_LAYERNAME=0
PYTORCH_NVML_BASED_CUDA_CHECK=1
TORCHINDUCTOR_COMPILE_THREADS=1

🐛 Describe the bug

Summary

rhymes-ai/Aria (arch AriaForConditionalGeneration, a 25.3B-param MoE VLM) serves and completes requests successfully at tensor_parallel_size > 1, but the generated text is genuinely incoherent noise, not merely low-quality or wrong-but-plausible output. This reproduces with VLLM_CPU_MODE-style pure-PyTorch execution (no GPU involved), confirming the bug is in vLLM's own AriaForConditionalGeneration/MoE combine logic, not a hardware backend issue. Coherence degrades proportionally with tensor_parallel_size: fully coherent at tp=1, partially coherent at tp=2, fully garbage at tp=4.

Reproduction

vllm serve rhymes-ai/Aria --tensor-parallel-size 4 --trust-remote-code=false --max-model-len 4096

curl -s http://localhost:8000/v1/completions -H "Content-Type: application/json" -d '{
  "model": "rhymes-ai/Aria",
  "prompt": "The capital of France is",
  "max_tokens": 30
}'

Observed output (tp=4): ". .\n\n of 1.\n\n , .\n\n ("

Expected: a coherent completion, e.g. " Paris."

This is a plain /v1/completions call with no chat template involved, ruling out any tokenizer/chat-template special-token issue as the cause.

Tensor-parallel-size sweep (same prompt, same checkpoint, only --tensor-parallel-size varies)

tensor_parallel_size	Output coherence
1	Coherent — e.g. " Paris.\n\n- The capital of Germany is Berlin.\n\n- The capital of Italy is"
2	Partially coherent — gets the right answer, then degrades, e.g. " Paris.这是一个非常好的城市。"
4	Fully garbage — e.g. "m.ofd.DPo Fr..,\n\n Pro林淼.pro.\n.\n\n"

Confirmed via VLLM_CPU_MODE-equivalent execution (pure PyTorch eager, no GPU, real gloo collectives for tensor_parallel_size > 1) — this rules out any GPU-specific numerics, kernel, or compiler issue as the cause. The degradation is purely a function of tensor_parallel_size, present identically with or without any accelerator involved.

Investigation and ruled-out hypotheses

1. Chat-template / special-token fragmentation. <|im_start|>/<|im_end|> are not registered in added_tokens_decoder, so they BPE-fragment into ~7 subword pieces each instead of being single tokens. This is real, but ruled out as the cause: plain /v1/completions (no chat template, no special tokens at all) produces the same garbage.

2. Weight-name mapping (hf_to_vllm_mapper). Built a standalone script loading the checkpoint's safetensors directly via vllm.model_executor.model_loader.weight_utils.safetensors_weights_iterator, then applying AriaForConditionalGeneration.hf_to_vllm_mapper.apply(). Confirmed all 794 checkpoint tensors map correctly onto the model's actual named_parameters() structure (language_model.layers.N.*, etc.). Ruled out.

3. AriaRoutedExperts.weight_loader's tensor-parallel shard/transpose/concat math (the custom MoE weight loader, vllm/model_executor/models/aria.py, as it exists on the v0.24.0 tag — this custom class has since been removed by a later weight-loading refactor, see "Note on current main" below, but the underlying computation it implemented is unchanged). Wrote a standalone numerical simulation reproducing the exact shard/transpose/concat sequence this loader performs, feeding one expert's weights through both the tensor_parallel_size=1 reference path and the sharded tensor_parallel_size>1 path (summing per-rank partial FFN outputs to mirror the real all-reduce). This matched exactly (torch.allclose true) for every tensor_parallel_size in {1, 2, 3, 4, 6} with moe_intermediate_size divisible by tensor_parallel_size. This means the weight-loading/sharding math itself is self-consistent — the bug is somewhere else in the tensor-parallel MoE combine path, not in how expert weights get sharded onto each rank.

4. Plugin-specific code path (only relevant for the AWS Neuron out-of-tree platform this investigation happened to be conducted from — included here only to establish this is not a platform-specific artifact). Reproduced the identical garbage under pure CPU/PyTorch execution with real gloo collectives, no accelerator-specific code involved at all. This rules out any hardware-backend-specific cause.

5. tensor_parallel_size=1 sanity check. Confirmed coherent at tp=1 on every execution mode tried. This confirms the bug requires tensor_parallel_size > 1 to manifest, and rules out a general model-loading or weight-conversion bug (which would presumably also break tp=1).

Suspected area (not pinpointed)

The bug is narrowed to somewhere in the tensor-parallel combine path for AriaTextMoELayer, most likely one of:

• How FusedMoE's runner combines per-rank routed-expert outputs across ranks (the all-reduce/gather step) when shared_experts= is also passed to the same FusedMoE instance, since Aria's MoE layer combines a shared-expert path and a routed-expert path per token (AriaTextMoELayer.forward → self.experts(hidden_states, router_output), where self.experts is a single FusedMoE(shared_experts=self.shared_experts, ...) instance).
• A possible renormalize convention mismatch: AriaTextMoELayer.__init__'s FusedMoE(...) call does not pass renormalize=False, so it defaults to True — but the reference HF implementation's AriaExperts.route_tokens_to_experts does not renormalize the top-k softmax routing weights. This mismatch exists at every tensor_parallel_size including 1, so it's unlikely to be the sole cause (tp=1 is empirically coherent), but it could compound with the real tp>1-specific bug once results are combined across ranks.

Confirmed this file's AriaTextMoELayer.forward() (return self.experts(hidden_states, router_output)) is byte-identical between the v0.24.0 tag and current main — so whatever the actual combine-path bug is, it predates v0.24.0 and is still present on main today.

Note on current main: the exact class name in this report has since been renamed/refactored, but the bug itself is unaffected

At the time of the investigation above, vllm/model_executor/models/aria.py's custom MoE weight loader was named AriaRoutedExperts. As of PR #48496 ("Remove even more unnecessary load_weights methods", merged 2026-07-18), this custom weight-loading class has been deleted entirely in favor of FusedMoE's own generic weight-loading via a renamed checkpoint mapping (experts.fc1 → experts.gate_up_proj, experts.fc2 → experts.down_proj). This refactor only changes how weights are loaded into FusedMoE, not the actual forward/combine computation — AriaTextMoELayer.forward() is unchanged, and item 3 above (the weight-sharding math) already confirmed the loading math itself was correct before this refactor, so this rename does not affect the bug described here. It is very likely still reproducible on current main.

Also worth noting: an unrelated earlier PR (#45054, "Fix weight loading issues caused by #41184") fixing a different Aria weight-loading crash includes the comment "I was unable to find an Aria model that would run" from its author — independent evidence that this model's tensor-parallel path is not well-covered by CI/manual testing upstream.

Searched for existing reports

Found one older, already-closed issue, #12241 ("[Bug]: The output of Aria model is not correct", closed 2025-01-22 via PR #12309). That issue was about a different bug — the vision tower producing all-zero output because Idefics3VisionTransformer's post_norm layer shouldn't be used for Aria — and was fixed by AriaVisionTransformer overriding post_layernorm with nn.Identity(), which is still present in the current model file. Notably, the final comment on that issue reads "Sadly, even after fixing the issue, the output is still nonsensical" — suggesting the tensor-parallel garbage-output bug described in this report may have been present even then, just never separately isolated or filed. No other open or closed issue found referencing AriaRoutedExperts, Aria + tensor-parallel, or Aria MoE garbage/incoherent output.

Environment

rhymes-ai/Aria, vLLM v0.24.0 and current main (combine-path code confirmed byte-identical between the two). Reproduces with tensor_parallel_size in {2, 4}; not reproducible at tensor_parallel_size=1. No GPU/accelerator required to reproduce — pure CPU/PyTorch eager execution with real collectives shows the same degradation.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[ArchanaChetan07]

Root cause found: the shared-experts output is all-reduced twice at TP>1.

AriaTextMoELayer builds its shared LlamaMLP without reduce_results=False, so the MLP's RowParallelLinear all-reduces its output — and then FusedMoE's runner reduces it again (the runner contract is that it performs the TP reduction itself, once: either on shared_output separately or on shared + fused combined, see the note in moe_runner.py). The shared-expert contribution therefore gets scaled by tp_size, which matches your symptom table exactly: identity at TP=1, 2× at TP=2 (partially coherent), 4× at TP=4 (garbage).

Every other in-tree model passing shared_experts into FusedMoE (deepseek_v2, glm4_moe, ernie45_moe, bailing_moe, hunyuan_v1, afmoe, exaone_moe, cohere2_moe) already constructs it with reduce_results=False — Aria was the single outlier.

This also fits your ruled-out list: it's not weight loading (your item 3), and it only manifests at TP>1 (your item 5), since reduce_results controls a collective that is the identity on one rank.

Opened #49366 with the one-line fix. Note the renormalize convention mismatch you flagged is real but separate (present at TP=1 too) — I left it out of the PR to keep it minimal.