Skip to content

todo.md

Latest commit

 

History

History
43 lines (27 loc) · 10.5 KB

todo.md

File metadata and controls

43 lines (27 loc) · 10.5 KB
Raw

TODO

Also see stabs

Grouped by the hardware a task needs, since that is usually what blocks it. The Parked networking items - 2026-08-02 group was dissolved into these sections on 2026-08-04.

No hardware needed

  • optional: give model 2 in the hierarchical-arithmetic list the same general-form treatment models 1 and 3 got. Its 2*(32-1)/32 * 4GiB now reads through the shared P/g/k/n symbols defined just above it, so this is cosmetic.

  • survey the drop-in replacements for the NCCL/RCCL collectives layer and write them up. The book currently treats NCCL as the only option: NCCL_NET_PLUGIN, UCCL, DeepEP, MSCCL/MSCCL++ and NVSHMEM appear nowhere in it. The two that prompted this: UCCL's UCCL-collective is a genuine drop-in - you point NCCL_NET_PLUGIN at a path the package prints, with no application change - and it rearchitects the transport in software (packet spraying over 256 paths, latency-based and receiver-driven congestion control, selective-repeat loss recovery), which is also why its wins should be largest on legacy/cloud NICs rather than on a well-tuned IB fabric; and DeepEP covers the MoE expert-parallel dispatch/combine path that the collectives sections never touch, with UCCL-EP re-implementing it portably on AMD and on EFA/Broadcom NICs. Research what else belongs before writing, so this lands as a map rather than two links - MSCCL/MSCCL++, NVSHMEM and IBGDA, aws-ofi-nccl (already referenced in the 2-node group below), Gloo on the CPU path. The caveat that decides the section's shape: every speedup published here is the project's own - UCCL claims up to 2.5x on all-reduce across six HGX H100 nodes with 8x400G CX-7 RoCE, and up to 3.7x on two AWS g4dn.8xlarge - and this book's rule is measured, not marketing. So either those get attributed explicitly as upstream claims, or this item moves to the 2-node group and one of them gets reproduced: the g4dn case is the cheap one and would at least prove the plugin path end to end, while the RoCE figure needs 2+ RDMA nodes. Absent hardware, adoption is the strongest evidence available - NVIDIA NeMo integrates UCCL-EP, NVIDIA NIXL takes UCCL-P2P as an RDMA backend, Red Hat/IBM/Google's llm-d uses it for KV-cache transfer, AMD Primus uses UCCL-EP, and AMD TheRock ships UCCL-Tran/EP/P2P.

    • read this before booking nodes: UCCL's own README says that on p5/p5e/p5en/p6 "the official aws-ofi-nccl NCCL plugin with proper env variables already makes NCCL perform excellent", and its EFA collective support is currently limited to p4d.24xlarge. The hardware this project actually gets - p5en.48xlarge, p6-b200.48xlarge - is therefore precisely the case where upstream expects no win, so a null result here would say nothing about the library. To reproduce a speedup you need either legacy/non-RDMA NICs (their AFXDP path covers AWS ENA and IBM VirtIO, which is where the 3.7x g4dn figure comes from) or a Broadcom/CX-7 RoCE fabric. Plan the claim around that or the measurement is wasted node time.
    • three separate components, do not conflate them when writing: UCCL-collective (a.k.a. UCCL-Tran) is the NCCL/RCCL drop-in; UCCL-P2P is initiator/target transfer with NIXL-style APIs, aimed at KV-cache and RL weight transfer on 800Gbps NICs; UCCL-EP is the DeepEP-compatible expert-parallel path. Only the first is relevant to the collectives chapters; the second belongs to inference/KV-cache material and the third to MoE.
    • enabling it is env-var only, which is what makes it a genuine drop-in and worth showing verbatim: NCCL_NET_PLUGIN=$(python -c "import uccl; print(uccl.nccl_plugin_path())") for NCCL over IB/RoCE, uccl.rccl_plugin_path() for RCCL, and on EFA p4d both LD_PRELOAD=$(python -c "import uccl; print(uccl.efa_nccl_path())") and NCCL_NET_PLUGIN=$(python -c "import uccl; print(uccl.efa_plugin_path())"). Build is bash build.sh [cu12|cu13|roc7|roc6|therock] [all|ccl_rdma|ccl_efa|p2p|ep] [py_version] --install, where cu12 means CUDA 12.8 and cu13 means 13.0, roc7 means ROCm 7.1 and roc6 means 6.4.
    • primary sources for citation, both USENIX OSDI 2026, so the section can rest on papers rather than a README: "UCCL-Tran: An Extensible Software Transport Layer for GPU Networking" and "UCCL-EP: Portable Expert-Parallel Communication" (UC Berkeley Sky Computing + UC Davis ArtSy; Apache-2.0).
    • maturity check before recommending anything: the roadmap still lists "re-architecting NCCL to unleash network hardware performance", SM-efficient communication kernels, and fine-grained compute/communication overlap as in progress, and the consumer-GPU work (4090/5090/GB10) as in progress too. So today's honest framing is "a drop-in transport replacement that helps on constrained NICs", not "a faster NCCL in general".

1 node, 8x accelerators

  • refresh the illustrative busbw table in network/benchmarks/README.md under ### all_reduce benchmark, whose top figure is 91.69GBps from an earlier cluster. Left alone on 2026-08-04 because it does illustrate the output format correctly and the 4-node plot beside it matches - so this is a "is a current example better than an old one" call, not a correctness fix. A current 8x H200 plot and its environment now sit directly beneath it if you want to swap the table too.

  • reference notes for any future attempt to force a collective onto the NIC path, which is harder than it looks: NCCL_P2P_DISABLE=1 alone does not do it, because NCCL falls back P2P -> SHM -> network, so NCCL_SHM_DISABLE=1 is needed as well, and even then libfabric's EFA provider serves intra-node traffic from the instance's shared memory unless FI_EFA_ENABLE_SHM_TRANSFER=0. Also confirm GPUDirect RDMA is actually active, since NCCL disables it when the accelerator-to-NIC distance exceeds its threshold and then stages through host RAM, and on a virtualized instance ACS cannot be turned off and redirects PCIe peer-to-peer traffic through the CPU root complex unless the adapter has ATS enabled - each of these changes what the measurement means.

2 nodes

All four items here were done on 2026-08-07 on a 4-node 8x H200 p5en.48xlarge allocation, and the section is kept only to record what was answered:

  • which algorithm a multi-node all-reduce selects - Ring at 4 nodes, confirmed by forcing rather than by reading a log enum: NCCL_ALGO=allreduce:ring gave 364.65GBps busbw against the default's 364.87, while allreduce:nvlstree was available but 15% slower at 310.07. This closed review item 1 and opened item 73, because the flat-ring model the chapter rejects turns out to fit its own measurements best once its one-NIC-per-hop premise is corrected.
  • NVLSTree at two nodes - it is selected there (forced 463.29 against default 463.55), so the code behaves as tuning.cc says. But the number is useless: 2-node busbw came out at 486.80GBps against a single node's 482.05, i.e. faster than pure NVLink, which is impossible for a real inter-node measurement. NCCL's own model special-cases it - min(bwIntra, nNodes <= 2 ? bwInter : bwInter/2). Never characterise a fabric on two nodes.
  • ib_write_bw -c SRD on EFA - 193.72Gbps on one adapter, 96.9% of its 200Gbps line rate. The "unconfirmed here" footnote is gone. perftest needed sudo apt-get install -y perftest on both hosts, and without -c SRD the run dies at Unable to create QP since EFA has no RC transport.
  • aws-ofi-nccl#890 - partly answered. The node exposes 16 EFA devices at 200Gbps each, 2 per accelerator, 3200Gbps/400GBps per node - which confirms the chapter's EFA v3 ... 16 200GbE line. The plugin's per-rank device assignment was not captured before the allocation was released, so the upstream question is still open; a NET/OFI grep of an NCCL_DEBUG=INFO multi-node log would finish it.

4 nodes

  • the busbw table in Inter-node speed depends on intra-node speed was reproduced on H200 rather than the published B200: 1 node 482.05GBps against 4 nodes 369.06GBps at 16GiB, so leaving the node costs 1.31x where B200 costs 2.2x. That difference is the section's own thesis - both platforms have the same 400GBps per node, but H200's NVLink 4 is 450GBps against B200's NVLink 5 at 900GBps, so the closer the two fabrics are the less the node boundary costs. Worth adding as a second table, but held until item 73 settles what the section concludes.

Specific hardware not currently to hand

  • still needs 4 nodes: verify which collective algorithm the published B200 busbw rows actually ran, on a 4-node P6-B200 allocation. Item 73 left the section honest but undecided: models 2 and 3 both fit the 22.05ms measurement within ~10%, and only the algorithm distinguishes them. One 4GiB run with NCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=INIT,TUNING via .deepspeed_env, then NCCL_ALGO=allreduce:ring and allreduce:nvlstree compared against the default - about three minutes of node time. H200 measured Ring, but the AWS tuner keys off the instance type - its log says base Tuner is chosen for platform: p5en.48xlarge - so a P6-B200 allocation gets a different tuner table and the H200 result does not transfer.

    • partial, 1x p6-b200.48xlarge on 2026-08-09 (stas-dev-1): confirmed NET/OFI Region base Tuner is chosen for platform: p6-b200.48xlarge (different tuner table than H200's p5en.48xlarge, as expected). Single-node default at 4GiB is Algo NVLS at 740.45GBps (published table 740.64) and at 16GiB 845.65 (published 845.67). NCCL_ALGO=allreduce:nvlstree is rejected on 1 node (invalid usage). The 4-node force-compare is what remains.

  • validate the SHARP/multicast granularity on an NVL36 or NVL72 system. The SHARP section now carries measured H200 and B200 HGX sweeps (B200 added 2026-08-09): H200 selects NVLS from 5 GPUs up, B200 stays on Ring at 5 and switches only from 6. The NVL36/NVL72 claim - granularity likely 4 GPUs from the partition guide - is still unvalidated on real NVL hardware; the two HGX generations already disagree, so the NVL case remains open.

  • suggestion 1: add the P6e-GB200 row, blocked on reading its per-NIC rate off a live instance. Parked rather than queued - it needs GB-series hardware this project does not have.