Lab
AllReduce, two ways
Every large-scale training run leans on collective communication to sum gradients across GPUs. There's no single "best" algorithm — Ring AllReduce is bandwidth-optimal and dominates for large tensors; Recursive Doubling (a butterfly/hypercube exchange) is latency-optimal and wins for small messages. NCCL picks between algorithm families like these based on message size and topology at runtime.
How to read this
Ring: the tensor is split into N chunks. In 2 phases of N−1 steps each, every GPU only ever talks to its two ring neighbors — reduce-scatter sums each chunk into one owner, then all-gather propagates the finished chunks back around. Every link carries roughly the same amount of traffic regardless of N, which is why it scales so well for large tensors.
Recursive Doubling: at round r, every GPU exchanges its current partial sum with the partner at XOR-distance 2r and both keep the combined value. After log₂N rounds every GPU independently converges on the full sum — no separate broadcast phase needed. Fewer steps, but each one moves the whole tensor, so it loses to Ring once the message gets large.
The time estimate is a simplified model (steps × (latency + bytes/bandwidth)) meant to build intuition for the two algorithms' scaling behavior — it ignores link contention, network topology, and protocol overhead, so treat it as illustrative, not a benchmark substitute.