并非所有高速互联都是平等的，使用错误的可以相当昂贵：在InfiniBand上运行相同的分布式训练工作而不是NVLink可以降低43%的性能。
历史上，NVLink仅限于在一个单一节点内的通信，指多节点AI工作量被迫通过InfiniBand或等同云(EFA on AWS,GPUDIrect-TCPX on GCP)进行通信。 兹克 1Z的NVL72系统通过将所有72个GPU用统一的NVLink织物连接到一个架子中来去除这一限制，因此不同节点上的GPU可以以更带宽的等分级来交换数据。
然而，NVL72系统的效益最大化在很大程度上取决于工作量安排：分散在架子上的多节点工作仍然被InfiniBand所瓶颈。 由于硬件故障导致节点脱线，在不同时间完成的工作将自由能力分散到架子上，保留架子位置变得更加困难。
以及SkyPilot平台如何通过地形意识定位、硬件故障现场恢复、共享集群自动分解。
大规模培训是一个沟通问题#
边境培训工作跨越了数千个GPU。 这样做需要回答两个问题：
- 你如何在GPU上压住你的模型，以确保它能真正运行？
- 根据你的计算预算，多少份被磨损的模型可以运行以最大化的培训吞吐量？
回答第一个问题通常要结合：
- Tensor并行论(TP)：在GPU上分层权重矩阵
- 专家平行主义(EP)：将专家混合型(MOE)层的专家放在不同的GPU上
- 管道平行主义(PP)：在GPU之间按模式分层
第二题的答案设定了数据平行化(DP)的程度：并行地进行训练的被磨损模型的复制件数量，每个复制件在训练数据的不同片段上。
在这些平行主义策略中，高压平行主义的全减和专家平行主义的全相交流模式在训练期间坐落在关键路径上，使它们成为保持NVLink(§9.1.2)的最重要。
日相平行主义
tensor并行主义将每层内重量矩阵分出跨出GPU，因此每个GPU只存储并计算出每层的一分。
变形器地层由两个块组成：自意和多层受体(MLP). 在其核心中，每个块为一行两个矩阵乘法(GEMM). 威震天-LM(§3)显示，通过将一个块的第一个GEMM列和第二个GEMM列相分出，GPU只需要通过每个块的全减一次同步，在第二个GEMM之后，而不是每块之后。
即便有了这种优化，在前行传道期间，拉伸平行性仍需要每个变压器层两个全还原。 这些全部减少的步骤在培训期间处于关键路径上，因为它们不能与计算完全相重叠。 因此，在NVLink上通常保持了收发率平行主义：Hugging Face测量出从NVLink移动到InfiniBand后，培训吞吐量下降了43%.
专家平行主义
在混合专家(MoE)层中，MLP块被许多更小的MLP所取而代之，称为"专家"，以及一个路由器，将每个符号发送给一个小子集的专家。 专家平行主义将教育部的层层分割开来，使这些小型、独立的专家被置于不同的GPU上。
在专家平行主义下，每个MOE层在前行，出行和结合过程中需要两个全相通的通信步骤。 一个路由器为每个符物选择顶端-k专家，并发送给GPU，这些符使这些专家被内存。 每个GPU然后根据它通过一个单一分组矩阵乘法收到的符号来计算其本地专家。 最后，专家产出被送回到每个令牌的原GPU并合并。
随着模型向更多的专家增加更多的MOE地层并给每个地物通路，全程交通量会增加，引入了沟通瓶颈。
以64道专家平行方式培训教育部模式，将教育部各层的专家分布在64个全球pu。 在标准8-GPU服务器上，通过更窄的InfiniBand互联将全程流量从高带宽NVLink上移出，其中未优化的通信可消耗高达60%的培训时间(§1.2). 在NVL72系统中，所有64个GPU都停留在一个单一的NVLink域内，有效消除了通信瓶颈(§9.2.2).
推论也需要快速的节点间联网
上文讨论的平行技术并非培训所独有。 除了使推论引擎能够服务于对单一GPU来说太大的模型外，模型平行主义还可以提高吞吐量。 将一个模型的权重分成更多的GPU可以释放出KV缓存的内存，因此引擎可以处理更多的并行请求或支持更大的上下文windows. 在某些情况下，被释放的内存增长快于GPU的数量：在一个vLLM 基准测试中，在两个GPU之间分出一个模型的拉诺-平行，使KV缓存块的数量增加了13.9x并提供了3.9x多的吞吐量。
专家平行主义遵循同样的逻辑。 由于专家平行程度低，每个GPU都持有多位专家，为KV缓存留下了较少的空间。 WideEP，指跨越多个节点的专家-平行配置，显著地减少了每GPU的专家人数。 取舍是网络流量：现在每个全通都跨出更多的GPU. 在标准8-GPU服务器上，一个专家平行组最多可以跨出8个GPU，然后才能通过InfiniBand进行通信，其中每GPU带宽降低18x. NVL72通过将所有72个GPU保存在NVLink上的架子上来去除这个限制。 在以交互速度服务DeepSeek-R1时，有32道专家并行的GB200 NVL72可提供高达4.4x的每GPU吞吐量高于8-GPU B200服务器，这些服务器被封顶为8道。
架子是新服务器 #
NVL72将NVLink从一个单一的节点延伸至整个架子，因此整个架子表现得像一个巨大的有共享内存的加速器。
NVL72架上每个GPU都有18个NVLink端口，每个端口提供100GB/s双向带宽，这概括为每个GPU总共1.8TB/s. 每个NVLink端口在给定的GPU地图上，被传送到18个NVSwitch芯片之一，这些芯片为架子的织物后飞机提供动力。 由于每个NVSwitch芯片接受72条NVLink线，每条GPU各取出一条，因此机架内的总内带宽为130 TB/s.
这种网络地貌保证了织物完全没有被阻塞，指架上的任何GPU都可以将数据传送给其他任何具有统一耐久性的GPU，使大规模分布的AI工作量成为理想。
有时更少是更多：在每个架子上使用较少的节点会大大提高可靠性#
有了NVL72系统，将整个18个节点的架子分配给工作量，以最大限度地利用集群，这是很诱人的。 在实践中，各队实际上通过在每个架子上留下几个备用节点来获得更多的可用能力。
GPU是不可靠的。 Meta在其千兆瓦级超级集群上测量了中断(MTBI)之间平均时间约为每4-GPU节点490天，之后将重大的工程努力用于提高可靠性。 NVL72集群使得72-GPU架成为计算学的原子单位，与构成架子的4-GPU节点相比，断层爆炸半径增加了18x. 架子的MTBI由：
根据上述公式，大约每490 / 18 * 27天会中断一个架子。 如果分配了整个18个节点的架子，这些中断都使得整个架子无法被排出，直到失败的节点被修复。 在128个架子集群上，这意味着一个架子大约每5小时就无法用于全架工作。
每个架子保留两个节点作为备用部分改变这个。 当一个节点出故障时，一个在同一架子上的备用件会取而代之，因此一个单一的节点出故障就不再使整个架子脱离委托。 成本是每个架子有两个节点闲置，而一切都是健康的。
为了量化这种取舍，NVIDIA模拟了随着不可用节点份额的增加，不同的切片大小如何影响128个架(2,304-节点)集群的可用容量。
在SkyPilot，我们管理着数以万计的客户GPU，而在生产NVL72集群时，我们测得的缺电率约为2%。 以这个速度，NVIDIA的型号为显示，16-节点切片提供的可用容量比18-节点切片(2,037对1,602节点)多27%. 他们还以4%击败了9个节点切片，尽管两个9个节点切片精确地填入了一个架子：用9个节点切片，每次出故障都要花半个架子，而用16个节点切片，一个架子的前两个被完全吸收。 相较于8个节点切片，16个节点切片给每个工作每个NVLink域提供了两倍多的GPU，同时放弃了不到1%的可使用容量，直到节点不可用率超过约3.3%.
16个节点也是分布式AI工作量的自然大小：64个GPU是2个的功率，这与平行主义团体的典型大小相匹配。
现代调度员如何将工作保留在NVLink上
迄今为止，我们已经涵盖了为什么最通信重的平行技术属于NVLink,NVL72如何将NVLink从一个单一的节点扩展至整个支架，为什么每个支架有两个备用节点的16个节点片从生产集群中获取最可用的容量。 SkyPilot通过将每个切片放入一个单个机架上，从硬件故障中恢复到位置，以及随着工作的来去，使整个机架保持自由来去。
调度员需要了解硬件
选择合适的切片尺寸只是第一步。 研究者如何确保每个多节点训练的片段在单个架上实际运行获得？
NIVLink(1.8 TB/s双向):
SkyPilot的地形-智能调度器(TAS)确保工人在同一架上合用：
对于多机架工作量，TAS将工作分成16个节点(可配置)切片，并分别用两个备用节点放入自己的机架上。 然后，当SkyPilot的GPU管理器检测出GPU故障时，SkyPilot Frontier Trainer可以热地将故障节点与其中一个保留的健康节点相交换，而剩下的节点则继续不间断地运行。 如果同一架上没有剩余备用件，"边疆训练员"会将故障架上切片重新排到健康的架上：
为了量化以切片为基础的调度的影响，NVIDIA的GR00T团队比较了两种不同的调度政策在人类机器人的多模式基础模型GR00T N1.7预训期间对跨128个节点的512GB300GPU的影响。 第一次跑会使用一个天真的排程器，将节点放入任意数量的架子上。 第二次跑步的排期是采用像图10中那样的有分块分叉的"快板"政策，它将其完全放入了8个快板上。 Rack-aware布置融资吞吐量为14.4%，每次迭接从1.59 s被削减到1.39 s.
在一万-GPU集群上，同样的速度节省了大约每个训练小时16.7K. 只要做出更好的安排决定
断裂块
迄今为止，我们已探索了如何孤立地进行地貌意识排行。 在实践中，大型集群通常是共享的：不同的工作量，如培训前运行、RL推出和批量推论，都争夺同样的资源。 TAS bin在排程时间将工作量打包在同一个架子内，但工作在不同的时间完成，使得自由节点分散在架子上。 这是分裂：集群有足够的自由GPU来运行一个架子尺度的工作，但由于它们分布在架子上，无法安排工作。
SkyPilot 自动破损( Auto Defrag) i
本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Not all high-speed interconnects are created equal, and using the wrong one can be quite costly: running the same distributed training job over InfiniBand rather than NVLink can degrade performance by 43%.
Historically, NVLink was limited to communication within a single node, meaning that multi-node AI workloads were forced to communicate over InfiniBand or a cloud equivalent (EFA on AWS, GPUDirect-TCPX on GCP). NVIDIA’s NVL72 systems remove this limitation by connecting all 72 GPUs in a rack with a unified NVLink fabric, so GPUs on different nodes can exchange data with an order of magnitude more bandwidth.
However, maximizing the benefits of NVL72 systems is highly dependent on workload placement: a multi-node job spread across racks is still bottlenecked by InfiniBand. Preserving rack-locality gets harder as hardware failures take nodes offline and jobs finishing at different times scatter free capacity across racks.
This blog covers the communication bottleneck in large-scale training and inference, the NVL72 rack-scale fabric, and slice sizing for resilience to GPU failures, as well as how the SkyPilot Platform keeps multi-node jobs rack-local through topology-aware placement, in-place recovery from hardware failures, and automatic defragmentation of shared clusters.
Training at scale is a communication problem #
Frontier training jobs run across thousands of GPUs. Doing so requires answering two questions:
- How do you shard your model across GPUs to ensure that it can actually run?
- Given your compute budget, how many copies of the sharded model can you run to maximize training throughput?
The first question is typically answered with a combination of:
- Tensor Parallelism (TP): splitting a layer’s weight matrices across GPUs
- Expert Parallelism (EP): placing a mixture-of-experts (MoE) layer’s experts on different GPUs
- Pipeline Parallelism (PP): splitting a model layer-wise across GPUs
The answer to the second question sets the degree of Data Parallelism (DP): the number of copies of the sharded model that train in parallel, each on a different slice of the training data.
Of these parallelism strategies, tensor parallelism’s all-reduce and expert parallelism’s all-to-all communication patterns sit on the critical path during training, making them the most important to keep on NVLink (§9.1.2).
Tensor Parallelism
Tensor parallelism splits the weight matrices inside each layer across GPUs, so every GPU only stores and computes a slice of each layer.
Transformer layers are made up of two blocks: self-attention and a multi-layer perceptron (MLP). At its core, each block is two matrix multiplications (GEMMs) in a row. Megatron-LM (§3) showed that by partitioning a block’s first GEMM column-wise and its second GEMM row-wise, the GPUs only need to synchronize via all-reduce once per block, after the second GEMM, instead of after each one.
Even with this optimization, tensor parallelism still requires two all-reduces per transformer layer during the forward pass. These all-reduce steps sit on the critical path during training because they cannot be fully overlapped with computation. As a result, tensor parallelism is typically kept on NVLink: Hugging Face measured a 43% drop in training throughput when it moved off NVLink onto InfiniBand.
Expert Parallelism
In a mixture-of-experts (MoE) layer, the MLP block is replaced by many smaller MLPs, called experts, and a router that sends each token to a small subset of experts. Expert parallelism splits MoE layers such that these small, independent experts are placed on different GPUs.
Under expert parallelism, each MoE layer requires two all-to-all communication steps during the forward pass, dispatch and combine. A router selects the top-k experts for each token and dispatches tokens to the GPUs that have those experts in memory. Each GPU then computes its local experts on the tokens it received via a single grouped matrix multiplication. Finally, the expert outputs are sent back to each token's original GPU and combined.
As models add more MoE layers and route each token to more experts, the volume of all-to-all traffic grows, introducing a communication bottleneck.
Training an MoE model with 64-way expert parallelism spreads each MoE layer’s experts across 64 GPUs. On standard 8-GPU servers, that moves the all-to-all traffic off high-bandwidth NVLink and through much narrower InfiniBand interconnects where unoptimized communication can consume up to 60% of training time (§1.2). On NVL72 systems, all 64 GPUs stay within a single NVLink domain, effectively eliminating the communication bottleneck (§9.2.2).
Inference needs fast inter-node networking too #
The parallelism techniques discussed above are not unique to training. Beyond enabling inference engines to serve models that are too large for a single GPU, model parallelism can also improve throughput. Splitting a model's weights across more GPUs frees up memory for the KV cache, so the engine can process more concurrent requests or support larger context windows. In some cases, the freed memory grows faster than the number of GPUs: in one vLLM benchmark, splitting a model tensor-parallel across two GPUs increased the amount of KV cache blocks by 13.9x and delivered 3.9x more throughput.
Expert parallelism follows the same logic. With a low degree of expert parallelism, each GPU holds many experts, leaving less room for the KV cache. WideEP, which refers to expert-parallel configurations spanning multiple nodes, significantly reduces the number of experts per GPU. The trade-off is network traffic: every all-to-all now spans more GPUs. On standard 8-GPU servers, an expert-parallel group can span at most eight GPUs before it has to communicate over InfiniBand, where per-GPU bandwidth drops 18x. NVL72 removes this limit by keeping all 72 GPUs in a rack on NVLink. When serving DeepSeek-R1 at interactive speeds, GB200 NVL72 with 32-way expert parallelism delivers up to 4.4x higher throughput per GPU than 8-GPU B200 servers, which are capped at 8-way.
The rack is the new server #
NVL72 extends NVLink from a single node to the entire rack, so the whole rack acts like a gigantic accelerator with shared memory.
Every GPU on the NVL72 rack has 18 NVLink ports each delivering 100 GB/s of bidirectional bandwidth, which sums up to a total of 1.8 TB/s for each GPU. Each NVLink port on a given GPU maps to one of 18 NVSwitch chips that power the rack's fabric backplane. Because each NVSwitch chip accepts 72 NVLink lines, one from each of the GPUs, the aggregate internal bandwidth within the rack is 130 TB/s.
This network topology ensures that the fabric is fully non-blocking, meaning that any GPU in the rack can transmit data to any other GPU with uniform latency, making it ideal for large-scale distributed AI workloads.
Sometimes less is more: using less nodes in each rack significantly improves reliability #
With NVL72 systems, it is tempting to allocate entire 18-node racks to workloads to maximize cluster utilization. In practice, teams actually get more usable capacity out of their cluster by leaving a couple of spare nodes in each rack.
GPUs are unreliable. On its gigawatt-scale supercluster, Meta measured a mean time between interruptions (MTBI) of roughly 490 days per 4-GPU node after putting significant engineering effort into improving reliability. NVL72 clusters make the 72-GPU rack the atomic unit of compute, which increases the fault blast radius by 18x compared to the 4-GPU nodes that make up the rack. The MTBI for the rack is given by:
Based on the formula above, a rack would be interrupted roughly every 490 / 18 ≈ 27 days. If jobs are allocated entire 18-node racks, each of those interruptions makes the entire rack unschedulable until the failed node is repaired. On a 128-rack cluster, this means that a rack would become unusable for full-rack jobs roughly every 5 hours.
Reserving two nodes per rack as spares changes this. When a node fails, a spare in the same rack replaces it, so a single node failure no longer takes the entire rack out of commission. The cost is that two nodes per rack sit idle while everything is healthy.
To quantify this trade-off, NVIDIA modeled how different slice sizes affect usable capacity on a 128-rack (2,304-node) cluster as the share of unavailable nodes grows.
At SkyPilot, we manage tens of thousands of our customers’ GPUs, and on production NVL72 clusters we have measured unavailability rates of around 2%. At that rate, NVIDIA’s model shows that 16-node slices provide 27% more usable capacity than 18-node slices (2,037 vs. 1,602 nodes). They also beat 9-node slices by 4%, even though two 9-node slices fill a rack exactly: with 9-node slices, every failure costs half a rack, while with 16-node slices, the first two failures in a rack are fully absorbed. Compared with 8-node slices, 16-node slices give each job twice as many GPUs per NVLink domain while giving up less than 1% of usable capacity until node unavailability exceeds about 3.3%.
Sixteen nodes is also a natural size for distributed AI workloads: 64 GPUs is a power of two, which matches how parallelism groups are typically sized.
How a modern scheduler keeps jobs on NVLink #
So far, we have covered why the most communication-heavy parallelism techniques belong on NVLink, how NVL72 extends NVLink from a single node to the whole rack, and why 16-node slices with two spare nodes per rack get the most usable capacity out of production clusters. SkyPilot puts this into practice by placing each slice on a single rack, recovering from hardware failures in-place, and keeping whole racks free as jobs come and go.
Schedulers need to understand the hardware
Choosing the right slice size is only the first step. How do researchers ensure that each slice of a multi-node training run actually lands on a single rack?
Naive schedulers split multi-node jobs across racks, forcing workers to communicate over InfiniBand (400 Gb/s per direction) instead of NVLink (1.8 TB/s bidirectional):
SkyPilot’s Topology-Aware Scheduler (TAS) ensures that workers are colocated on the same rack:
For multi-rack workloads, TAS splits the job into 16-node (configurable) slices, and places them each on their own rack with two spare nodes. Then, when SkyPilot’s GPU Manager detects GPU failures, SkyPilot Frontier Trainer can hot swap the faulty node with one of the reserved healthy nodes while the remaining nodes continue running without interruption. If there are no remaining spares in the same rack, Frontier Trainer will reschedule the slice on the faulty rack to a healthy rack:
To quantify the impact of slice-based scheduling, NVIDIA’s GR00T team compared the effect of two different scheduling policies during pre-training of GR00T N1.7, a multimodal foundation model for humanoid robots, on 512 GB300 GPUs across 128 nodes. The first run used a naive scheduler that placed nodes across an arbitrary number of racks. The second run was scheduled with a rack-aware, slice-based policy like the one in Figure 10, which placed it on exactly 8 racks. Rack-aware placement raised throughput by 14.4%, cutting each iteration from 1.59 s to 1.39 s.
On a 10,000-GPU cluster, the same speedup saves roughly $16.7K per training hour. That’s about $400K saved for every training day, just by making better scheduling decisions.
Fragmentation blocks rack-scale jobs
So far we have explored how topology-aware scheduling works in isolation. In practice, large clusters are typically shared: heterogeneous workloads such as pre-training runs, RL rollouts, and batch inference all compete for the same resources. TAS bin packs workloads within the same rack at scheduling time, but jobs finish at different times, leaving free nodes scattered across racks. This is fragmentation: the cluster has enough free GPUs to run a rack-scale job, but because they’re spread across racks, the job can’t be scheduled.
SkyPilot’s Automatic Defragmentation (Auto Defrag) i
Auto-fetched and translated by AI Gem Box. All rights belong to the original author.