在集群尺度上，GPU断层是一个现实。 Xid出错，双位ECC出错，GPU从PCIe总线上掉出，或者将InfiniBand链接相扇会拖住分布式训练工作，或者将推想复制到下。 发现什么是错的通常是容易的难点在于断层与健康节点之间的一切
今年早些时候，我们宣布我们完全自动节点补救功能的发布。 我们的GPU集群客户和我们的内部推论平台一直利用这种能力来帮助驱动工作量回升并尽量减少下行时间。 我们的客户喜欢这种能力，我们想花一些时间打破系统，解释它是如何工作的，为什么我们是这样设计的。
我们首先实施了“大规模健康”检查制度，在你的VM内部和在它的下方的宿主上持续观察每个节点。 当一次检查发现一个故障时，自动修复建议最轻的修复动作，显示你支持推荐背后的证据。 你当面决定，过失决定，修理是等待批准还是自行进行。 您也决定汽车修理需要多久等待运行工作量完成。 你自己的探测器可以通过在节点上写出单个注释来打开同一管道的修复。
可用性
今天普遍可以使用：被动健康检查，修复建议，自动补救政策(包括每个故障和每个节点的同意)，等待控制，升级阶梯，安全rails，以及API和控制台修复。
预览：基于节点标签的补救合同，可以带来自己的检测系统，如NVSentinel或自定义控制器。
我们不断增加新的探测信号和补救行动，以确保恢复的最大覆盖面和最低时间。
何以不备也？
单个信号上大多数节点自动修复密钥：节点不留。 这个信号看起来是相同的无论一个节点是断了还是短暂的无法到达。 当一个存储器或网络事件同时触及到许多节点时，每个被撞击的节点都会掉入"不留". 我们的系统有一个断路器用来防止节点的大量更换 NotReady也没有说节点为什么会出故障，所以在重开机和新机之间没有选择的依据。
因此，汽车修理遵循了一条规则：节点修理发生时，后面没有进行失败的健康检查，每一次修理都带有其佐证证据。
输油管
管道有四个阶段：检测出一个断层，决定是否和何时行动，修复节点，并重新验证机器。 前三个阶段都与你的集群相对立第四个运行在我们的一边，在物理主机你的节点刚刚空出。 节点标签请求您自己的检测工具进入决定阶段并得到完全相同的治疗我们的。
1. 从超视线两侧探测
GPU在GPU Cluster中直接传递到你的VM，用于光金属性能。 这意味着GPU断层从VM内部可见，而主机和布料断层则从主机可见。 因此，探测器和获得的两侧运行在一个地方。
在你的VM，出口商观看：
- 内核日志，用于Xid出错和从PCIe总线上掉下来的GPU
- DCGM，用于ECC双位错误、行重映射失败、PCIe重放率和热减速
- 节点问题探测器，用于只读文件系统、XFS关闭、内核僵局和重复的Kubelet或容器重启
在Slurm上，出口商报告标有下标或DRAIN的节点。
在主机上，我们观看了InfiniBand铁路状态和连接相拍，这归功于拥有这些rails的VM. 我们还观察宿主的可达性，所以一个死去的宿主显示作为一个问题在你的节点上，而不是沉默。
每艘警戒舰都有一个稳定的概要，一个简单的语言描述，和一个机器可读的证据有效载荷。 对于一个Xid，这是每个代码和它的NVIDIA mnemonic. 对于高温节流，是最高温度和节流阀计数。 控制平面通过未触动的字符串，所以控制台，API(作为链接的提醒)，你的通知总是这么说。
我们观察与可行动性分开每一个与修复有关的警报都被记录下来，并标记出你的节点降解。 只有带有动作映射的信号才产生修复。 例如：显示向上但不会触发重开，因为重开不会修复。 一个没有地图的警报被统计和记录，从来没有默默投放，并引导到我们的支援团队调查。
2. 政策：我们决定什么，你决定是否以及何时
由平台拥有的每一个信号映射到最轻的动作来修正它，所以你不需要知道Xid的意思是什么：
是否由你决定。 在修补分页中，自动补救政策默认在修补前批准：每条建议都等待一个人，他可以接受或推翻操作。 你可以将默认切换为全能自动，但它不是一个被子开关。 修补动作核对表通过它们触发的修补来组合出故障，你选择了您信任的运行无人关注的缺陷。 在Xid集团内部，你可以设定同意每个代码。 常见的设置自动批复为Xid 119和95，并保持出行门。 Xid 检查表在修复前也会在批准下工作，如果您想要在其它一切等待您的审查时自动运行简短的断层列表。
什么时候也是你的。 修补会排出节点，所以等闲置后再保留到节点实际闲置。 在Kubernetes上，这意味着节点上没有运行的吊舱请求一个GPU. Slurm上，指调度器报告节点没有分配。 最大等值会约束被扣，默认为2小时，上限为24. 不中断运行任务会完全去除束缚。 将等待设定为至少您的检查站间隔。 如果捆绑期满，则记录修复进度和超时。
每一份要求修复的修复记录(将健康检查器联合起来进行自动修复)，谁批准它(一个命名的用户或自动批准)，谁决定，什么时候决定。
3. 携带自己的探测器
运行大型GPU车队的团队已经有他们信任的检测：DCGM仪表板，NVSentinel，内部扫描仪，工作级别检查。 每一次修理，无论来源如何，都成为我们控制飞机的同一记录。 这样就可以进行能力检查审计安全rails 完全一个家
有三个方法：
- API。 - API。 建立对一个有模式的实例的修复，一个可选择的客户端提供的ID来进行一能，以及可选择的健康检查运行或提醒你引用作为证据。 您也可以列出、获取、批准、拒绝和取消修复。
- 指挥台任意节点上的 QQ 菜单会打开包含检测出的问题和建议的动作的修复对话框。
- 节点标签(审查)。 任何经RBAC许可写出节点元数据的探测器，可以通过在节点上添加一个注释和一个标签来打开一个修复。 注释带有探测器名称，稳定的请求ID，修复模式，当断层被检测出时，以及人可读的理由。
我们的集群操作员将节点标签映射到控制平面。 修复时会以探测器作为外部请求器被记录下来，因此它总是可以归属，永远不能冒充一个用户。 探测器名称外加请求ID是整个集群的等效键，包括了已完成的修复，因此在每次扫描中重新加固相同断层的探测器无法重开两次节点。 授权是你们自己的RBAC，没有API密钥和登机票。
注释请求可以请求再启动，快速再提供，或者Migrate. 仅限人类删除就像控制台点击，这些请求跳过批准队列，但他们仍然通过相同的证据门，能力检查，和预算和其他一切。 每组盖指窃听探测器造成的破坏有限，而不是灾难。
4. 修理行动和升级梯子
每一次修理都遵循同样的生命周期：警戒线，等待(按照你的政策)，排水，行动，重新加入。 控制台显示节点通过 Cordoning Draining 修补/Migrating 加入跑道。
从轻到重的行动：
- 重新开始。 重新启用越战队本地/切和/tmp数据能活下来。
- 快速再补给。 从一个健康的宿主上一个干净的形象重建 VM. 本地数据已丢失。
- 搬去新主机在不同的硬件上重建VM，并发去旧主机重新验证。
- 拿开为RMA拉出节点。 总是由人类引起的。
相同的修补不会循环。 如果在同一个动作已经运行了两次之后，同一机器上又出现了一个故障，则下一个建议会移动一跑。 第三轮"西德回放"成为再提供，第三轮再提供成为移民。 梯子被跟踪到每个机器，集群，警报，和Xid代码。 超过一周的尝试不算，取消或自行解决的建议也不算。
移民是有证据的。 迁移是消耗实际能力的一个动作。 这需要在过去十天内对节点进行紧急警报，或对同一机器进行紧急警报。 满足两个条件的请求都以快速重置方式运行。
主机动力循环(预览). 一些故障，其中Xid 79，需要GPU失去动力，尽管硬件可能很好，但VM重启还是无法做到。 而不是消耗一个新的宿主，我们可以用电循环你正在运行的。 系统中任何地方都没有业连管证书。 启动ID检查阻止重试循环已经返回的主机。
需要有保证的替换窗口的集群可以保留热余能，因此迁移可以吸收储备而不是一般可用。 问问你的帐务组详细情况
5. Rails 假设警报风暴将发生
糟糕的驱动推力，存储中断，或网络分区可以让几十个节点一看就恶心。 节点级修复无法修复集群级原因，因此rails被分层并独立：
- 坚持下去信号在成为建议之前必须保持每个信号的持续时间。
- 冷静经过任何修复后，一个节点在生成新建议前约需要30分钟。
- 货币预算。 预算规模取自集群节点计数，操作员每次运行每个节点池一次修复。
- 断路器当一个集群产生一个相通的自动修复的爆发时，断路器会出行。 许多节点一次失败就指向一个集群级的原因，一个接一个地修复节点不会修复。
-每次修理一个目标计划执行，所以一个虫子不能在节点之间扇出。
- 自动解决。 一条建议，在任何人的行为自行消失之前，警报就解除了。
控制平面被复制，每个书写都具有"一能"并对照所期望的状态检查。 两个复制品在同一节点上赛跑，产生一个修复，而不是两个。
6. 机器重新验证本身
当一个节点从一个机器上移出，或者从一个机器上移出，那台机器不应该直接回到游泳池里，它不应该坐在排队里等待别人看诊断。 老VM一去不复返，机器就自动进入验证。 一个测试VM运行着相同的GPU,NCCL,InfiniBand，并且存储检查每个全新的机器在加入舰队之前都会通过。 在节点离开和判决落地之间没人碰机器
有三项判决：
-通过机还入池。 没有罚单，也没有任何人被呼叫，所以一个瞬间断层仍然处于瞬间。
-失败了自动提交硬件票，并附上每个测试结果和记录。 机器一直待到修好再通过
- 已经中止了检查无法运行，例如因为主机被重新定位或者测试VM无法被放置。 该机被恢复到以前的状态，没有任何被谴责。
只有明显的失败才会产生罚单。
用于延长一个加速器
修复关于提醒、节点和机器的决定密钥，而不是GPU ve
本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
At cluster scale, GPU faults are a reality. An Xid error, a double-bit ECC error, a GPU falling off the PCIe bus or a flapping InfiniBand link can stall a distributed training job or take an inference replica down. Spotting that something is wrong is usually the easy part. The hard part is everything between the fault and a healthy node again
Earlier this year, we announced the launch of our fully Automatic node remediation features. Our GPU clusters customers and our internal Inference platform has been using this capability to help drive workload resiliency and minimize down time. Our customers love this capability, and we wanted to take some time break the system down and explain how it works and why we designed it this way.
We first implemented “Passive health” checks system, that watches every node continuously, both inside your VM and on the host underneath it. When a check finds a fault, auto repair recommends the lightest action to fix and shows you supporting evidence behind the recommendation. You decide up front, fault by fault, whether a repair waits for approval or runs on its own. You also decide how long auto repair waits for running workload to finish. Your own detectors can open repairs through the same pipeline by writing a single annotation on a node.
Availability
Generally available today: passive health checks, repair recommendations, the auto-remediation policy (including per-fault and per-node consent), wait-for-idle controls, the escalation ladder, the safety rails, and API and console repair.
Preview: node-label based remediation contracts, which let you bring your own detection systems such as NVSentinel or custom controllers.
We are constantly adding new detection signals and remediation actions to ensure maximum coverage and minimum time to recovery.
Why NotReady is not enough
Most node auto-repair keys on a single signal: the node went NotReady. That signal looks the same whether a node is broken or just briefly unreachable. When a storage or network event touches many nodes at once, every impacted node drops to NotReady. Our system has a circuit breaker built in to prevent mass replacement of nodes. NotReady also does not say why the node failed, so there's no basis for choosing between a reboot and a new machine.
So auto repair follows one rule: no node repair occurs without a failed health check behind it, and every repair carries its supporting evidence.
The pipeline
The pipeline has four stages: detect a fault, decide whether and when to act, repair the node, and re-validate the machine. The first three stages run against your cluster. The fourth runs on our side, on the physical host your node just vacated. Node-label requests from your own detection tooling enter at the decide stage and get exactly the same treatment as ours.
1. Detection from both sides of the hypervisor
GPUs in Together GPU Clusters are passed straight through to your VM for bare-metal performance. That means GPU faults are visible from inside the VM, while host and fabric faults are visible from the host. Detection therefore runs on both sides of the hypervisor and lands in one place.
Inside your VM, exporters watch:
- the kernel log, for Xid errors and GPUs falling off the PCIe bus
- DCGM, for ECC double-bit errors, row-remap failures, PCIe replay rate, and thermal throttling
- Node Problem Detector, for read-only filesystems, XFS shutdowns, kernel deadlocks, and repeated kubelet or containerd restarts
On Slurm, an exporter reports nodes marked DOWN or DRAIN.
On the host, we watch InfiniBand rail state and link flapping, attributed to the VM that owns those rails. We also watch host reachability, so a dead host shows up as a problem on your node rather than as silence.
Every alert ships a stable summary, a plain-language description, and a machine-readable evidence payload. For an Xid, that's each code and its NVIDIA mnemonic. For thermal throttling, it's the peak temperature and throttle count. The control plane passes those strings through untouched, so the console, the API (as linked_alerts), and your notifications always say the same thing.
We keep observed separate from actionable. Every repair-relevant alert is recorded and marks your node Degraded. Only signals with an action mapping create a repair. Disk pressure, for example, shows up but doesn't trigger a reboot, because a reboot won't fix it. An alert with no mapping is counted and logged, never silently dropped, and routed to our support team for investigation.
2. Policy: we decide what, you decide whether and when
What is owned by the platform. Each signal maps to the lightest action that fixes it, so you never need to know which Xid means what:
Whether is yours to decide. In the Repairs tab, the Auto-remediation policy defaults to Approve before repair: every recommendation waits for a person, who can accept or override the action. You can switch the default to Fully automatic, but it isn't a blanket switch. The Repair actions checklist groups faults by the repair they trigger, and you select the ones you trust to run unattended. Inside the Xid group, you can set consent per code. A common setup auto-approves reboots for Xid 119 and 95 and keeps migrations gated. The Xid checklist also works under Approve before repair, if you want a short list of faults to run automatically while everything else waits for your review.
When is yours as well. A repair drains the node, so Wait for idle holds it until the node is actually idle. On Kubernetes, that means no running pod on the node is requesting a GPU. On Slurm, it means the scheduler reports no allocation on the node. Maximum wait bounds the hold, with a default of 2 hours and a cap of 24. Do not interrupt running jobs removes the bound entirely. Set the wait to at least your checkpoint interval. If the bound expires, the repair proceeds and the timeout is recorded.
Every repair records who requested it (Together Health Checker for automatic repairs), who approved it (a named user or Auto-Approved), which rule decided, and when.
3. Bring your own detector
Teams running large GPU fleets already have detection they trust: DCGM dashboards, NVSentinel, in-house scanners, job-level checks. Every repair, whatever its source, becomes the same record in our control plane. That gives capacity checks, audit, and safety rails exactly one home.
There are three ways in:
- API. Create a repair against an instance with a mode, an optional client-supplied ID for idempotency, and optionally the health check run or alert you're citing as evidence. You can also list, get, approve, reject, and cancel repairs.
- Console. The ⋮ menu on any node opens the Repair dialog with the detected issues and the recommended action.
- Node labels (Preview). Any detector with RBAC permission to write node metadata can open a repair by adding one annotation and one label to the node. The annotation carries the detector name, a stable request ID, the repair mode, when the fault was detected, and a human-readable reason.
Our cluster operator mirrors the node label up to the control plane. The repair is recorded with the detector as an external requester, so it's always attributable and can never impersonate a user. Detector name plus request ID is the idempotency key across the cluster, finished repairs included, so a detector that re-asserts the same fault on every scan can't reboot a node twice. Authorization is your cluster's own RBAC, with no API key and no onboarding ticket.
Annotation requests can ask for Reboot, Quick reprovision, or Migrate. Remove stays human-only. Like a console click, these requests skip the approval queue, but they still pass the same evidence gate, capacity check, and budgets as everything else. A per-cluster cap means a buggy detector causes limited damage rather than a catastrophe.
4. Repair actions and the escalation ladder
Every repair follows the same lifecycle: cordon, wait (per your policy), drain, act, rejoin. The console shows the node moving through Cordoning → Draining → Repairing / Migrating → Joining → Running.
The actions, from lightest to heaviest:
- Reboot. Restarts the VM in place. Local /scratch and /tmp data survive.
- Quick reprovision. Rebuilds the VM from a clean image on a healthy host. Local data is lost.
- Migrate to new host. Rebuilds the VM on different hardware and sends the old host to re-validation.
- Remove. Pulls the node for RMA. Always human-initiated.
The same fix doesn't loop. If a fault comes back on the same machine after the same action has already run twice, the next recommendation moves one rung up. A third round of Xid reboots becomes a reprovision, and a third reprovision becomes a migration. The ladder is tracked per machine, cluster, alert, and Xid code. Attempts older than a week don't count, and neither do cancelled or self-resolved recommendations.
Migrations are evidence-gated. Migrate is the one action that consumes real capacity. It requires either a critical alert currently firing on the node or a critical alert on the same machine within the last ten days. A request that meets neither condition runs as a Quick reprovision instead.
Host power cycle (Preview). Some faults, Xid 79 among them, need the GPU to lose power, which a VM reboot can't do even though the hardware may be fine. Rather than consuming a new host, we can power-cycle the one you're on. No BMC credentials exist anywhere in the system. A boot-ID check prevents a retry from cycling a host that has already come back.
Clusters that need a guaranteed replacement window can reserve hot spare capacity, so migrations draw from that reserve rather than general availability. Ask your account team for details.
5. Rails that assume the alert storm will happen
A bad driver push, a storage outage, or a network partition can make dozens of nodes look sick at once. Node-level repair can't fix a cluster-level cause, so the rails are layered and independent:
- Persistence. A signal must hold for a per-signal duration before it becomes a recommendation.
- Cooldown. After any repair, a node gets roughly 30 minutes before a new recommendation can be generated.
- Concurrency budget. The budget is sized from cluster node count, and the operator runs one repair per node pool at a time.
- Circuit breaker. When one cluster generates a burst of correlated automatic repairs, the breaker trips. Many nodes failing at once points to a cluster-level cause, and repairing nodes one by one will not fix it.
- One target per repair. The schema enforces it, so a bug can't fan out across nodes.
- Auto-resolve. A recommendation whose alert clears before anyone acts disappears on its own.
The control plane is replicated, and every write is idempotent and checked against the state it expects. Two replicas racing on the same node produce one repair, not two.
6. The machine re-validates itself
When a node migrates off a machine, or is removed from one, that machine shouldn't go straight back into the pool, and it shouldn't sit in a queue waiting for someone to read diagnostics either. As soon as the old VM is gone, the machine moves into validation automatically. A test VM runs the same GPU, NCCL, InfiniBand, and storage checks that every brand-new machine passes before joining the fleet. No person touches the machine between your node leaving and the verdict landing.
There are three verdicts:
- Passed. The machine returns to the pool. No ticket is filed and no one is paged, so a transient fault stays transient.
- Failed. A hardware ticket is filed automatically with per-test results and logs attached. The machine stays out until it's fixed and passes again.
- Aborted. The checks couldn't run, for example because the host was reallocated or the test VM couldn't be placed. The machine is restored to its prior state, and nothing is condemned.
Only an explicit failure produces a ticket.
Built to extend past one accelerator
Repair decisions key on alerts, nodes, and machines, not on the GPU ve
Auto-fetched and translated by AI Gem Box. All rights belong to the original author.