人工智能
与Ray Train DLCs和EKS普及教育组织分配培训
通常阻止PyTorch训练从缩放到集群的是容器，而不是模型代码。 每个容器需要相同的 CUDA 运行时间，PyTorch 构建基于它编译的 NCL (NVIDIA 集体通信库)，弹性 Fabric 适配器 (EFA) 用户空间库，AWS OFI NCL 插件将 NCL 连接到 Emplo，集群操作员同意的Ray版本。 建造和重塑这幅图像每一次这些棋子的移动都属于它自己的项目，和你开始训练的模型无关。 雷克列车深层学习容器(DLC)就是由AWS所建造的这个图像。 AWS将堆栈组装，测试为一单元，以Amazon弹性容器注册(Amazon ECR)公开出版并保持补丁。
Ray在机器之间运行着一个工作，作为一个具有两个角色的单一集群。 Ray头协调和运行跟踪集群状态的全局控制商店，您提交任务时的仪表板和工作服务器。 雷工节点提供GPU，并运行培训流程。 每个工人加入通过在头部开始时连接到它的地址。 在Kubernetes上，每个都有一个舱。 Kubernetes对于一个雷星团没有内在的想法，所以有些东西必须创造出头和工人的吊舱，让雷在每一个角色中开始正确的角色，把工人指向头部，并更换出故障的吊舱。 库贝雷会这么做开源 Kubernetes操作员将新增作为RayCluster资源：您在一份清单中描述一个头组和一个工人组，KubeRay从中构建并维护集群。
一个被标记的单一图像作为雷头和雷工同时运行，无论是KubeRay在Amazon弹性Kubernetes服务(Amazon EKS)上启动，还是你自己在Amazon弹性计算云(Amazon EC2)上启动。 您添加的是配置：哪个实例，每个 pock 得到多少 GPU，哪个网络传输 NCCL 使用，还有哪个额外的 Python 包需要工作。 皆从外相所出。
这篇文章通过一个样本，在两个g6.12x大例上，该样本完全使用公布的DLC到微调 Qwen2.5-1.5B，并带有PyTorch Fully Sharted Data Affirl (FSDP)，跨越了8个NVIDIA L4 GPU. 训练代码为一函数，集群为一RayCluster Express 外加变量文件，任何一点都无法构建出图像。 由于DLC已经承载了全民教育的堆栈，一旦吊舱提出要求，两个实例之间的集体就会冲出全民教育。 整个溶液都在这个样本库中，它使用与我们的多节点Ray Service走过相同的KubeRay-on-EKS基础。
你得到什么，没有建造任何东西
雷克列车DLC为PyTorch训练形象，雷克的训练图书馆位于顶部。 标记这个样本牵引，public. ecr.aws/deep-learning-containers/ray:train-ml-cuda-v1.1，替换一个本来会像这样的建筑：
网络排队是最不想拥有的不匹配不会产生错误。 NCCL下降为TCP套接字，任务以速度的一小部分完成。 在DLC中，这些图书馆是一起建造和测试的，AWS门在多节点全民教育和多节点KubeRay培训测试上各发布，这些正是本样本使用的路径。
再删除几个属性，否则您每个组都会完成的工作：
- 没有设置登记簿图像是公开的，因此节点拉出它没有证书，没有任何东西被建造或推倒在路上。
- 没有匹配的司机在启动时，切入点会检查主机的 NVIDIA 驱动程序，如果它比图像的 CUDA 年长，则自动打开捆绑的 CUDA 前置兼容库。
——没有头-工-相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相相不管什么开始容器供应射线开始命令，所以一个标签服务每个舱。
- 没有单独的推想图像家族 – 雷电列车与雷Serve DLC和雷LLM DLC共享射线ECR寄存器，在列车-ml标记前缀下。
配置它，不要重建它
一种训练工作与下一种训练工作之间的一切差异，都是围绕画面设定的。 样本使用了其中的每个机制：
包每个工作。 图片载HuggingFace训练堆，因此这个样本的工作不需要添加。 当一个实验确实需要图像不携带的库时，Ray在提交时通过runtime env为该工作安装，图像不变。 没有新图像，没有新标签。
运输每份工作。 图像默认为FI PROVIDER=efa，其NCCL配置排除了回路和Docker桥接接口。 在 EKS 上，您可以将 NCCL SOCKET IFNAME 指向 sockET IFNAME 的 seth0。 样本在清单中设置了全部四个网络变量，因此，行为在一个文件中可见，因为这些是平坦的环境变量，单项工作可以通过运行时间 env来覆盖它们。
您选择的版本。 样本取出"列车-ml-cuda-v1.1"，即目前的"小放出"，因此一个集群从一跑到一跑都可以再生。 vX.Y标签随安全重建而移动。 完全合格的vX. Y.Z标记在最初的出厂时会被冻结，所以当需要字节相同的跑步时会给一个文摘。 同样的形象也可以从私人ECR URI中获取。
A. 如何划分分组
所有节点都坐落在一个虚拟私有云(VPC)的私人子网中，并且只能通过NAT网关进入互联网，这就是每个节点如何从Amazon ECR Public中拉出DLC，以及工人如何从Hugging Face下载模型重量。 一个m7i.x大组成了系统节点组，托管了kube-system add-ons,KubeRay操作员和Ray头。 头部在8265号港口和6379号雷的控股商店运行工作服务器，它广告上没有GPU，因此没有培训工作获得在上面。 工作从头舱内以kubectl exec提交，通过EKS API，因此8265端口从未在集群外被曝光。
培训涉及2个g6.12x大实例，每个实例有4个NVIDIA L4 GPU、48个vCPU、192个GB内存、40个Gbps联网和1个全民教育接口。 Ray Train将4名工人放在每个舱内：一次排名0-3，另一次排名4-7。 因为全民教育流量不会跨越可用区
雷克列车在集装箱里做什么
DLC提供堆栈。 Ray Train提供本来会成为发射器代码的部分。 多节点 PyTorch通常指在每个主机上开始一个每个GPU的过程，并给出每个主机的等级，世界大小和一个会合地址。 使用Ray Train将训练循环写成一个函数，并用缩放Config传递给火炬Trainer. 雷在集群有GPU的地方，每GPU开始一个工人，形成火炬。 分配流程组，并在每个工人身上运行您的职能。
这个样本向Ray Train询问FSDP,FSDP将模型的权重，梯度和优化状态分成了所有8个GPU，而不是将其复制到每一个。 bfloat16中的Qwen2.5-1.5B需要约12GB用于这四个收分。 被硬化了8个方式，每个L4GPU持有大约1.5GB. 价格是通信：FSDP在使用前会收集每层的权重，然后减少散射梯度，因此整个模型在每一步的分级之间交叉。 在这两起事件中，交通是DLC的全民教育堆装。
先决条件
在开始前，您必须具备下列条件：
- 工具：AWS 命令行接口(AWS CLI)v2 签入目标账户，eksctl, kubectl, Helm，从 GNU 获取文本的envsubs.
- 定额：在AWS地区，至少有一个区提供两个G-Family On-Demand能力。 如果您已经为实例保留了待发能力，请选择 CAPACITY RESERVATION ID。 脚本将发布节点组插入其中并跳过容量搜索。
- 预算：大约每小时9.50美元，而GPU的情况正在增加，涵盖两个情况，即EKS控制平面和NAT Gateway。
快速确认CLI指向您期望的账户：
训练守则：一个职能和一个训练员
代码/train.py是整个工作，它并不包含容器，集群或网络的特定内容。 雷运行列车时每GPU一次。 它要求雷的级别，加载模型，手为FSDP准备模型，并运行一个普通的PyTorch回路：
代码/train.py——每GPU函数(简写)
驱动程序加入运行集群，数出其GPU并启动教练。 由于工人计数来自集群，同一个脚本在一个节点或许多节点上运行不变：
代码/train.py——司机(简写).
分批是随机的代号ID，这样运行就只依赖于集群。 样本是根据真正的FSDP集体是否跨越了节点和完成的工作来判断的。 对于真实的微调，用Ray Data或DataLoader管道来替换随机的收发器。 函数中没有任何变化。
名单：DLC 唯一的配置地点
express/ray cluster.yaml是图像与集群相接的地方，它很短，因为图像需要如此之少。 头部唯一的特殊设置是NVIDIA VISIBLE DEVICES=void，它阻止了容器运行时间的NVIDIA钩子在一个没有GPU的节点上发生故障。 工人小组拥有其他一切：
活体/射线组群.yaml——工人组群(精简)
读为三决。
该图像：${DLC IMAGE}，未修改，既适用于头部，也适用于工人。
哪个硬件：4个GPU和节点的每个吊舱的单一全民教育接口。 要求vpc.amazonaws.com/efa是普及教育设备插件将接口挂入容器的原因。 NCCL看不到普及教育
也就是四个环境变量所以没有其他东西可以自动检测了
两个更小的细节保持了多节点工作的稳定。 首先，要求应具有同等的限制，使工人处于Kubernetes保证服务质量(QoS)等级。 这很重要，因为单节奏或被逐出的军衔会拖住另外7个军衔。 第二，num-gpus是从和GPU请求相同的变量得到的，因此Ray对容量的看法总是与Kubernetes相匹配。
提出来：一个是基础设施的命令，然后是工作
配置活在一个地方，脚本/env.sh：集群名，区域，Kubernetes版本，节点类型和计数，图像，被绑定的库贝雷和两个设备插件版本，以及训练参数。 将其中任何一种导出以改变它运行。 所出五部正文皆能一一相并能安住重行。 部署 croup.sh，部署 node group.sh，在创建或触碰集群资源前，部署 ray train job.sh。 安装 gpu plugins.sh并安装 kuberay.sh是调和器并直接运行。
./dept all.sh – 发现实际容量在哪里，然后在那里建设基础设施。
g6.12x 每天在各地区和可提供区之间的大容量转移，因此这是起点，而不是在env.sh中被预先猜到的区域。 它首先运行 find gpu capacity.sh，检查每个候选区域：每个可用区是否提供 g6. 12x 宽度，该账户的 On-Demand G/VT-family vCPU 配额是否为要求的节点计数空间，以及EC2 Spot 放置分数作为容量信号。 用 -- probe，默认部署 all.sh通过，它通过在每个候选区创建并立即取消真正的能力保留，得到一个明确的答案。 然后显示计划并问一次：
回答 y 运行，顺序是部署 croup.sh，部署 node group.sh，安装 gpu plugins.sh，并在推荐区域安装 kuberay.sh，每人跳过自己的重复确认，因为这一个已经要求( -y 也跳过此提示，和 -- plan 显示计划而不创建任何东西)。 如果 GPU 节点组不能在最佳区域中发布，脚本会删除它并尝试探测器的下个区域
本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Artificial Intelligence
Distributed training with Ray Train DLCs and EFA on EKS
What usually stops PyTorch training from scaling out to a cluster is the container, not the model code. Every container needs the same CUDA runtime, a PyTorch build compiled against it, NCCL (NVIDIA Collective Communications Library), the Elastic Fabric Adapter (EFA) userspace libraries, the AWS OFI NCCL plugin that connects NCCL to EFA, and a Ray version the cluster operator agrees with. Building that image and rebuilding it each time one of those pieces moves is a project of its own, and it has nothing to do with the model you set out to train. The Ray Train Deep Learning Container (DLC) is that image, built by AWS. AWS assembles the stack, tests it as one unit, publishes it in Amazon Elastic Container Registry (Amazon ECR) Public and keeps it patched.
Ray runs a job across machines as a single cluster with two roles. The Ray head coordinates and runs the global control store that tracks cluster state, the dashboard and job server you submit jobs to. Ray worker nodes supply the GPUs and run the training processes. Each worker joins the cluster by connecting to the head’s address when it starts. On Kubernetes, each of these is a pod. Kubernetes has no built-in idea of a Ray cluster, so something must create the head and worker pods, start Ray in the right role in each, point the workers at the head, and replace pods that fail. KubeRay does this. It’s an open source Kubernetes operator that adds a RayCluster resource: you describe a head group and a worker group in one manifest, and KubeRay builds and maintains the cluster from it.
A single tagged image runs as both the Ray head and the Ray workers, whether KubeRay starts it on Amazon Elastic Kubernetes Service (Amazon EKS) or you start it yourself on Amazon Elastic Compute Cloud (Amazon EC2). What you add is configuration: which instances, how many GPUs each pod gets, which network transport NCCL uses, and which extra Python packages a job needs. All of it is set from outside the image.
This post walks through a sample that uses the DLC exactly as published to fine-tune Qwen2.5-1.5B with PyTorch Fully Sharded Data Parallel (FSDP) across eight NVIDIA L4 GPUs on two g6.12xlarge instances. The training code is one function, the cluster is one RayCluster manifest plus a file of variables, and no image is built at any point. Because the DLC already carries the EFA stack, collectives between the two instances run over the EFA as soon as the pods request it. The entire solution is in this sample repository, and it uses the same KubeRay-on-EKS foundation as our multi-node Ray Serve walkthrough.
What you get without building anything
The Ray Train DLC is the PyTorch training image with Ray’s training libraries on top. The tag this sample pulls, public.ecr.aws/deep-learning-containers/ray:train-ml-cuda-v1.1, replaces a build that would otherwise look like this:
The networking row is the one most teams would least like to own. A mismatch there doesn’t produce an error. NCCL drops to TCP sockets, and the job runs to completion at a fraction of the speed. In the DLC, those libraries are built and tested together, and AWS gates each release on multi-node EFA and multi-node KubeRay training tests, which are exactly the paths this sample uses.
A few more properties remove work that you would otherwise do per cluster:
- No registry setup – The image is public, so nodes pull it without credentials and nothing is built or pushed on the way.
- No driver matching – On startup the entrypoint checks the host’s NVIDIA driver and, if it is older than the image’s CUDA, turns on the bundled CUDA forward-compatibility libraries automatically.
- No head-versus-worker images – The image has no fixed role. Whatever starts the container supplies the ray start command, so one tag serves every pod.
- No separate inference image family – Ray Train shares the ray ECR repository with the Ray Serve DLC and Ray LLM DLCs, under the train-ml tag prefix.
Configure it, don’t rebuild it
Everything that differs between one training job and the next is set around the image. The sample uses each of these mechanisms:
Packages per job. The image ships the HuggingFace training stack, so this sample’s job needs nothing added. When an experiment does need a library the image does not carry, Ray installs it for that job at submit time through runtime_env and the image is unchanged. No new image, no new tag.
Transport per job. The image defaults to FI_PROVIDER=efa, and its NCCL configuration excludes the loopback and Docker bridge interfaces. On EKS you point NCCL_SOCKET_IFNAME at the pod’s eth0. The sample sets all four networking variables in the manifest anyway, so the behavior is visible in one file and doesn’t depend on image defaults. Because these are plain environment variables, a single job can override them through its runtime_env.
Versions you choose. The sample pins train-ml-cuda-v1.1, the current minor release, so a cluster is reproducible from one run to the next. The vX.Y tags move with security rebuilds. The fully qualified vX.Y.Z tags stay frozen at the original release build, so pin a digest when you need a byte-identical run. The same image is also available from private ECR URIs.
How the cluster is laid out
All nodes sit in private subnets of one virtual private cloud (VPC) and reach the internet only through a NAT Gateway, which is how every node pulls the DLC from Amazon ECR Public and how workers download model weights from Hugging Face. One m7i.xlarge forms the system node group, hosting kube-system add-ons, the KubeRay operator and the Ray head. The head runs the job server on port 8265 and Ray’s control store on 6379, and it advertises no GPUs, so no training work lands on it. Jobs are submitted from inside the head pod with kubectl exec, through the EKS API, so port 8265 is never exposed outside the cluster.
Training runs on two g6.12xlarge instances, each with four NVIDIA L4 GPUs, 48 vCPUs, 192 GiB of memory, 40 Gbps of networking and one EFA interface. Each runs one Ray worker pod that claims all four GPUs and the EFA interface, and Ray Train places four workers in each pod: ranks 0–3 on one instance, 4–7 on the other. Because EFA traffic doesn’t cross Availability Zones, the scripts place both instances in a single zone that has been checked to offer g6.12xlarge.
What Ray Train does inside the container
The DLC provides the stack. Ray Train provides the part that would otherwise be launcher code. Multi-node PyTorch normally means starting a process per GPU on every host and giving each one its rank, the world size and a rendezvous address. With Ray Train you write the training loop as a function and pass it to TorchTrainer with a ScalingConfig. Ray starts one worker per GPU wherever the cluster has GPUs, forms the torch.distributed process group, and runs your function on each worker.
This sample asks Ray Train for FSDP, which splits the model’s weights, gradients and optimizer state across all eight GPUs rather than copying them to each one. Qwen2.5-1.5B in bfloat16 needs about 12 GB for those four tensors. Sharded eight ways, each L4 GPU holds roughly 1.5 GB. The price is communication: FSDP gathers each layer’s weights before using it and reduce-scatters gradients afterwards, so the whole model crosses between ranks on every step. Between the two instances, that traffic is what the DLC’s EFA stack carries.
Prerequisites
You must have the following in place before you start:
- Tools: AWS Command Line Interface (AWS CLI) v2 signed in to the target account, eksctl, kubectl, Helm, and envsubst from GNU gettext.
- Quota: 96 vCPUs of G-family On-Demand capacity for the two g6.12xlarge instances, in an AWS Region where at least one zone offers them. If you already hold an On-Demand Capacity Reservation for the instances, set CAPACITY_RESERVATION_ID instead. The scripts launch the node group into it and skip the capacity search.
- Budget: about $9.50 an hour while the GPU instances are up, covering both instances, the EKS control plane and the NAT Gateway.
A quick way to confirm that the CLI is pointed at the account that you expect:
The training code: One function and a trainer
code/train.py is the whole job, and it contains nothing specific to the container, the cluster or the network. Ray runs train_func once per GPU. It asks Ray for its rank, loads the model, hands it to prepare_model for FSDP, and runs an ordinary PyTorch loop:
code/train.py — the per-GPU function (abridged)
The driver joins the running cluster, counts its GPUs and starts the trainer. Because the worker count comes from the cluster, the same script runs unchanged on one node or many:
code/train.py — the driver (abridged)
The batches are random token IDs so that the run depends on nothing but the cluster. The sample is judged on whether real FSDP collectives crossed both nodes and the job completed. For a real fine-tune, replace the random tensor with a Ray Data or DataLoader pipeline. Nothing else in the function changes.
The manifest: The only place the DLC is configured
manifest/raycluster.yaml is where the image meets the cluster, and it is short because the image needs so little. The head’s only special setting is NVIDIA_VISIBLE_DEVICES=void, which stops the container runtime’s NVIDIA hook from failing on a node that has no GPU. The worker group holds everything else:
manifest/raycluster.yaml — worker group (abridged)
Read it as three decisions.
Which image: ${DLC_IMAGE}, unmodified, for both head and workers.
Which hardware: four GPUs and the node’s single EFA interface per pod. Requesting vpc.amazonaws.com/efa is what makes the EFA device plugin mount the interface into the container. Without it, NCCL can’t see EFA and falls back to sockets.
Which transport: four environment variables, so nothing is left to auto-detection.
Two smaller details keep a multi-node job stable. First, requests should equal limits, which places the workers in Kubernetes’ Guaranteed Quality of Service (QoS) class. This matters because a single throttled or evicted rank stalls the other seven. Second, num-gpus is derived from the same variable as the GPU request, so Ray’s view of capacity always matches Kubernetes’.
Bringing it up: One command for the infrastructure, then the job
Configuration lives in one place, scripts/env.sh: cluster name, Region, Kubernetes version, node types and counts, the image, pinned versions of KubeRay and both device plugins, and the training parameters. Export any of them to change it for a run. All five scripts are idempotent and safe to re-run. deploy_cluster.sh, deploy_node_group.sh, and deploy_ray_train_job.sh each prompt before creating or touching cluster resources. install_gpu_plugins.sh and install_kuberay.sh are reconcilers and run straight through.
./deploy_all.sh – Finds where the capacity actually is, then builds the infrastructure there.
g6.12xlarge capacity shifts between Regions and Availability Zones day to day, so this is the starting point rather than a Region guessed up front in env.sh. It first runs find_gpu_capacity.sh, which checks, per candidate Region: whether g6.12xlarge is offered in each Availability Zone, whether the account’s On-Demand G/VT-family vCPU quota has room for the requested node count, and the EC2 Spot placement score as a capacity signal. With --probe, which deploy_all.sh passes by default, it gets a definitive answer by creating and immediately canceling a real Capacity Reservation in each candidate zone. It then shows the plan and asks once:
Answering y runs, in order, deploy_cluster.sh, deploy_node_group.sh, install_gpu_plugins.sh, and install_kuberay.sh in the recommended Region, each skipping its own repeat confirmation because this one already asked (-y skips this prompt too, and --plan shows the plan without creating anything). If the GPU node group can’t launch in the best zone, the script removes it and tries the next zone the probe confi
Auto-fetched and translated by AI Gem Box. All rights belong to the original author.