OCT. 20, 2026 (英语).
在过去几年中，Meta的火炬号成为了PyTorch生态系统用于大规模基因化AI模型培训的首要清洁室参考平台。 TorchTitan完全以PyTorch-内生缩放原始人为基础，为训练前旗舰稠密和混合型专家(MoE)建筑提供未支配，高度模块化的展示，如Llama 3,Qwen 3和FLUX.
历史上，想在Google Cloud TPU上训练PyTorch模型的开发者面临一些挑战：要么进行完整的代码库重写为JAX，要么与遗留的火炬-xla相搏。 这种模式已经正式改变。 随着火炬TPU的发布，Cloud TPU现在具有了一种本土后端(torch.device ("tpu"))直接被集成到PyTorch的核心接口中。 允许培训脚本使用 TPU 现在只需要对设备初始化代码进行最小的修改。 然而，提供设备相容性只是战斗的一半，开发人员也需要高性能的生产级，分布式的培训食谱。 TPU上的"火炬"就在这里出现。 TorchTitan TPU是Meta的TorchTitan执行的叉子，被改造为能进行Google Cloud TPU的模型训练。 该分叉可在https://github.com/google-pytorch/torchtitan/上查阅。
Google和Meta共同开发的TPU本土执行口令直接在火炬铁道内部，与其要求工程团队发明自定义的TPU硬化策略或手动低级XLA通信从头通过。 火炬Titan作为Google云TPU上缩放PyTorch的定型车。
火炬公司为训练前大型密集型号(如Llama 3 70B/405B和Qwen 3 8B/32B)，专家混合型(如Qwen 3 235B MoE)以及扩散变压器(如FLUX.1 12B)提供经过战斗测试的正式食谱。
除了初步的预训外，火炬公司还充当培训后管道内的可伸缩教练，如VERL用于集团相对政策优化(GRPO)和强化学习(RL)相接。 在这些工作流程中，火炬处理参数更新步骤，同时将重量与高通量服务引擎同步，如vLLM-TorchTPU.
TorchTitan中的模型定义用纯的，单发的PyTorch (nn.Module)来写，没有CUDA特有的代码。 集体要么被可堆叠的框架层(如FSDP)所插入，要么被宣示性硬化计划所插入(如lator-平行主义)，要么被作为后端特定插件(如MOE层)所插入。 火炬TPU支持 C10D 集体 API 和 PyTorch 设备Mesh 抽象。 这让TPU路径一起可以选择哪些并行性可以申请一个模型，哪些执行可以使用。
火炬泰坦从配置树上构建出一个模型，在构建模型之前可以重写出该树。 这提供了一个干净的方法，可以将一个模型的单个块替换为在TPU上运行较好的版本，而无需触及模型定义或共享的教练回路。
我们积极地使用这个。 TPU配方用Splash Notice来取代标准注意，这是一个Pallas内核，它处理因果，滑动-窗口和被打包的文件口罩而未实现注意矩阵；将最终投影和交叉内向丢失替换为一个从未将完整对数的Pallas内核来写入HBM；并用分组的Pallas内核来取代MO专家matmuls. 每个内核是一个JAX函数被火炬TPU的火炬.tpu.pallas.jax op装饰器转换为PyTorch自定义op，这是新内核需要的所有胶水。 每个被替换者都是一面配置旗，因此一个配方可以打开，一个测试可以比作它所替换的平地PyTorch路径。 内核可以来自tokamax，用于TPU和GPU的OpenXLA内核库。
FSDP2围绕GPU执行模式而建。 其全相加和减相分量由模块钩子作为明确的集体调用来发布，与计算重叠来自在专用CUDA流上运行，与事件同步。 编译者看不到这些：集体坐落在被追踪区域之外，因此在TPU上是一盏火炬。 每个FSDP钩上编译区域断块，一个全步图都完全无法捕捉。 火炬TPU只提供不op流和事件对象，以便FSDP2运行；XLA对其没有用处，因为它从数据依赖性来调度工作。
SimpleFSDP取而代之的是编译器的视图。 参数sharding是一款DTensor参数化：读取一个模块的重量进行从被sharded到复制的再分配，这是全加法，其自发分解后是还原分解。 没有钩子，也没有明确的集体呼号，所以两个集体在所追踪的图中都是普通的节点，无论该图是每块火炬。 编译区域或图形导师的单个 make fx步骤。 XLA编译器在需要的matmuls之前排出全加太，并和后向计算器相重叠，在一个程序中。 混合精度是同一机制的一部分，这一政策将集合d型(bf16参数)和减少d型(fp32或bf16梯度)设定在该再分配上。 混合硬化为多晶片在二维网格上是相同的代码：在ICI上方的切片内有硬化，在DCN上方的切片上有replicate.
FLUX上，这让火炬与JAX MaxDifuncation的性能均分达到99%:SimleFSDP块的每层HLO是一个单一的模块，其重量为全加太，matmuls，注意力和规范融合在一起，这就是XLA构建程序的外形是为了优化。
许多PyTorch训练员或者用CUDA图来急切地跑去剪接发布高架，或者应用火炬。 编译到被隔离的区域，一般一次一个变压器块。 火炬TPU尚不支持PyTorch火炬。 流式抽象，这限制了将热心的FSDP2与分层火炬相混合的性能。 编译。 SimpleFSDP帮助用与编译相兼容的方式表达参数sharding，无论是使用分层火炬。 编译或全图抓取。
火炬泰坦的"地图导师"提供了完整的训练步骤捕捉(向前，丢失和后向)，用PyTorch的make fx跟踪器提前生成出一幅FX图，跳过AOTAutograd分出，因此整个计算都可见于XLA. GraphTrainer还运行了优化通道：执行基于图表的活化内存政策，决定如何保存，再计算或卸载。 列车步骤随后作为单XLA程序执行。
这是XLA的型号所建，它使得PyTorch关于TPU的训练在概念上与JAX相同，因为JAX的整节列车台阶是一个被拼接的功能。 将整个步骤作为一图来拥有也是使得大型模型编译技术成为可能的方法：将the decoder分层作为单回路体进行扫描，使编译时间和内存停止以深度增长；将削减分出，该分出步骤单独编译出前向和后向程序，并卸载活化，以在两者之间主机存储。 在Qwen3 235B上，这条路达到29°MFU.
优化者也可以加入图表：虽然fund adam优化器性能很好，但功能性的AdamW更新可以被追溯到列车-step(功能-adamw step)，并确保迭接器在第一个前置操作和重写-后置之间没有主机-侧工作。 在Qwen3-8B上，这条路径达到了51-52%的MFU.
下方的每一把杠杆都由"火炬Titan TPU"优化指南来描述，几乎所有的都已经是wired入食谱了。 对于大多数模型来说，在TPU上获得一个好MFU是对现有食谱而非新代码的配置变化序列。 命令很重要，因为每一步都改变了下一步的措施。 火炬(TorchTitan)中的许多优化都可以作为可即时使用的构件来提供，可以通过旗子翻转来启用(见下文插图).
1. 确定基线
- 选择该步骤如何执行：急切、编译或图形导师。 Eager模式在TPU上运行了PyTorch模式，几乎没有变化，这使得它成为将一个模型带出并调试的正确地点。 这不是一个性能配置：每个操作都是自己的程序，TPU闲置在发布之间。 火炬TPU的被推迟并被放入的急切模式(-tpu config.eager mode=DEFER AND FUSE)通过让XLA熔接跨越了业务边界来回收部分代码，而食谱则保持其覆盖所编译区域以外的任何内容。 启用每块火炬。 compile(-compile.enable)是将调试配置变成一个能表演的：每个变压器块会变成一个单一的XLA程序，因此其操作被熔接了，发布一次而不是一次一个，并与块的集体一起排出。 块必须追踪，它排除了里面依赖数据的 Python，这就是模型放弃。 Graph Trainer走得更远(-tpu config.use graph trainer)，让整个培训步骤，前进，丢失，后退和优化的更新，被捕获为一个程序。 整个步骤必须追踪，你得到的是一个步骤，没有急切的胶水，加上大 -- 第5步中的模型技术。
- Shard with SimpleFSP and a mix-精密政策。 SimpleFSDP在后继图(-expareism.use simple fsdp)中表达为集体的硬化，因此编译器可以将全加法和减分法排在它们周围的矩阵相乘上。 FSDP2本身被优化为急切的性能，但也使得难以被追踪到一个图表中。 与 bf16 参数和 fp32 梯度还原对等：计算以 bf16 为单位运行，而主重和还原则停留在 fp32 为单位，这是设置每个TPU 食谱用途。
- 对于大型模型(~100B+参数)，允许扫描或分解图表，并卸载活化。 一个未滚动的阶梯图会随着深度的增大而停止编译，它会让每层所保存的输入都活到后向通过。 扫描多层(-tpu config.use scan layers)将the decoder作为单回路体运行，因此编译时间和内存会随着深度而停止生长，并卸下一个堆起的活化到主机内存；这是Qwen3 235B和Llama 3 405B取走的路径。 图表分解器(-tpu config.use graph split)保持层未滚动，但会向后编译，并且可选地将每个层组编译为单独的程序，其中激活器在两者之间卸载。 两者都建立在"图形导师"之上。
2. Tune计算和Kernals
- 打开帕拉斯内核喷出注意(-splash attention 内核.use splash attention 内核) 从未实现注意矩阵，并处理因果，滑入窗口和被打包的文档口罩；在长序列长度时，它最为重要。 被保险的线性横切损耗(PallasCrossEntropyLoss in the cude)计算出LM-头相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接的相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接相接它的主要价值是内存，下一步变为通量。 对于MOE模型，分组的专家matmul和对接的门/上内核(use pallas paired group mm)用每层一个内核来取而代之的绕行专家。 每一个都是食谱中的开关，每个用它所取代的平地PyTorch路径测试。
- 内核的瓦片大小喷出注意力，被保险的丢失和MOE内核各暴露出其块大小(sa blockQQ和好友)，而默认是一个起点而不是最佳。 在范围VMEM预算(xla tpu scoped vmem limit kib)范围内，将它们用于模型形状和TPU生成，并期望在Trilium上调取的值不会被结转到Ironwood. 食谱中包含每个模型和地貌的调值。
3. 优化内存和批量大小
- 把货抬高在小尺度上，每芯片批量是单个最大的杠杆，因为融资是每个发布程序完成的工作。 将它推到模型被HBM捆绑；被保险的丢失和被关注的内核是释放内存这样做的方法。 在大片上，请记住，全球批量是每芯片批量乘以芯片计数，所以大小可以达到模型所能使用的程度，而不是芯片所能持有的程度。
4. Tune编译器，C
本文由 AI宝盒自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
OCT. 20, 2026
Over the past years, Meta’s TorchTitan has emerged as the PyTorch ecosystem’s premier clean-room reference platform for large-scale generative AI model training. Built entirely on PyTorch-native scaling primitives, TorchTitan provides an unencumbered, highly modular showcase for pre-training flagship dense and Mixture-of-Experts (MoE) architectures like Llama 3, Qwen 3, and FLUX.
Historically, developers looking to train PyTorch models on Google Cloud TPUs faced some challenges: either undertake a complete codebase rewrite into JAX, or grapple with legacy torch_xla. That paradigm has officially shifted. With the launch of TorchTPU, Cloud TPUs now feature a native backend (torch.device("tpu")) integrated directly into PyTorch's core interface. Enabling a training script to use TPUs now only requires minimal changes to your device initialization code. Providing device compatibility, however, is only half the battle, developers also need high performance production-grade, distributed training recipes. This is where TorchTitan on TPU comes in. TorchTitan TPU is a fork of Meta’s TorchTitan implementation that is adapted to enable model training on Google Cloud TPUs. The fork is available at https://github.com/google-pytorch/torchtitan/
Rather than requiring engineering teams to invent custom TPU sharding strategies or hand-craft low-level XLA communication passes from scratch, Google and Meta co-developed TPU-native execution passes directly inside TorchTitan. TorchTitan serves as the definitive vehicle for scaling PyTorch on Google Cloud TPUs.
TorchTitan provides official, battle-tested recipes for pre-training large dense models (such as Llama 3 70B/405B and Qwen 3 8B/32B), Mixture-of-Experts (such as Qwen 3 235B MoE), and diffusion transformers (such as FLUX.1 12B).
Beyond initial pre-training, TorchTitan serves as the scalable trainer inside post-training pipelines like VeRL for Group Relative Policy Optimization (GRPO) and Reinforcement Learning (RL) alignment. In these workflows, TorchTitan handles the parameter update step while synchronizing weights with high-throughput serving engines like vLLM-TorchTPU.
Model definitions in TorchTitan are written in pure, single-device PyTorch (nn.Module) with no CUDA-specific code. Collectives are inserted either by a composable framework layer (e.g. FSDP), by declarative sharding plans (e.g. tensor-parallelism), or as backend-specific plugins (e.g. MoE layers). TorchTPU supports C10D collective APIs and the PyTorch DeviceMesh abstraction. Together, this allows the TPU path to choose which parallelisms to apply for a model, and which implementations to use.
TorchTitan builds a model from a configuration tree, and that tree can be rewritten before the model is constructed. This offers a clean way to replace individual blocks of a model with versions that run better on TPU, without touching the model definition or the shared trainer loop.
We use this actively. The TPU recipes replace standard attention with Splash Attention, a Pallas kernel that handles causal, sliding-window and packed-document masks without materializing the attention matrix; replace the final projection and cross-entropy loss with a fused Pallas kernel that never writes the full logits tensor to HBM; and replace MoE expert matmuls with grouped Pallas kernels. Each kernel is a JAX function turned into a PyTorch custom op by TorchTPU's torch.tpu.pallas.jax_op decorator, which is all the glue a new kernel needs. Each replacement is a config flag, so a recipe can turn it on, and a test can compare it against the plain PyTorch path it replaces. Kernels can come from tokamax, the OpenXLA kernel library for TPU and GPU.
FSDP2 is built around the GPU execution model. Its all-gathers and reduce-scatters are issued from module hooks as explicit collective calls, and overlap with compute comes from running them on dedicated CUDA streams synchronized with events. None of that is visible to a compiler: the collectives sit outside the traced region, so on TPU a torch.compile region breaks at every FSDP hook and a whole-step graph cannot be captured at all. TorchTPU provides no-op stream and event objects only so that FSDP2 runs; XLA has no use for them, since it schedules work from data dependencies.
SimpleFSDP takes the compiler's view instead. Parameter sharding is a DTensor parametrization: reading a module's weight performs a redistribute from sharded to replicated, which is the all-gather, and its autograd backward is the reduce-scatter. There are no hooks and no explicit collective calls, so both collectives are ordinary nodes in the traced graph, whether that graph is a per-block torch.compile region or the Graph Trainer's single make_fx step. The XLA compiler schedules the all-gathers ahead of the matmuls that need them and overlaps the reduce-scatters with the backward compute, inside one program. Mixed precision is part of the same mechanism, a policy that sets the gather dtype (bf16 parameters) and the reduce dtype (fp32 or bf16 gradients) on that redistribute. Hybrid sharding for multislice is the same code on a 2-D mesh: shard within a slice over ICI, replicate across slices over DCN.
On FLUX this is what brought TorchTitan to 99% performance parity with JAX MaxDiffusion: the per-layer HLO for a SimpleFSDP block is a single module with the weight all-gathers, matmuls, attention and norms fused together, which is the shape of program XLA was built to optimize.
Many PyTorch trainers either run in eager mode with CUDA graphs to cut launch overhead, or apply torch.compile to isolated regions, typically one transformer block at a time. TorchTPU does not yet support the PyTorch torch.Stream abstraction, which limits the performance of mixing eager-based FSDP2 with layer-wise torch.compile. SimpleFSDP helps by expressing parameter sharding in a way that is compatible with compilation, whether using layer-wise torch.compile or full-graph capture.
TorchTitan's Graph Trainer provides full training-step capture (forward, loss and backward), producing one FX graph ahead of time with PyTorch's make_fx tracer, skipping AOTAutograd partitioning, so the whole computation is visible to XLA). GraphTrainer also runs optimization passes: to implement graph-based activation memory policy that decides what to save, recompute or offload. The train step then executes as a single XLA program.
This is the model XLA was built for, and it makes PyTorch training on TPU conceptually the same as JAX, where the whole train step is one jitted function. Owning the whole step as one graph is also what makes the large-model compilation techniques possible: scan over layers, which runs the decoder layers as a single loop body so that compile time and memory stop growing with depth, and the graph splitter, which cuts the step into separately compiled forward and backward programs with activations offloaded to host memory in between. On Qwen3 235B this path reaches 29%+ MFU.
The optimizer can join the graph as well: While the fused_adam optimizer is very performant, a functional AdamW update can be traced into the train-step (functional_adamw_step) and ensures an iteration has no host-side work between the first forward op and the weight write-back. On Qwen3-8B this path reaches 51-52% MFU.
Every lever below is described in the TorchTitan TPU optimization guide, and nearly all of them are already wired into the recipes. For most models, getting a good MFU on TPU is a sequence of configuration changes on an existing recipe rather than new code. The order matters, because each step changes what the next one measures. A lot of optimizations in TorchTitan are available as ready-to-use building blocks that can be enabled by a flag flip (see illustration below).
1. Establish a Baseline
- Choose how the step executes: eager, compile, or Graph Trainer. Eager mode runs PyTorch models on TPU with virtually no changes, which makes it the right place to bring a model up and debug it. It is not a performance configuration: every op is its own program and the TPU idles between launches. TorchTPU's deferred-and-fused eager mode (--tpu_config.eager_mode=DEFER_AND_FUSE) recovers some of that with no code change by letting XLA fuse across op boundaries, and the recipes keep it on to cover whatever sits outside compiled regions. Enabling per-block torch.compile (--compile.enable) is what turns a debug configuration into a performant one: each transformer block becomes a single XLA program, so its ops are fused, launched once instead of one at a time, and scheduled together with the block's collectives. The block has to trace, which rules out data-dependent Python inside it, and that is all the model gives up. Graph Trainer goes further (--tpu_config.use_graph_trainer) and lets the whole training step, forward, loss, backward and optimizer update, be captured as one program. The whole step has to trace, and what you get back is a step with no eager glue at all, plus the large-model techniques in step 5.
- Shard with SimpleFSDP and a mixed-precision policy. SimpleFSDP expresses sharding as collectives inside the traced graph (--parallelism.use_simple_fsdp), so the compiler can schedule all-gathers and reduce-scatters against the matrix multiplies around them. FSDP2 itself is optimized for eager performance, but that also makes it difficult to trace into a graph. Pair it with bf16 parameters and fp32 gradient reduction: compute runs in bf16 while master weights and reductions stay in fp32, which is the setting every TPU recipe uses.
- For large models (~100B+ parameters), enable scan or split the graph, and offload activations. One unrolled step graph stops compiling as depth grows, and it keeps every layer's saved input live until the backward pass. Scan over layers (--tpu_config.use_scan_layers) runs the decoder as a single loop body, so compile time and memory stop growing with depth, and offloads the one stacked activation to host memory; this is the path Qwen3 235B and Llama 3 405B take. The graph splitter (--tpu_config.use_graph_split) keeps the layers unrolled but compiles forward and backward, and optionally each layer group, as separate programs with activations offloaded in between. Both build on Graph Trainer.
2. Tune Compute and Kernals
- Turn on the Pallas kernels. Splash attention (--splash_attention_kernel.use_splash_attention_kernel) never materializes the attention matrix and handles causal, sliding-window and packed-document masks; it matters most at long sequence lengths. The fused linear cross-entropy loss (PallasCrossEntropyLoss in the recipe) computes the LM-head matmul, softmax and loss one vocabulary tile at a time, so the full logits tensor is never written to HBM. Its main value is memory, which the next step turns into throughput. For MoE models, the grouped expert matmul and the paired gate/up kernel (use_pallas_paired_grouped_mm) replace a loop over experts with one kernel per layer. Each of these is a switch in the recipe, and each is tested against the plain PyTorch path it replaces.
- Tune the kernel tile sizes. Splash attention, the fused loss and the MoE kernels each expose their block sizes (sa_block_* and friends), and the defaults are a starting point rather than an optimum. Tune them for the model shape and the TPU generation, within the scoped VMEM budget (xla_tpu_scoped_vmem_limit_kib), and expect values tuned on Trillium not to carry over to Ironwood. The recipes carry the tuned values for each model and topology.
3. Optimize Memory and Batch Size
- Raise the batch. At small scale the per-chip batch is the single largest lever, because it raises the work done per launched program. Push it until the model is HBM-bound; the fused loss and the attention kernel are what free the memory to do so. On a large slice, remember that the global batch is the per-chip batch times the chip count, so size it to what the model can use, not to what the chips can hold
4. Tune Compiler, C
Auto-fetched and translated by AI Gem Box. All rights belong to the original author.