Prime Intellect拥有发布 Prime Mega MoE，是DeepSeek已熔化的Mega MoE内核的训练扩展。 兹克 1Z内核仅跑出前传。 Prime Intellect增加了一个快速，重叠的后传。 这使得被熔化的内核可用于训练，而不仅仅是推论。 在其质地-rl教练器中，Qwen3MOE模型的全面训练步骤运行得快到34%. 对火炬，收益达到55%。 对于训练混合专家模型的团队来说，这是直接削减GPU时数。
TL; DR 数据
- 大小：GPU内核，不是模型。 在Qwen3-30B-A3B和Qwen3-235B-A22B(128位专家，前8位路线)上被试。
- 运行于：NVIDIA Hopper (blockwise FP8)和 Blackwell (MXFP8) GPUs, BF16 两种上。 已公布的数字使用8×H200节点，32到128个GPU.
- 业绩：32%至34%的全面培训步骤比初级水平基线更快。 比火炬快48%到55%.
- 最佳：3.11x更快地向前传出Qwen3-30B-A3B MoE地层 (120至3.9米).
- 最差：1.24x后传Qwen3-235B-A22B MoE层(30.5米到24.6米).
何为"超级精英"?
Prime Mega MoE是一个由Prime Intellectric提供训练的有引信，相重叠的MOE内核。 它建立在DeepSeek的开源 DeepGEMM库中的Mega MoE内核上。
为什么教育部培训的通信是受约束的？
在MOE模型中，每个符号都流向一小群专家。 这些专家坐在不同的GPU上。 所以每一层都发出信号，运行专家数学，并发送结果回来。 这发生在前传，再发生在后传。
在标准设置中，这些是单独的操作。 GPU排序符并等于运输层。 然后它计算，手结果回来，再等待。 那些等待是闲置的GPU时间。
熔接的内核是怎样工作的？
DeepSeek的"Mega MoE"于2026年4月被添加到DeepGEMM中，取而代之的是这个链条，取而代之的是一内核呼叫。 它将专家的平行发送，2个专家的线性层相接而成，并结合在一起。 虽然一个象征团体正在过境，但内核计算一个已经到达的团体。 它还通过使用只使用GPU的刮痕缓冲来跳过中间的分数。 但是，它只是提前执行。
Prime Intellect写了缺失的后向内核，同样重叠。 所支持的类型有：BF16,MXFP8 (仅Blackwell)和阻断式FP8 (仅Hopper). Prime-rl配置还曝光了fp8 bwd模式：BF16向前，FP8后向。
超级大魔王的动作有多快？
以下所有数字均由Prime Intellect自报。 训练步骤使用8×H200节点，FP8,16,384-token序列并全面启动检查。 每个结果都是步骤6至20的平均值。
完整的训练步骤，秒( 下等更好):
针对火炬泰坦，30B型机车的吞吐量从5,337升至7,886个令牌/s/GPU. 在235B型机车上，由1,094升至1,697. primer-rl基线使用火炬全能外加DeepGEMM FP8专家。
单 MoE 层，毫秒(1个节点，8×H200,EP=8,128个专家，前8名，随机路由):
模式是明确的。 前进收益较大，后退收益较小。 标题“最多3.11x”仅指前方通行证。
如何与其他已安装引信的教育部选项进行比较？
火炬组织41%的收益来自不同的设置(B200,671B模型，BF16基线). 它不具有直接可比性。
开发者如何使用它？
Prime Mega MoE是Preme-rl的调度后端。 设置类型= [trainer.model.moe.dispatch]下"mega moe". 从 bf16 选择精度，mxfp8, fp8 或 fp8 bwd. 训练内核飞船在从质内核预先建造的DeepGEMM轮中。 股票DeepGEMM的建设缺少它们。
关键外卖
- Prime Mega MoE 新增是DeepSeek的Mega MoE的相接后传。
- 完整的培训步骤比基础基线快32%至34%。
- 在Qwen3MOE型号上，步数为48%至55%。
- 前期收益达到3.11x，但后期收益下降至大层1.24x。
- 它以"最上等"为主线进行运输
请检查access-date=中的日期值 (帮助) Prime Mega MoE博客文章，并加入我们的通讯。 等等，你在电报里吗？现在你也可以加入我们的电报。
需要和我们合作推广您的 GitHub Repo OR Hugging Face Page OR Product Release OR Webinar等吗？联系我们
Asif Razzaq是Marktechpost AI Media Inc.的首席执行官。 作为一个有远见的企业家和工程师，阿西夫致力于利用人工智能的潜力为社会造福。 他最近的努力是人工智能媒体平台的发布，该平台以深入报道机器学习和深层学习新闻而出名，这些新闻在技术上既可靠又为广大观众所容易理解。 该平台每月有超过200万次观看，表明其受观众欢迎。
本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Prime Intellect has released Prime Mega MoE, a training extension of DeepSeek’s fused Mega MoE kernel. DeepSeek’s kernel only ran the forward pass. Prime Intellect added a fast, overlapping backward pass. That makes the fused kernel usable for training, not just inference. In its prime-rl trainer, full training steps on Qwen3 MoE models run up to 34% faster. Against TorchTitan, the gain reaches 55%. For teams training mixture-of-experts (MoE) models, that is a direct cut in GPU hours.
TL;DR
- Size: A GPU kernel, not a model. Tested on Qwen3-30B-A3B and Qwen3-235B-A22B (128 experts, top-8 routing).
- Runs on: NVIDIA Hopper (blockwise FP8) and Blackwell (MXFP8) GPUs, BF16 on both. Published numbers use 8×H200 nodes, 32 to 128 GPUs.
- Performance: 32% to 34% faster full training steps than the prime-rl baseline. 48% to 55% faster than TorchTitan.
- Best: 3.11x faster forward pass on a Qwen3-30B-A3B MoE layer (12.0 ms to 3.9 ms).
- Worst: 1.24x backward pass on a Qwen3-235B-A22B MoE layer (30.5 ms to 24.6 ms).
What is Prime Mega MoE?
Prime Mega MoE is a fused, overlapping MoE kernel for training from Prime Intellect. It builds on the Mega MoE kernel in DeepSeek’s open-source DeepGEMM library.
Why is MoE training communication-bound?
In an MoE model, each token goes to a small subset of experts. Those experts sit on different GPUs. So every layer sends tokens out, runs expert math, and sends results back. This happens in the forward pass and again in the backward pass.
In a standard setup, these are separate operations. The GPU sorts tokens and waits on the transport layer. Then it computes, hands results back, and waits again. Those waits are idle GPU time.
How does the fused kernel work?
DeepSeek’s Mega MoE, added to DeepGEMM in April 2026, replaces that chain with one kernel call. It fuses expert-parallel dispatch, the 2 expert linear layers, SwiGLU, and combine. While one token group is in transit, the kernel computes on a group that already arrived. It also skips intermediate tensors by using GPU-only scratch buffers. But it only shipped a forward implementation.
Prime Intellect wrote the missing backward kernel, also overlapping. Supported types are BF16, MXFP8 (Blackwell only) and blockwise FP8 (Hopper only). The prime-rl config also exposes an fp8_bwd mode: BF16 forward, FP8 backward.
How fast is Prime Mega MoE?
All numbers below are self-reported by Prime Intellect. Training steps use 8×H200 nodes, FP8, 16,384-token sequences and full activation checkpointing. Each result is the mean of steps 6 to 20.
Full training step, seconds (lower is better):
Against TorchTitan, throughput rose from 5,337 to 7,886 tokens/s/GPU on the 30B model. On the 235B model, it rose from 1,094 to 1,697. The prime-rl baseline uses torch all-to-all plus DeepGEMM FP8 experts.
Single MoE layer, milliseconds (1 node, 8×H200, EP=8, 128 experts, top-8, random routing):
The pattern is clear. Forward gains are large, and backward gains are smaller. The headline “up to 3.11x” refers to the forward pass only.
How does it compare with other fused MoE options?
TorchTitan’s 41% gain comes from a different setup (B200, 671B model, BF16 baseline). It is not directly comparable.
How do developers use it?
Prime Mega MoE is a dispatch backend in prime-rl. Set type = "mega_moe" under [trainer.model.moe.dispatch]. Choose precision from bf16, mxfp8, fp8 or fp8_bwd. The training kernels ship in a prebuilt DeepGEMM wheel from prime-kernels. The stock DeepGEMM build lacks them.
Key Takeaways
- Prime Mega MoE adds a fused, overlapping backward pass to DeepSeek’s Mega MoE.
- Full training steps run 32% to 34% faster than the prime-rl baseline.
- Against TorchTitan, steps run 48% to 55% faster on Qwen3 MoE models.
- Forward gains reach 3.11x, but backward gains drop to 1.24x on large layers.
- It ships in prime-rl as the mega_moe dispatch backend.
Check out the Prime Mega MoE blog post, the announcement on X, and the prime-rl repo. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us
Asif Razzaq is the CEO of Marktechpost AI Media Inc.. As a visionary entrepreneur and engineer, Asif is committed to harnessing the potential of Artificial Intelligence for social good. His most recent endeavor is the launch of an Artificial Intelligence Media Platform, Marktechpost, which stands out for its in-depth coverage of machine learning and deep learning news that is both technically sound and easily understandable by a wide audience. The platform boasts of over 2 million monthly views, illustrating its popularity among audiences.
Auto-fetched and translated by AI Gem Box. All rights belong to the original author.