3 对 NVIDIA DGX Spark 推进 MoE 推论的方法

更快的生成符从确定什么是极限推论性能开始。 在NVIDIA DGX Spark上的Ling-3.0-flash的批量大小一评价中，剖析显示解码是内存绑定的：读取模型加权驱动执行时间。 两种操作很突出——混合专家(MOE)的计算和产出层的预测可以产生整个模型的分数。

]]>

本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。

Generating tokens faster starts with identifying what limits inference performance. In the batch-size-1 evaluation of Ling-3.0-flash from Inclusion AI on NVIDIA DGX Spark, profiling showed that decoding was memory-bound: reading model weights drove execution time. Two operations stood out—the mixture-of-experts (MoE) computation and the output-layer projection that produces scores across the model’…

]]>

Auto-fetched and translated by AI Gem Box. All rights belong to the original author.