人工智能
在Amazon EKS和SageMaker HyperPod EKS上使用NVIDIA Nemotron模型的加速代理AI
NVIDIA 尼莫特龙 3.5 超级VL 克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克克 AWS上已有闪电模型。 它是一个120B参数的专家混合模型，每个符号激活12B参数。 它将推理和工具使用与文本，图像，文档，和视频理解结合起来，因此一个模型既服务于文本，也服务于视觉任务。 这个帖子探索了它的架构并使用了大小写，然后是显示如何在Amazon弹性Kubernetes服务(Amazon EKS),Amazon EKS自动模式和Amazon SageMaker HyperPod EKS上部署。
关于尼莫特龙3.5超级VL
NVIDIA 尼莫特龙 3.5 Super VL是一款由尼莫特龙3 Ultra蒸馏而成的开放式多式模型。 NVIDIA发布权重，培训数据集，以及培训后食谱，这样您就可以自定义特定域工作量的模型，并在您控制的基础设施上运行。 模型细节：
- 120B总参数，每个符号12B活性。
- 混合Mamba-2,MOE，和注意分层与RADIO视觉编码器。
- 上下文长度可达1M个令牌。
- 输入：文本、图像、文档和视频。
- 输出：文本。
- BF16和NVFP4检查站。
NVIDIA报告，赫尔梅斯法官座席为63.44%,LangChain agent评价为87.05%,CharXiv图理基准测试为80.8%. (技术报告)
示范结构
Nemotron 3.5 Super VL使用混合的Mamba-Transformer MoE架构，将Mamba-2，注意力相接，并有专家层。 Sparse路由激活了120B参数中每令牌的12B，因此每令牌计算与更小的模型相匹配，而GPU内存仍然能保持全重。 Mamba-2地层在生成过程中承载了恒定大小的状态，因此它们的内存不会像KV缓存那样以序列长度生长出，这让长长的上下文可以承受。 递归共享多托肯预测(MTP)头支持本土投机解码。
使用大小写
Nemotron 3.5 Super VL支持agent工作流程将推理和视觉理解结合起来：
- 编码：测试和验证代码，分析错误，推荐修正。
- 网络研究：相互参照，并核实通过搜索工具返回的来源。
- 文件情报：阅读表格、图表和页面布局，以编制摘要。
- 网络安全：查明脆弱性和分类威胁。
- 视频理解：搜索、内容、审查和事件分析。
尼莫特龙 3.5 Super VL通过NVIDIA参考执行，烹饪本和NVIDIA Agent工具包与主要开放的agent框架和管弦系统相融合。 支持框架的完整列表见NVIDIA开发者博客。
最小 GPU 配置
在AWS部署
本节概述架构、验证和绩效评价办法。 aws-do-eks项目载有部署Nemotron 3.5 Super VL和Nemotron 3 Ultra在AWS上的详细走道，包括先决条件、集群创建、模型重量下载、部署、测试、基准和清理。 同样的模式也可以被应用于NVIDIA尼莫特龙家族中的其他模型。
部署结构
参考执行运行于标准Amazon EKS,EKS自动模式和SageMaker HyperPod EKS. AWS在EKS自动模式下代表您管理集群中的缩放，GPU，和全民教育支持。 用于模型部署的Kubernetes表单对每个管弦乐器选项都是相同的。 每个集群都有一个主控Kubernetes API的控制平面和一个CPU和GPU节点的数据平面。 CPU节点运行系统和测试舱，而GPU节点运行模型。 Amazon弹性容器注册(Amazon ECR)存储了容器图像，一个用于Lustre卷的共享Amazon FSx持有模型重量，弹性纤维适配器(EFA)连接了GPU节点。 下图显示为标准Amazon EKS集群。
NVIDIA 时速 Dynamo服务于在一个单一节点上汇总的模型，或者在不同的节点上以预填和解码来分解。 对于视觉语言模型，图像/视频编码阶段还可以按照自己的节点进行分解。
审定和业绩
这种架构在p6-b200.48x大(8×NVIDIA B200 GPU)和p5en.48x大(8×NVIDIA H200 GPU)两个实例上都得到了验证。 您可以用 NVIDIA 集体通信库(NCCL)验证集群及其网络全减微-基准测试。
在NCCL测试运行期间，看到所有GPU的利用率和所有全民教育适配器的流量，就证明集群正在按预期运行。
您可以用 NVIDIA aiperf 工具测量一个在集群上部署的模型的性能。
在左上方的平板显示键性能指标(KPIs)中，aiperf输出像时间到第一令牌(TTFT)，相接纬度(ITL)，和每秒的令牌(TPS)的吞吐量。 通过在不同货币级别进行这些测量，我们可以研究长期性和吞吐量如何取决于同时提出的请求的数量。 这些信息对于周密规划能力和示范部署地形至关重要。 aws-do-eks容器提供可视化GPU和全民教育利用的自动化，以及使用单一命令进行Aiperf和通货扫描。
前面的货币扫描图揭示认为，Nemotron 3.5 Super VL的推论具有较低的TTFT，在两个节点的汇总部署和分解部署之间也大致相同。
这样的图表在作出部署决定时是有用的。 绘制您作为水平线的延迟预算，例如TTFT低于600毫秒(约一秒)，而ITL低于100毫秒每令牌。 两行下最右的点是这一地貌的一复制品的吞吐量可以在服务级目标(SLO)范围内服务，这里为c32，或每秒约320个活物。 要规划容量，将目标符每秒除以这个每个replica的吞吐量，并找到要部署的预填解码对的数量。 在变化后(精度，TP/PP形状，汇总与分解，新迪纳摩或vLLM放出)重跑同一次扫荡，使取舍立即可见。 一个左右移动的曲线就是胜利。 由于TTFT和ILFT是分开绘制的，它也告诉你要标出哪一边：上升的ILFT 称会增加解码能力，上升的TTFT 称会增加预填能力。
结论
这篇文章涵盖了在Amazon EKS,EKS Auto Mode，以及SageMaker HyperPod EKS和NVIDIA Dynamo及vLLLM上服务NVIDIA Nemotron模型的架构和性能评价模式。 要开始，在自己的账户中部署Nemotron 3.5 Super VL，使用参考执行。 关于EKS项目的AI中也有更多的推论蓝图。 为了获得更多关于EKS推论的亲身经验，登记参加我们即将举行的讲习班。
鸣谢
作者们特别感谢参与编写、验证、审查和出版这些内容的所有人，包括但不限于：Eliuth Triana、刘家红、Kshitiz Gupta、Abhijeet Patil、Mike Stefaniak、Chris Splinter、Shidal Joshi、Aditya Ramakrishnan、Natarajan Kumar、Anish Maddipoti。
本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Artificial Intelligence
Accelerate agentic AI with NVIDIA Nemotron models on Amazon EKS and SageMaker HyperPod EKS
NVIDIA Nemotron 3.5 Super VL joins the NVIDIA Nemotron 3 Nano, Nano Omni, Super, Ultra, and Nemotron 3.5 Lightning models already available on AWS. It’s a 120B-parameter mixture-of-experts (MoE) model that activates 12B parameters per token. It combines reasoning and tool use with text, image, document, and video understanding, so one model serves both text and visual tasks. This post explores its architecture and use cases, then shows how to deploy it on Amazon Elastic Kubernetes Service (Amazon EKS), Amazon EKS Auto Mode, and Amazon SageMaker HyperPod EKS.
About Nemotron 3.5 Super VL
NVIDIA Nemotron 3.5 Super VL is an open multimodal model distilled from Nemotron 3 Ultra. NVIDIA publishes the weights, training datasets, and post-training recipes, so you can customize the model for domain-specific workloads and run it on infrastructure you control. Model details:
- 120B total parameters, 12B active per token.
- Hybrid Mamba-2, MoE, and attention layers with a RADIO vision encoder.
- Context length up to 1M tokens.
- Input: text, image, document, and video.
- Output: text.
- BF16 and NVFP4 checkpoints.
NVIDIA reports 63.44 percent on Hermes Bench and 87.05 percent on LangChain agent evaluations, and 80.8 percent on the CharXiv chart-reasoning benchmark. (technical report)
Model architecture
Nemotron 3.5 Super VL uses a hybrid Mamba-Transformer MoE architecture, interleaving Mamba-2, attention, and expert layers. Sparse routing activates 12B of the 120B parameters per token, so compute per token matches a much smaller model while GPU memory still holds the full weights. Mamba-2 layers carry a constant-size state during generation, so their memory does not grow with sequence length the way KV cache does, which keeps long contexts affordable. A recursively shared Multi-Token Prediction (MTP) head supports native speculative decoding.
Use cases
Nemotron 3.5 Super VL supports agent workflows that combine reasoning with visual understanding:
- Coding: test and validate code, analyze errors, recommend fixes.
- Web research: cross-reference and verify sources returned by search tools.
- Document intelligence: read tables, charts, and page layouts to produce summaries.
- Cyber security: identify vulnerabilities and triage threats.
- Video understanding: search, content, review, and incident analysis.
Nemotron 3.5 Super VL integrates with major open agent frameworks and orchestration systems through NVIDIA reference implementations, cookbooks, and the NVIDIA Agent Toolkit. For the full list of supported frameworks, see the NVIDIA developer blog.
Minimum GPU configurations
Deployment on AWS
This section gives an overview of architecture, validation, and performance evaluation approaches. A detailed walkthrough of deploying Nemotron 3.5 Super VL and Nemotron 3 Ultra on AWS is available in the aws-do-eks project, including prerequisites, cluster creation, model weights download, deployment, testing, benchmarking, and cleanup. The same pattern can also be applied to other models in the NVIDIA Nemotron family.
Deployment architecture
The reference implementation runs on standard Amazon EKS, EKS Auto Mode, and SageMaker HyperPod EKS. With EKS Auto Mode, AWS manages scaling, GPU, and EFA support in the cluster on your behalf. The Kubernetes manifests for model deployment are the same for each of the orchestrator options. Each cluster has a control plane hosting the Kubernetes API and a data plane of CPU and GPU nodes. CPU nodes run system and test pods, while the GPU nodes run the model. Amazon Elastic Container Registry (Amazon ECR) stores the container images, a shared Amazon FSx for Lustre volume holds the model weights, and Elastic Fabric Adapter (EFA) connects the GPU nodes. The following figure shows a standard Amazon EKS cluster.
NVIDIA Dynamo serves the model aggregated on a single node or disaggregated with prefill and decode on different nodes. For vision-language models, the image/video encoding stage can in addition run disaggregated on its own node.
Validation and performance
This architecture has been validated on both p6-b200.48xlarge (8 x NVIDIA B200 GPUs) and p5en.48xlarge (8 x NVIDIA H200 GPUs) instances. You can validate the cluster and its network with an NVIDIA Collective Communications Library (NCCL) all-reduce micro-benchmark.
Seeing utilization on all GPUs and traffic on all EFA adapters while the NCCL test is running, verifies that the cluster is working as expected.
You can measure the performance of a deployed model on the cluster with the NVIDIA aiperf tool.
The aiperf output in the upper left pane shows key performance indicators (KPIs) like time to first token (TTFT), inter-token latency (ITL), and throughput in tokens per second (TPS). By taking these measurements at various levels of concurrency, we can study how latency and throughput depend on the number of concurrent requests. This information is essential to thoughtfully plan capacity and model deployment topology. The aws-do-eks container provides automation for visualizing GPU and EFA utilization as well as running aiperf and concurrency sweeps with single commands.
The preceding concurrency sweep diagram reveals that the inference of Nemotron 3.5 Super VL has lower TTFT and about the same ITL between aggregated and disaggregated deployments on two nodes.
A chart like this is useful when making deployment decisions. Draw your latency budget as horizontal lines, for example TTFT under 600 ms (about 1 second) and ITL under 100 ms per token. The rightmost point under both lines is the throughput one replica of this topology can serve within the service level objective (SLO), here c32, or about 320 tokens per second. To plan capacity, divide your target tokens per second by this per-replica throughput and find the number of prefill-decode pairs to deploy. Re-running the same sweep after a change (precision, TP/PP shape, aggregated versus disaggregated, a new Dynamo or vLLM release) makes the trade-off visible immediately. A curve that moves right and down is a win. Because TTFT and ITL are plotted separately, it also tells you which side to scale: rising ITL says add decode capacity, and rising TTFT says add prefill capacity.
Conclusion
This post covers the architecture and performance evaluation patterns for serving NVIDIA Nemotron models on Amazon EKS, EKS Auto Mode, and SageMaker HyperPod EKS with NVIDIA Dynamo and vLLM. To get started, deploy Nemotron 3.5 Super VL in your own account, using the reference implementation. More inference blueprints are also available in the AI on EKS project. To get further hands-on experience with inference on EKS, register for our upcoming workshops.
Acknowledgements
The authors would like to convey special thanks to everyone involved in developing, validating, reviewing, and publishing this content, including but not limited to: Eliuth Triana, Jiahong Liu, Kshitiz Gupta, Abhijeet Patil, Mike Stefaniak, Chris Splinter, Sheetal Joshi, Aditya Ramakrishnan, Natarajan Kumar, Anish Maddipoti.
Auto-fetched and translated by AI Gem Box. All rights belong to the original author.