NVIDIA 尼莫特龙 3.5 Trancis and Granary v2：智能多语种 Transcription You Can Temple and adaption 互联网档案馆的存檔，存档日期2013-12-2.
一个多语种转录，扬声器归属，翻译，和音频QQA的开放模式，具有顶级领导板结果，外加开放的Granary v2数据管道和微调配方来适应。
笔录可以捕捉出每个口语单词，仍然会留下用户或应用程序去想谁称什么，如何翻译，或者如何格式化域特有词汇。 多语种的呼叫，快节奏的会议，以及医疗口述都需要不同的输出：一个翻译，扬声器标签，或者正确拼写的医疗名词。
NVIDIA 尼莫特龙 3.5 Translax是一种开放的，可快速的模型，可以将所录的语音变为有用的文本，现在可以在Hugging Face和ModelScope上找到。 它将带来语录、扬声器归属(Diarization)、翻译和音频问题解答为一个检查点：它将40种语言译出，将其中31种语言译出或译出英语，并将扬声器归为13种。 它处理长达一小时的录音，并给长达八名发言者贴上标签。 开发者可以用自然语言指导输出，选择其音频处理的地方，并适应域名词汇和口音的识别。
Transis在Hugging Face开放的ASR引导板上为英语的开放量级模型中排名第一位，并坐落在人工分析的精度-活性参数边框上。 在David AI的21种语言对等化领导板上，它运行在从DGX Spark到Blackwell数据中心系统的单一NVIDIA GPU上。
Granary v2的多语种数据(可在Hugging Face和ModelScope上获取)和可再使用的可调用管线，连同NeMo Speech 微调工具，将即时控制扩展至培训：开发者可以定义他们需要的书面形式并调整模型以适应他们的应用程序遇到的语音。 在一项医学案例研究中，微调将被搁置的扬声器的词出错率相对于最佳快取器削减了45.8%.
用自然语言翻译和分析指南
同一检查站可以根据以下指示，将一个会议录音变成不同的产出：
- “记录和标出每个发言者的录音”。
- “将这一音频翻译成英文。”
- “这次会议作出了什么决定？”
在下面的视频中体验一下这个模式，它处理1940年的"螺旋球喜剧"(His Girl Friday)中一个经典的场景——这部电影以其快速起火而出名，对话高度重叠。
尼莫特龙 3.5 转录一个来自他女友周五(1940年)的场景并给发言者贴上标签，包括重叠的对话。
对于不熟悉的术语，在训练前要尝试上下文。 例如，在保健方面：
用原文转述这个医学讨论。 可能术语：HbA1c，元素。 仅在音频支持时使用这些名词。 保留标准医学拼写并用常规书面形式写出数字。
模型卡推荐了预设，包括音频问题的推理，加上完整的即时指南和逐个任务矩阵：语言支持因任务而异，音频问题回答则接受英语培训。 也支持文字层面的时间戳。 与任何语音模型一样，抽查姓名，扬声器更改，并回答录音。
从现场课到持久的课。 Agora，他的实时语音和视频平台在app中授权直播对话，在课堂演示中使用了单词级别的时间戳：Nemotron 3.5 转录会把一个被录制的班级变成一个与回放同时显示的笔录，这样学生就可以看到什么是称，什么时候。 Agora的博客贯穿了他们是如何构建的。
排行榜从抄写到扬声器归属获胜
在开放量级模型中，Transis领先称；在谁的称上，它是超过两个扬声器-分配基准测试直接。
Hugging Face开放ASR导板。 领跑板在公共测试集中以字误率(WER)衡量抄写精度，分为三个道：
- 英语：以5.33%的平均WER(图1)位居开放重量型号首位。
多语种：4.62%是7种语言的WER，在模式卡的开放ASR比较中，在评价的开放量级模型中最低。
- 长式：10.98%是WER，也是这一比较中最低的。
图1. 内莫特龙 3.5 Transis在Hugging Face Open ASR Leaderboard上以5.33%的平均WER排名开放量级模型中排名第一，API模型被排除在外。 更低的WER表示抄写错误较少。
人造分析语音转文本导板。 人工分析独立地被AA-WER将语音转文本模型排入了它的词错误率指数。 尼莫特龙 3.5 用2.8%的AA-WER在开量模型中转换出第一位的关联，其中大约4B活性参数与其关联的模型相对有24B活性参数。 在AA-AgentTalk上，占AA-WER指数50%的被搁置的语音-agent语音设置，Transis评分为1.8%WER，相较于次等最佳开量模型2.3%. 转录其语音编码器为混合Mamba - Transformer 混合专家语言模型，因此其30B参数中只有约4B活性每个符。 这说明人工分析的开放式Pareto 精确度和活性参数的前沿。 在AA的全模图上，它排在几个被广泛使用的专有言论API前。
图2. 有效参数六分之一的AA精度：内莫特龙3.5 Trancis实现2.80%的AA-WER，拥有约4B活性参数。
不同语文的议长归属和重叠。 Sortformer扬声器提示到达带有相声特性的语言模型，因此Transis可以在生成期间给每个转弯打上标记，而不是仅仅依靠后一步将扬声器附加到单词上。
大卫·AI-AAI-ASR-I18N主板。 在David AI的DAI-ASR-I18N 综合收录式和口语式配音轨迹上，Trantis在评价的开放量级模型中排名第一，总体排名第二。 在基准测试的分数轨道上，在超过21种语言时会说话的分数上，它拥有被测试的14个系统的最低分数误差率，略高于NVIDIA专用的尼莫特龙3分数模型。
图3. NVIDIA模型在David AI的多语言对称引导板上占据了前两个斑点：Nemotron 3.5 Transcritics在21种语言中的总对称误差率最低(DER)为18.26%，其次是Nemotron 3 Diarization为18.38%. DER 测量扬声器-时差；下调更好。
(原始内容存档于2017-10-10). VoiceArena Diarization-Bench. 在VoiceArena的英语全会基准测试上，Transis刊登了14个系统中最低通量最小的WER(cpWER)，并在当面和在线设置中排名第一。 这也降低了最少的词：删除为参考词的8.32%，而中位数约为31%.
图4 NVIDIA模型在VoiceArena的14个系统比较中占据了前两个位置：Nemotron 3.5 Translax leads at 32.4% cpWER，再后是NVIDIA多语言者Parakeet和Nemotron 3 Diarization at 34.2%. Diarization-Bench v1涵盖英语完整会话，重叠演讲和2-8个演讲人。 cpWER 在匹配扬声器标签后计算出单词错误；下等更好。
更多评价。 Ocular AI的Conversion-STT评价将Transis列为顶级开放模式。 Josh Talks的"印度之声"领导板报道了印度各语种的译出结果。
从边缘运行到数据中心。 Trancis运行于单NVIDIA GPU上，从Jetson AGX Thor和DGX Spark到布莱克威尔数据中心GPU. 在一台有NVFP4精度的NVIDIA RTX PRO 6000 Blackwell Server Edition GPU上，它在三分钟内处理出100小时的音频，用高通量的30秒剪辑来测量。
今天用VLLM来检查检查站或者用NeMo-Speech.cpp在当地运行收音机是用来录制音频的，而不是实时流线的。
专门的录音仍然可以要求适应域名词汇和抄录常规。 Granary v2 和 NeMo Speech 微调地址，从Trancis 背后的数据开始。
Granary v2: Translatic 背后的开放数据校正管道
快速的语音模式需要培训数据，并有明确的格式目标。 Granary v2是Hugging Face和ModelScope上尼莫特龙语音集的一部分。 由NVIDIA NeMo Curator建造，其可重复使用的管道准备了Nemotron 3.5 Trancis培训中使用的多语种演讲，制作了对齐的语录变体，任务标签和可检查的质量元数据。 该公开名单涵盖48种语言的463,062小时和16个源氏体，高于Granary v1的25种语言。 这些数字说明了数据集的释放情况，而不是Transis的全部训练混合物或其取样权重。
Granary v2为每次录制生成3个笔录表格(表1)，同样的逐字记录和干净的样式开发者可以从Transis中请求，并带有一个快取。 除了抄写外，管道还制作短音频字幕，密码交换还原出借词，语音定位的QQA对，上下文的ASR偏差名词，以及自动质量分数的翻译。
表1. Granary v2 同一语句的培训目标。
管道分为四个可再利用阶段：
1. 音频准备(Metadata)：部分为原始或长式音频，在全程录音中跟踪扬声器，并标记音响事件和语言。 Granary v2 的源头已经分解，因此此版本跳过了这个阶段。
- 2. 转录(ASR)：将每种语言通向主语和恢复ASR模型，当主语不能进行幻觉检查时，返回。 人文参考笔录有助于保存可信赖的拼写，同时添加口语形式和不通俗。
- 3. Task Label Crowth (Text): Derives 语录变体，然后检查字符率，幻觉模式，以及模型或人类参考协议。 只有被接受的记录才收到背景-ASR术语和翻译。
- 4. HiFieldity(HiFi)：过滤出有噪音、被剪接或含有多发音的片段，保留一套较小的高质量录音，用于ASR以外的用途。 对于VoxPopuli数据集，它将欧洲22种语言的354k小时缩短到111k小时。
图 6. Granary v2数据校正管道，采用从元数据、ASR和Text阶段的原始音频到培训实例的样本。
开发者可以在自己的录音或经授权的公用公司(public corpora)上运行有雷和NVIDIA Dynamo后端的"NeMo Currator"管线，也可以在Granary v2公用货单上更上一层楼。 出货单包括tn raw,tn raw，和tn clean三种笔录表单，上面描述的标签，实体提示，衍生出指令对，出货出货，以及每个记录的质量旗。 依创用CC授权使用
适应重要的词汇识别
从一个指定语言，脚本，和笔录格式的开发设置提示开始。 在下面的案例研究中，将英文和拉丁字母命名在快切的全基准测试 WER从30.8%到23.8%. 对于还剩的域名错误，"NeMo Speech 微调"工作流程将代表性发言与明确的书面目标对齐。
NeMo Speech Medical ASR案例研究显示，该研究从生成的医学文本和公共医学目录中构建了适应数据，然后结合了大约153小时的合成语音，29小时来自Common Voice的重口音英语，63小时来自MultiMed的医学录音。
图7 在Eka Care的印度医用口语基准测试的全套测试中，一个更好的快速削减 WER从30.8%到23.8%，微调带来到11.8%，比出箱的模型少了约62%的转录错误。 全套包括用于选择即时检查点和检查站的855个调试片段；以下被搁置的结果是更清洁的测试。
本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
NVIDIA Nemotron 3.5 Transcribe and Granary v2: Intelligent Multilingual Transcription You Can Prompt and Adapt
One open model for multilingual transcription, speaker attribution, translation, and audio Q&A, with top leaderboard results, plus the open Granary v2 data pipeline and fine-tuning recipes to adapt it.
A transcript can capture every spoken word and still leave users or applications wondering who said what, how to translate it, or how to format domain-specific jargon. Multilingual calls, fast-paced meetings, and medical dictation each need different output: a translation, speaker labels, or correctly spelled medical terms.
NVIDIA Nemotron 3.5 Transcribe is an open-weight, promptable model that turns recorded speech into useful text, now available on Hugging Face and ModelScope. It brings transcription, speaker attribution (diarization), translation, and audio question answering into one checkpoint: it transcribes 40 languages, translates 31 of them to or from English, and attributes speakers in 13. It handles recordings up to an hour long and labels up to eight speakers. Developers can guide output with natural language, choose where their audio is processed, and adapt recognition to domain vocabulary and accents.
Transcribe is first among open-weight models on the Hugging Face Open ASR Leaderboard for English and sits on Artificial Analysis’s accuracy–active-parameter frontier. For speaker attribution, it posts the lowest error rate on David AI’s 21-language diarization leaderboard and ranks first in VoiceArena’s 14-system comparison. It runs on a single NVIDIA GPU, from DGX Spark to Blackwell data-center systems.
Granary v2’s multilingual data (available on Hugging Face and ModelScope) and reusable curation pipeline, together with NeMo Speech fine-tuning tools, extend prompt-level control to training: developers can define the written form they need and adapt the model to the speech their applications encounter. In a medical case study, fine-tuning cut word error rate (WER) on held-out speakers by 45.8% relative to the best prompt.
Guide transcription and analysis with natural language
The same checkpoint can turn one meeting recording into different outputs, depending on the instruction:
- “Transcribe this recording and label each speaker.”
- “Translate this audio into English.”
- “What decisions were made in this meeting?”
Experience the model in action in the video below, where it tackles a classic scene from the 1940 screwball comedy His Girl Friday—a film famous for its rapid-fire, heavily overlapping dialogue.
Nemotron 3.5 Transcribe captions a scene from His Girl Friday (1940) and labels speakers, including overlapping dialogue.
For unfamiliar terms, try context before training. In a healthcare setting, for example:
Transcribe this medical discussion in the original language. Possible terms: HbA1c, metformin. Use these terms only when supported by the audio. Preserve standard medical spelling and write numbers in conventional written form.
The model card has recommended presets, including reasoning for audio questions, plus the full prompt guide and a task-by-language matrix: language support differs by task, and audio question answering is trained in English. Word-level timestamps are also supported. As with any speech model, spot-check names, speaker changes, and answers against the recording.
From live classes to lasting lessons. Agora, whose real-time voice and video platform powers live conversations in apps, used word-level timestamps in a classroom demo: Nemotron 3.5 Transcribe turns a recorded class into a transcript shown alongside the replay, so students can see what was said and when. Agora’s blog walks through how they built it.
Leaderboard wins from transcription to speaker attribution
Among open-weight models, Transcribe leads on what was said; on who said it, it tops two speaker-attribution benchmarks outright.
Hugging Face Open ASR Leaderboard. The leaderboard measures transcription accuracy as word error rate (WER) on public test sets, in three tracks:
- English: first among open-weight models at 5.33% average WER (Figure 1).
- Multilingual: 4.62% mean WER across seven languages, the lowest among evaluated open-weight models in the model card’s Open ASR comparison.
- Long-form: 10.98% mean WER, also the lowest in that comparison.
Figure 1. Nemotron 3.5 Transcribe ranks first among open-weight models on the Hugging Face Open ASR Leaderboard (English) at 5.33% average WER, with API models excluded. Lower WER means fewer transcription errors.
Artificial Analysis Speech-to-Text Leaderboard. Artificial Analysis independently ranks speech-to-text models by AA-WER, its word error rate index. Nemotron 3.5 Transcribe ties for first among open-weight models at 2.8% AA-WER, with about 4B active parameters versus 24B for the model it ties. On AA-AgentTalk, the held-out voice-agent speech set that makes up 50% of the AA-WER index, Transcribe scores 1.8% WER, versus 2.3% for the next-best open-weight model. Transcribe pairs its speech encoder with a hybrid Mamba–Transformer mixture-of-experts language model, so only about 4B of its 30B parameters are active per token. That puts it on Artificial Analysis’s open-weight Pareto frontier of accuracy versus active parameters. On AA’s all-model chart, it ranks ahead of several widely used proprietary speech APIs.
Figure 2. Leading AA accuracy at about one-sixth the active parameters: Nemotron 3.5 Transcribe achieves 2.80% AA-WER with approximately 4B active parameters.
Speaker attribution across languages and overlap. Sortformer speaker cues reach the language model with the acoustic features, so Transcribe can tag each turn during generation instead of relying solely on a later step to attach speakers to words.
David AI DAI-ASR-I18N Leaderboard. On David AI’s DAI-ASR-I18N combined transcription-and-speaker-attribution track, Transcribe ranks first among evaluated open-weight models and second overall. On the benchmark’s diarization track, which scores who spoke when across 21 languages, it has the lowest diarization error rate of the 14 systems tested, slightly ahead of NVIDIA’s dedicated Nemotron 3 Diarization model.
Figure 3. NVIDIA models hold the top two spots on David AI’s multilingual diarization leaderboard: Nemotron 3.5 Transcribe posts the lowest overall diarization error rate (DER), 18.26% across 21 languages, followed by Nemotron 3 Diarization at 18.38%. DER measures speaker-time errors; lower is better.
VoiceArena Diarization-Bench. On VoiceArena’s English full-session benchmark, Transcribe posts the lowest concatenated minimum-permutation WER (cpWER) of 14 systems and ranks first in both in-person and online settings. It also drops the fewest words: deletions are 8.32% of reference words, versus a median of about 31%.
Figure 4. NVIDIA models hold the top two spots in VoiceArena’s 14-system comparison: Nemotron 3.5 Transcribe leads at 32.4% cpWER, followed by NVIDIA Multitalker Parakeet with Nemotron 3 Diarization at 34.2%. Diarization-Bench v1 covers English full sessions, overlapping speech and 2–8 speakers. cpWER counts word errors after matching speaker labels; lower is better.
More evaluations. Ocular AI’s Converse-STT evaluation ranks Transcribe the top open model. Josh Talks’ Voice of India leaderboard reports Transcribe’s results across Indian languages.
Runs from edge to data center. Transcribe runs on a single NVIDIA GPU, from Jetson AGX Thor and DGX Spark to Blackwell data-center GPUs. On one NVIDIA RTX PRO 6000 Blackwell Server Edition GPU with NVFP4 precision, it processes 100 hours of audio in under three minutes, measured on 30-second clips at high concurrency.
Serve the checkpoint with vLLM today or run it locally with NeMo-Speech.cpp. Transcribe is built for recorded audio, not live streaming.
Specialized recordings can still call for adaptation to domain vocabulary and transcript conventions. Granary v2 and NeMo Speech fine-tuning address that, starting with the data behind Transcribe.
Granary v2: The open data curation pipeline behind Transcribe
A promptable speech model requires training data with unambiguous formatting targets. Granary v2 is part of the Nemotron speech collection on Hugging Face and ModelScope. Built with NVIDIA NeMo Curator, its reusable pipeline prepared multilingual speech used in Nemotron 3.5 Transcribe’s training, producing aligned transcript variants, task labels, and inspectable quality metadata. The public manifest covers 463,062 hours across 48 languages and 16 source corpora, up from 25 languages in Granary v1. These figures describe the dataset release, not Transcribe’s full training mixture or its sampling weights.
Granary v2 generates three transcript forms for each recording (Table 1), the same verbatim and clean styles developers can request from Transcribe with a prompt. Beyond transcription, the pipeline produces short audio captions, code-switching restoration of loanwords, speech-grounded Q&A pairs, contextual ASR biasing terms, and translations with automatic quality scores.
Table 1. Granary v2 training targets for the same utterance.
The pipeline is organized into four reusable stages:
- 1. Audio Preparation (Metadata): Segments raw or long-form audio, tracks speakers across the full recording, and tags acoustic events and language. Granary v2’s sources were already segmented, so this release skipped this stage.
- 2. Transcription (ASR): Routes each language to primary and recovery ASR models, falling back when the primary fails a hallucination check. Human reference transcripts help preserve trusted spellings while adding spoken forms and disfluencies.
- 3. Task Label Generation (Text): Derives transcript variants, then checks character rate, hallucination patterns, and model or human-reference agreement. Only accepted records receive contextual-ASR terms and translations.
- 4. High Fidelity (HiFi): Filters out segments that are noisy, clipped, or contain multiple speakers, keeping a smaller set of high-quality recordings for uses beyond ASR. For the VoxPopuli dataset, it reduced 354k hours to 111k hours across 22 European languages.
Figure 6. Granary v2 data curation pipeline, following one sample from raw audio through the Metadata, ASR, and Text stages to a training example.
Developers can run the NeMo Curator pipeline with Ray and NVIDIA Dynamo backends on their own recordings or authorized public corpora, or build on the Granary v2 public manifests. The manifests include the three transcript forms as tn_raw, itn_raw, and itn_clean, the labels described above, entity hints, derived instruction pairs, provenance, and a per-record quality flag. Download source audio separately under each corpus’s original license, and follow the release’s published terms.
Adapt recognition to the vocabulary that matters
Start with a development-set prompt that specifies language, script, and transcript format. In the case study below, naming English and the Latin alphabet in the prompt cut full-benchmark WER from 30.8% to 23.8%. For domain errors that remain, the NeMo Speech fine-tuning workflow pairs representative speech with explicit written targets.
The NeMo Speech medical ASR case study shows this progression on Eka Care’s open benchmark of recordings from Indian doctors and medical students, which has 3,619 clips and no designated training split. The study builds adaptation data from generated medical text and public medicine catalogs, then combines approximately 153 hours of synthetic speech, 29 hours of accented English from Common Voice, and 63 hours of medical recordings from MultiMed.
Figure 7. On the full test set of Eka Care’s Indian medical speech benchmark, a better prompt cuts WER from 30.8% to 23.8%, and fine-tuning brings it to 11.8%, about 62% fewer transcription errors than the model out of the box. The full set includes the 855 tuning clips used to choose the prompt and checkpoints; the held-out results below are the cleaner test
Auto-fetched and translated by AI Gem Box. All rights belong to the original author.