2026年10月20日 —— 8分钟读取
杰西·唐克夫利特和李俊豪
高级软件工程师
当我使用一个开放量级模型时，我看到的结果与质量基准测试的表示相去甚远。
软件开发商最近与AI模型合作
工具调用精度是服务开放量级模型的重要组成部分。 当您的工具调用不准确时，模型建立在非感性上下文上。 这导致低质量的结果，特别是多回合会议常见的代理工作量。
可惜许多供应商和广泛使用的推论引擎并未完全通过工具调用精度测试。 例如，当我们用 vLLM 版本 0.28 服务 Kimi K2.6 时，我们只得到了92%的时间的预期工具调用计划。 换句话说，每25个尝试使用的工具电话中就有2个失败。
幸运的是，VLLM最近增加了对受约束的解码的支持，这是一种保证计划遵守的技术。 这个功能后来在vLLM0.31中被发布作为服务器侧旗。 通过启用它，可以得到较新的Kimi和DeepSeek模型的100%的计划精度。
在博客文章中，然后我们谈谈受限制的解码以及如何解决计划准确性问题。
您可以让 LLM 在两个步骤中调用外部工具：(1) 告知AI模型存在外部可用的工具调用和(2) 向模型提供如何输出结构化工具调用的指示。 例如，OpenCode连接到Kimi时，OpenCode发送了Kimi一个描述可用工具的JSON阵列。 此列表中的工具之一是取取网页内容的 Webfatch:
[{
"类型":"功能"
"函数":{
"名":"网相"作"相".
"描述":"- 从指定的 URL\n 获取内容 - 以 URL 和可选格式作为输入\n - 获取 URL 内容，转换为请求的格式( 默认情况下的标记下)\n- 返回指定格式的内容\n- 当您需要检索和分析网页内容时使用此工具\ n\ nUsage 指出：\ n [内容已编辑为简洁]\ n]
"参数":{
"$schema":"https://json-schema.org/draft/2020-12/schema" (中文(简体) ).
"类型":"对象"
"财产":{
"url": {
"类型":"弦"
"描述":"取出内容的URL"
},
“格式”:{
"类型":"弦"
"enum": [
"文字" (中文(简体) ).
"减分"
"html" (中文(简体) ).
],
"描述":"返回内容在(文本，标记下，或html)中的格式。 默认的标记下".
"默认":"标记"
},
"超时":{
"类型":"数"
"描述":"可选超时数以秒(最大为120)".
}
},
"需要":[
内脏
]
}
}
}, ...]
推论引擎然后将这个JSON描述用LLM特有符号包裹后再传递到LLM，以匹配LLM如何被训练来识别工具声明。 在服务Kimi时，这看起来像：
\\ im system tool declare im middle\ [{'type:"功能""功能":{...},...}\ \ im end }
JSON对象描述工具(或功能，上方为JSON)包括工具的参数，其类型，需要哪些参数等。 这被称为工具的计划。 为了能成功执行由LLM生成的工具调用，调用必须遵循其计划中规定的规则。
然后，当一个用户要求通过他们的agent取取网页时，LLM可能会输出出以下的令牌：
我去拿那页 QQToogle calls section begin tool call begin causes.webfetch:0XXtool call argument begin "url":"https://ubicloud.com","格式":"markdown"XQQToogle call end tool calls section end " ) 互联网档案馆的存檔，存档日期2014-12-02.
这些信使由推论服务器的解析器解析出，取出LLM特有信使并将调用变成JSON对象等：
{
""作用":"助将".
"内容":"我去拿那页".
"工具调用":[
{
"id":"函数.webfetch:0"
"类型":"功能"
"函数":{
"名":"网相"作"相".
"论语":"{\":"{\""https://example.com\","格式":"mark""markdown\"".
}
}
]
}
这随后被送回到您的被子( OpenCode，以我们为例)，它运行它的 Webfatch 工具，并将其输出返回到模型中。
要成功调用工具，必须坚持该工具的策略。
简单的工具调用方法是让模型生成工具自由调用，并相信模型的训练足以生成正确的调用。 这和我们如何相信模型以产生在战术上正确的传言相似。
简单的方法在大多数时候是有效的，因为现代模型擅长遵循结构化的规律。 但是，成功“大部分时间”意味着许多工具呼叫仍然可能失败，导致上下文窗口污染的原因是错误信息，模型性能降低，以及agent不得不重试一次或多次导致成本增加。
为避免在使用Kimi时出现这样的经历，Moonshot AI发布了他们的K2V ToolCall 基准测试。 这个基准测试让提供者验证它们服务Kimi的质量与原作相同，包括它的默认行为，即只生成有效的工具呼叫。
K2VV ToolCall 的工作量包括选定2000项请求，以评价LLM何时生成工具调用，以及生成的工具调用是否在策略上正确(即遵守所提供的计划). 基准测试允许通过量化两个公分量来分析模型的质量：触发相似性和计划精度。 Trigger相似性考虑服务何时决定触发工具呼叫，以及它的决定与Moonshot AI自己的服务有多相类似。 Schema精度将坚持工具的策略从而能够成功引用的工具调用比例量化。 例如，当工具调用精度为50%时，模型产生的工具调用有一半是错的。
由于语言模型的概率性，Schema 精度通常会下降。 一个模型根据概率生成符号。 它没有运行编译器或类型检查器。 即使有明显的快取，它仍然可以产生错误的工具调用。 例如，这可以是由于模型的训练数据与目前的推断方法不匹配。 对模型进行了许多工具调用数据集的培训，每个数据集都有自己的工具、计划和公约。 因此，一个模型可能会生成它在训练期间所见出的工具或场名，即使在当前生产环境中并不存在。 例如，当可用的工具是web search时，它可能会生成搜索 web，或者当活动计划预期位置时使用city name.
LLM计算出每个符号的概率为序列中下一个。 然后它通过进行加权抽样来选择下一个符号。 受约束的解码通过防止模型采样会导致综合无效的工具调用来避免工具调用不当。 当使用受约束的解码时，与其让模型采样符自由并验证结果之后，推论引擎会一步一步地引导生成。 每一步只允许有有效的标志。
允许哪些符号由所提供的工具方案决定。 vLLM 使用 XGrammar 库中的结构标记执行此功能。 显示下方的片段摘录了XGrammar如何在Kimi K2.6中执行受约束的解码。 它定义了工具调用的有效结构，包括调用开始地，索引出现地，参数开始地，以及必须生成schema-construction JSON的地方。
# Kimi K2.6工具调用语法：
# # 工具 call begin 函数。 name}: {index}
{\fn方正粗倩简体\fs12\an8\1cHFFFF00\b0}# 工具呼叫参数开始 {\fn方正粗倩简体\fs12\an8\1cHFFFF00\b0}
# 工具呼叫结束 # #
标签格式( T)
开始=f"{TOOL CALL BEGIN PREFI}\\叫 TOOL CALL SUFFI}"
内容=序列格式( S)
元素=[
regexFormat( pattern=r"\d+") (韩语).
ContStringFormat( 值=TOOL ALL ARGUMENT BEGIN)
JSONSchemaFormat (英语：
json schema=参数
任意命令=任意命令
最大值白空 cnt=max 白空 cnt
),
]
),
结束=TOOL CALL 结束
)
你可以在这里找到完全执行。
根据这一定义，世代必须遵循一条有效的路径：
开始工具调用工具名称工具调用索引结束工具呼叫
无效的路径在生成时被屏蔽。 例如，当生成达到工具调用指数相后，引擎将采样限制在整数符上并遮盖出其他一切。 下图取自XGrammar的文章
出自：董作霖，叶克辛等"XGrammar：为大型语言模型而灵活高效的结构化生成引擎". "机器学习与系统7"的议事录 (2025). [link].
这改变了计划准确性，从最佳方法转变为运行时间保证。 模型仍然决定要生成哪个符号，但现在该符号是从一个限制的和在策略上正确的集合中选择的。
值得注意的是，这种运行时间执法新增可忽略不计的性能间接费用。 XGrammar通过预先计算每个阶段的代号口罩，并在生成时在GPU上高效地应用这些口罩，保证正确的工具调用而不交换显著性能。
Moonshot AI建议打开受限制的解码，它自己的API也这样做。 因此，我们最初在我们的fork of vLLM中加入了受约束的解码。 VLLM 在v0.25中添加了这个特性后，我们从上游切换到使用这个特性来为Kimi和DeepSeek. 不幸的是，不久后，我们发现我们的工具召唤精度分数回落到91%!
经过分析，我们确认VLLM实施受限制的解码仍然是正确的。 相反，我们发现 vLLM 更改了其默认行为对于没有明确打开受约束解码的请求。 这主要是因为vLLM的设计是为了支持广泛的模型。 支持受限制的解码是一项按模式进行的任务，问题可分为三个阵营：
- 一般模型问题：对于一些模型来说，其不受限制的输出与工具的图案或模型的调用工具格式有很大不同。 这迫使模型选择它本来不会拥有的活物，导致它偏离了航向[1]. 例如，gpt-oss-20b,Qwen 3.6，以及GLM 5.1用户在默认启用此功能后报告了问题。
- 语法翻译bugs:vLLM可能会有bugs将模型的工具计划或工具调用格式翻译为模型的具体语法。 这会造成问题，包括无效的工具调用，请求失败，默默地放弃受约束的解码，以及降低工具调用精度[1,2]等。
- 特性交互bugs：在受约束的解码功能和其他服务引擎功能之间的相互作用中存在bug. 例如，受约束的解码和投机解码相互作用中的出错可能导致投机符被语法错误地拒绝，导致请求失败[1,2].
由于这些问题，vLLM维护者报告说，在启用了受限的解码后，看到几个模型出现问题。 这使得被默认限制的解码功能失效，成为了VLLM等模型不可知性服务引擎更安全的选项。 后来，vLLM添加了-tool-限制性级别旗，这样用户就可以为他们的部署启用工具调用严格性。
Ubicloud的焦点与VLLM不同。 我们注重服务Kimi和DeepSeek模型，其质量是用户期望的第一端提供者。 为此，我们启用了受限制的解码并重新运行了K2V ToolCall 基准测试。 一旦我们核实了100%的工具呼叫计划精度在所有服务模型版本，我们部署的改变生产。 然后我们输入这篇博客文章
当使用 LLMs 时，你观察到的质量并不仅仅基于模型。 推论引擎也起重要作用。 改变推论引擎及其设置，你就会改变模型的输出质量。
在服务Kimi K2.6和DeepSeek V4 Flash时，我们观察到由于工具调用计划不准确导致的可见质量退化。 我们解决了这个问题，首先携带我们自己的vLLM叉子，然后让新的Co
本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
October 20, 2026 · 8 min read
Jesse Donkervliet and Junhao Li
Senior Software Engineers
When I use an open-weight model, I see results that are nowhere close to what the quality benchmarks indicate.
Software developer working with AI models these days
Tool call accuracy is an important part of serving open-weight models. When your tool calls aren’t accurate, the model builds up on nonsensical context. This leads to low quality results, particularly for agentic workloads where multi-turn sessions are common.
Sadly, many providers and widely-used inference engines don’t fully pass tool-call accuracy tests. For example, when we served Kimi K2.6 with vLLM version 0.28, we got the expected tool call schema only 92% of the time. In other words, 2 out of every 25 attempted tool calls failed.
Fortunately, vLLM recently added support for constrained decoding, a technique to guarantee schema-adherence. This feature has then been released as a server-side flag in vLLM 0.31. By enabling it, you can get to 100% schema accuracy with the newer Kimi and DeepSeek models.
In this blog post, we first describe tool call schema accuracy and why it’s important in serving agentic workloads. We then talk about constrained decoding and how it fixes the schema accuracy problem.
You can enable an LLM to call an external tool in two steps: (1) inform the AI model about the existence of externally available tool calls and (2) provide the model with instructions on how to output structured tool-calls. For example, when connecting OpenCode to Kimi, OpenCode sends Kimi a JSON array describing available tools. One of the tools in this list is webfetch, which fetches Web page contents:
[{
"type": "function",
"function": {
"name": "webfetch",
"description": "- Fetches content from a specified URL\n- Takes a URL and optional format as input\n- Fetches the URL content, converts to requested format (markdown by default)\n- Returns the content in the specified format\n- Use this tool when you need to retrieve and analyze web content\n\nUsage notes:\n [content redacted for brevity]\n",
"parameters": {
"$schema": "https://json-schema.org/draft/2020-12/schema",
"type": "object",
"properties": {
"url": {
"type": "string",
"description": "The URL to fetch content from"
},
"format": {
"type": "string",
"enum": [
"text",
"markdown",
"html"
],
"description": "The format to return the content in (text, markdown, or html). Defaults to markdown.",
"default": "markdown"
},
"timeout": {
"type": "number",
"description": "Optional timeout in seconds (max 120)"
}
},
"required": [
"url"
]
}
}
}, ...]
The inference engine then wraps this JSON description in LLM-specific tokens before passing it into the LLM, to match how the LLM was trained to recognize tool declarations. When serving Kimi, this looks something like:
<|im_system|>tool_declare<|im_middle|>[{"type":"function","function":{...}, ...]<|im_end|>
The JSON object describing the tool (or function, in the JSON above) includes the tool’s parameters, their types, which of the parameters are required, etc. This is known as the tool’s schema. For an agent harness (e.g., OpenCode) to successfully execute a tool call generated by the LLM, the call must follow the rules specified in its schema.
Then, when a user asks to fetch a Web page via their agent harness, the LLM may output the following tokens:
I'll fetch that page. <|tool_calls_section_begin|><|tool_call_begin|>functions.webfetch:0<|tool_call_argument_begin|>{"url": "https://ubicloud.com", "format": "markdown"}<|tool_call_end|><|tool_calls_section_end|>
These tokens are parsed by the inference server’s parser, extracting the LLM-specific tokens and turning the call into a JSON object such as:
{
"role": "assistant",
"content": "I'll fetch that page.",
"tool_calls": [
{
"id": "functions.webfetch:0",
"type": "function",
"function": {
"name": "webfetch",
"arguments": "{\"url\": \"https://example.com\", \"format\": \"markdown\"}"
}
}
]
}
This is then sent back to your harness (OpenCode, in our example), which runs its webfetch tool and sends its output back to the model.
For a tool-call to succeed, the call must adhere to the tool’s schema.
The simple approach to tool calling is to let the model generate tool calls freely and trust that the model’s training is sufficient to generate correct calls. This is similar to how we trust the model to generate syntactically correct prose.
The simple approach works most of the time because modern models are good at following structured patterns. But success “most of the time” implies that many tool calls can still fail, resulting in context window pollution due to error messages, lower model performance, and increased cost due to the agent having to retry the call one or more times.
To avoid such experiences when using Kimi, Moonshot AI published their K2VV ToolCall benchmark. This benchmark lets providers verify that they serve Kimi with the same quality as the original, including its default behavior of only generating valid tool calls.
K2VV ToolCall’s workload consists of 2000 requests selected to evaluate when the LLM generates tool calls and whether the generated tool calls are syntactically correct (i.e., adhere to the provided schema). The benchmark allows analyzing the model’s quality by quantifying two metrics: trigger similarity and schema accuracy. Trigger similarity considers when the service decides to trigger a tool call, and how similar its decisions are to Moonshot AI’s own service. Schema accuracy quantifies the fraction of tool-calls that adhere to the tool’s schema and can therefore be invoked successfully. For example, when tool-call accuracy is 50%, half of the tool calls generated by the model are malformed.
Schema accuracy usually falls short due to the probabilistic nature of language models. A model generates tokens based on likelihood. It is not running a compiler or type checker. Even with a clear prompt, it can still produce malformed tool calls. This can be caused by, for example, a mismatch between the model’s training data and the current inference harness. Models are trained on many tool-calling datasets, each with its own tools, schemas, and conventions. As a result, a model may generate a tool or field name that it has seen during the training, even if it does not exist in the current production environment. For example, it may generate search_web when the available tool is web_search, or use city_name when the active schema expects location.
An LLM computes for each token the probability that it is the next in the sequence. It then selects the next token by performing a weighted sample. Constrained decoding avoids malformed tool calls by preventing models from sampling tokens that would result in a syntactically invalid tool call. When using constrained decoding, instead of letting the model sampling tokens freely and validating the result afterward, the inference engine guides generation step by step. At each step, only valid tokens are allowed.
Which tokens are allowed is determined by the provided tool schema. vLLM implements this using structural tags from the XGrammar library. The snippet below shows an excerpt of how XGrammar implements constrained decoding in Kimi K2.6. It defines the valid structure of a tool call, including where the call begins, where the index appears, where the arguments begin, and where schema-constrained JSON must be generated.
# Kimi K2.6 tool call syntax:
# <|tool_call_begin|>{function.name}:{index}
# <|tool_call_argument_begin|>{arguments}
# <|tool_call_end|>
TagFormat(
begin=f"{TOOL_CALL_BEGIN_PREFIX}{name}{TOOL_CALL_SUFFIX}",
content=SequenceFormat(
elements=[
RegexFormat(pattern=r"\d+"),
ConstStringFormat(value=TOOL_CALL_ARGUMENT_BEGIN),
JSONSchemaFormat(
json_schema=parameters,
any_order=any_order,
max_whitespace_cnt=max_whitespace_cnt,
),
]
),
end=TOOL_CALL_END,
)
You can find the full implementation here.
With this definition, generation must follow a valid path:
Begin tool call → Tool name → Tool call index → Begin arguments → Schema-constrained JSON → End tool call
Invalid paths are blocked during generation. For example, when generation reaches the tool-call index phase, the engine restricts sampling to integer tokens and masks out everything else. The figure below, reproduced from XGrammar’s article, provides a good visualization of how invalid tokens are masked.
Source: Dong, Yixin, et al. "XGrammar: Flexible and efficient structured generation engine for large language models." Proceedings of Machine Learning and Systems 7 (2025). [link]
This changes schema accuracy from a best-effort approach to a runtime guarantee. The model still decides which token to generate, but now the token is selected from a restricted and syntactically-correct set.
Notably, this runtime enforcement adds negligible performance overhead. By precomputing token masks for each phase, and applying them efficiently on the GPU during generation, XGrammar guarantees correct tool calls without trading off significant performance.
Moonshot AI recommends constrained decoding to be turned on and its own API behaves as such. So we initially added constrained decoding in our fork of vLLM. After vLLM added this feature in v0.25, we switched to using this feature from upstream for Kimi and DeepSeek. Unfortunately not long after, we found out that our tool-call accuracy score dropped back to 91%!
After analyzing the problem, we confirmed that vLLM’s implementation of constrained decoding was still correct. Instead, we found that vLLM changed its default behavior for requests that do not explicitly turn constrained decoding on. This is primarily because vLLM is designed to support a wide range of models. Supporting constrained decoding is a per-model task and problems fall into three camps:
- General model issues: For some models, their unconstrained output diverges significantly from the tool's schema or model's tool-calling format. This forces the model to select tokens it otherwise wouldn't have, leading it off course [1]. For example, gpt-oss-20b, Qwen 3.6, and GLM 5.1 users reported issues after having this feature enabled by default.
- Grammar translation bugs: vLLM might have bugs in translating a model's tool schema or tool-calling format into the model's specific grammar. This can result in problems including invalid tool calls, request failures, silently dropping the constrained decoding, and lower tool-calling accuracy [1, 2].
- Feature interaction bugs: There are bugs in the interaction between constrained decoding and other serving-engine features. For example, a bug in the interplay between constrained decoding and speculative decoding can result in speculative tokens being wrongly rejected by the grammar, leading to failing requests [1, 2].
Due to these issues, vLLM maintainers reported seeing problems with several models after constrained decoding was enabled. This made disabling constrained decoding by default the safer option for a model-agnostic serving engine such as vLLM. Later, vLLM added the --tool-strict-level flag so that users can enable tool-call strictness for their deployments.
Ubicloud’s focus is different from that of vLLM however. We focus on serving Kimi and DeepSeek models with the quality that users expect from first-party providers. For this reason, we enabled constrained decoding and re-ran the K2VV ToolCall benchmark. Once we verified 100% tool-call schema accuracy across all served model versions, we deployed the change to production. We then typed up this blog post to share our learnings.
When using LLMs, the quality you observe isn’t based on the model alone. The inference engine also plays an important role. Change the inference engine and its settings, and you change the model’s output quality.
When serving Kimi K2.6 and DeepSeek V4 Flash, we observed visible quality degradations due to tool-call schema inaccuracies. We solved this problem by first carrying our own vLLM fork and then enabling the new co
Auto-fetched and translated by AI Gem Box. All rights belong to the original author.