扩散语言模型开始向更大规模迈进
大语言模型一定要从左到右、逐个生成 Token 吗？
过去几年，以 GPT、Claude、Qwen 为代表的自回归语言模型一直占据主流。随着模型能力不断提升，逐 Token 解码带来的串行计算开销，也成为影响推理速度的重要因素之一。
扩散语言模型（Diffusion Large Language Model，dLLM）正在探索另一条技术路线。它不必严格按照从左到右的顺序生成文本，而是通过迭代去噪、并行更新多个 Token 完成生成，有望在特定场景下获得更高的解码效率。
但这种路线也面临一个问题：当模型规模不断扩大、任务复杂度持续提高时，能否在保持速度优势的同时，达到主流自回归模型的能力水平？
10 月 28 日，深圳 AI 创业公司扩散智能（DiffuSpace）发布并开源 Clover-35B-A3B-Preview，尝试回答这一问题。
DiffuSpace 由香港大学 NLP 实验室联合主管孔令鹏教授及其博士生龚珊三、叶佳成创立，团队此前曾推出 DiffuSeq、Dream 7B 等扩散语言模型相关研究成果。近期，公司连续完成两轮融资，总金额达数亿元人民币。
此次发布的 Clover-35B-A3B-Preview 采用混合专家（MoE）架构，总参数量为 350 亿，每次推理激活约 30 亿参数。相比上一代 Clover 模型，其总参数规模扩大至约 5 倍。
根据 DiffuSpace 公布的内部测试结果，新模型的整体生成速度达到 680 tokens/s。在部分 Agent 任务和通用能力评测中，其表现达到或超过了同等规模的自回归模型。
对于扩散语言模型而言，这次更新值得关注的地方，在于它试图同时解决两个问题：一是如何在更大参数规模下维持并行解码的效率，二是如何将速度优势转化为真实任务中的执行能力。
680 tokens/s，速度优势能否转化为任务效率？
DiffuSpace 此次公布的测试数据，主要涉及模型生成速度、Agent 任务能力以及端到端执行时间。
在官方展示的一项无人机空域管理 Demo 中，模型需要读取 20 条空管报文，从中提取地点、半径等信息，并在三维地图上设置相应禁飞区。Clover 完成该任务耗时 7.32 秒，对照使用的 Qwen3.5 模型耗时 62.3 秒，两者相差约 8.5 倍。
这一案例展示了扩散模型在结构化信息处理任务中的潜在优势。不过，Demo 中的执行时间也受到模型配置、硬件环境、输出长度和工具调用流程等因素影响。
在 WindTunnel 的一项包含 49 个任务的测试中，Clover-35B-A3B-Preview 的中位执行时间为 1.52 秒，Jev 与 Mercury 2.5 的组合为 3.21 秒，GPT-5.6 Luna 为 5.74 秒，Gemini 3.6 Flash 为 7.22 秒。按这组结果计算，Clover 的中位任务耗时约为 Jev 与 Mercury 2.5 组合的 47%。
不过，不同模型组合的推理部署方式、任务处理流程以及计费规则可能存在差异，仅凭中位执行时间，还不足以判断其在实际生产环境中的综合效率。
对于 Agent 应用来说，这一点尤其重要。模型生成 Token 的速度，并不等同于 Agent 完成任务的速度。一个实际任务可能涉及多轮规划、工具调用、外部服务等待和结果校验。即使模型生成速度提高数倍，如果主要耗时集中在外部工具或任务执行环节，最终收益也可能有限。
因此，相比单纯比较 tokens/s，Clover 在完整 Agent 工作流中的表现更值得关注。
Agent 编程能力达到什么水平？
扩散语言模型过去的一项主要挑战，是在提高生成效率的同时，维持复杂推理、代码生成和指令遵循等能力。DiffuSpace 此次公布的评测结果显示，Clover-35B-A3B-Preview 在多项 Agent 基准测试中取得了较高分数。其中，SWE-bench Verified 得分为 73.6，Terminal-Bench 2.1 为 51.7，τ³ Overall 为 65.3。
根据团队提供的对比数据，Clover 在上述评测中的表现超过了 Mercury 2.5，以及同等参数规模的 Qwen3.6-35B-A3B。这几项测试分别涉及真实软件问题修复、终端环境中的任务执行，以及带有工具交互的复杂任务处理。相比传统知识问答或代码补全，它们更接近 Agent 需要完成的多步骤工作。
在通用能力方面，Clover 还接受了知识、数学、代码、逻辑推理、指令遵循和长上下文理解等测试。从团队公布的数据来看，Clover 在部分指标上超过了同尺寸自回归模型，但并非所有项目都占优。例如，在部分数学和代码评测中，其他模型仍然保持领先。
这也说明，扩散语言模型的能力提升并不是全面、均匀发生的。不同任务对于生成顺序、上下文建模和推理过程的要求不同，模型架构的优势也会随之变化。
重构 GDN：扩散模型如何兼顾长上下文与并行解码？
相比参数规模和测试分数，Clover 此次更值得关注的技术变化，是对底层网络架构的调整。Clover 并非从零训练，而是基于 Qwen3.6-35B-A3B 构建，在保留原有混合 GDN/Attention MoE 骨干架构的基础上，引入块级扩散生成机制，并对门控 Delta 网络（Gated Delta Network，GDN）进行双向化改造，以适应并行去噪过程。
传统自回归模型通常依赖因果注意力机制，当前 Token 只能访问已经生成的历史信息。而扩散语言模型可以在一个文本块内部同时处理多个 Token，并通过迭代更新逐步形成最终输出。
这种生成方式带来了不同的架构需求。一方面，模型需要保留足够的历史上下文，以支持长文本理解和多轮任务；另一方面，又需要允许同一生成块内的 Token 充分交换信息，避免并行生成过程中出现语义不一致。Clover 的设计尝试将这两类需求结合起来。
根据团队介绍，模型通过递归状态对历史上下文进行压缩，减少重复处理历史信息的计算开销；同时，在当前生成块内引入双向注意力，使不同位置的 Token 可以进行更细粒度的信息交互。
在此基础上，模型还结合了可学习的块大小校准机制，以及双向局部特征提取，以适应不同的生成需求。这一设计的核心，是让模型在处理历史信息和当前生成块时采用不同机制：历史信息通过压缩状态保留，当前块内部则允许双向交互与并行更新。
对于扩散语言模型来说，这种设计具有一定的针对性。如果完全依赖标准注意力机制，随着上下文长度和模型规模增加，计算与存储成本可能持续上升；如果过度压缩历史信息，又可能损失复杂任务所需的上下文细节。Clover 试图通过混合结构在两者之间取得平衡。
目前 Clover 展示了一种适用于大规模扩散语言模型的架构设计思路，但其效率优势能否在不同模型规模和任务场景中稳定延续还需要进一步验证。
从文本生成到 Agent，扩散模型还有多远？
Clover-35B-A3B-Preview 的发布，反映出扩散语言模型正在尝试从高速文本生成，向更复杂的任务执行场景扩展。过去，dLLM 最容易展示的优势是生成速度。通过并行更新多个 Token，它有机会减少部分文本生成任务所需的串行解码步骤。
但在 Agent 场景中，模型需要完成的工作远不止生成文本。它需要理解任务目标、规划执行步骤、调用外部工具，并根据执行结果持续调整后续行为。部分任务还要求模型输出严格符合特定格式，或者在多个相关判断之间保持逻辑一致。这些需求也意味着，生成速度只是 Agent 效率的一部分。
DiffuSpace 透露，接下来计划基于 Clover-35B-A3B-Preview 推出决策模型和多模态模型。其中，决策模型将面向快速、结构化的判断任务，计划支持在一次生成过程中联合处理最多 8 个关联问题。按照团队的设想，这种方式可以让模型在多个判断之间共享上下文，减少逐项独立决策可能带来的逻辑冲突。不过，这一能力目前仍属于后续产品规划，实际效果尚未可知。
从更广泛的行业发展来看，扩散语言模型已经不再只是实验室中的研究方向。Google 曾推出 Gemini Diffusion，Inception 持续迭代 Mercury 系列模型，LLaDA 等开源研究也在探索扩散式文本生成的能力边界。不同团队的技术路线并不完全一致，但都在尝试突破传统自回归解码的效率限制。
与此同时，自回归模型也在通过推测解码、模型结构优化、推理引擎改进等方式持续提高生成效率。因此，扩散模型的速度优势并不意味着它一定会取代自回归架构。未来两类技术路线可能在不同任务和部署条件下形成各自的优势。
对开发者而言，更有价值的问题是：在相同硬件、相近模型能力和可接受的质量约束下，哪种方案能够以更低的成本完成任务？这也是扩散语言模型从研究成果走向生产环境时必须回答的问题。
目前，Clover-35B-A3B-Preview 已正式开源，模型权重和相关代码分别发布在 Hugging Face、GitHub 和 ModelScope。模型采用 Apache-2.0 许可证，并提供 SGLang、llama.cpp 等推理后端的相关支持。
随着更多大规模扩散语言模型开源，行业将有机会在统一的硬件和任务条件下，进一步比较两种生成范式的实际效率。对于 Clover 而言，35B 参数规模和 680 tokens/s 的速度数据只是起点。能否将这种速度优势稳定转化为复杂 Agent 任务中的成本和效率收益，才是下一阶段更重要的考验。
本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
The proliferation of language models is starting to move to a larger scale.
Do big language models have to be left to right, token by one?
The self-repatriation language model, represented by GPT, Claude and Qwen, has been in the mainstream over the past few years. As model capabilities continue to grow, the serial costs of token decode are also one of the important factors influencing the speed of reasoning.
The Diffusion Large Language Model, d LLM is exploring another technical route. It does not have to produce text in a strict left-to-right order, but is produced by inverted noise, multiple token updates in parallel, and is expected to achieve higher decode efficiency in a given scenario.
But this route also faces a problem: Will it be possible to achieve the level of capacity of the mainstream self-return model, while maintaining the speed advantage, when the size of the model is growing and the complexity of the task continues to increase?
On 28 October, Shenzhen AI Enterprise tried to answer this question by spreading DiffuSpace publish and opening Clover-35B-A3B-Preview.
DiffuSpace was created by Prof. Koon-Peng and his Ph.D. Dr. Jinshan and Yip Jia Sung, co-director of the NLP Laboratories of the University of Hong Kong. The team has previously produced research related to proliferation language models such as DiffuSeq and Dream 7B. Recently, companies have completed two consecutive rounds of financing, totalling hundreds of millions of yuan.
Clover-35B-A3B-Preview of publish this time uses a mix of experts (MoE) structure totalling parameter count 35 billion and activates approximately 3 billion parameters per reason. Compared to the previous Clover model, its total parameter size has increased about five times.
According to the internal test results published by DiffuSpace, the new model was generated at a total rate of 680 tokens/s. In some Agent missions and generic capability assessments, performance reached or exceeded a self-return model of the same size.
For the proliferation of language models, this update is a matter of concern because it seeks to address two issues at the same time: how to maintain the efficiency of parallel decodes on a larger scale of parameters, and how to translate the speed advantage into implementation capacity in real task.
680 tokens/s, can speed advantages translate into mission efficiency?
DiffuSpace released test data on model production speed, Agent task capacity and end-to-end execution time.
In an officially displayed drone space management Demo, the model would need to read 20 airway messages, extract from them information on location, radius, etc., and set up a corresponding no-fly zone on a three-dimensional map. Clover takes 7.32 seconds to complete this task, which is about 8.5 times different from the Qwen3.5 model used in 62.3 seconds.
This case illustrates the potential advantages of proliferation models in structured information processing missions. However, the implementation time in Demo was also affected by modelling, the hardware environment, output length and tool call processes.
In a WindTunnel test with 49 tasks, the median execution time for Clover-35B-A3B-Preview was 1.52 seconds, Jev and Mercury 2.5 were 3.21 seconds, GPT-5.6 Luna was 5.74 seconds and Gemini 3.6 Flash was 7.22 seconds. Based on this set of results, the median task for Clover is approximately 47 per cent of the Jev-Mercury 2.5 mix.
However, there may be differences between the reasoning deployments of different models, mission processing processes and costing rules, which are not sufficient to judge their combined efficiency in the actual production environment on the basis of the medium implementation time.
This is particularly important for Agent applications. The speed of the model 's generation of Token is not equal to the speed of Agent 's mission. One practical task may involve multiple rounds of planning, tool call, external service waiting and results validation. Even if the speed of model generation increases several times, the final benefits may be limited if the main time-consuming focus is on external tools or task implementation.
Therefore, the performance of Clover in the complete Agent workflow deserves more attention than a simple comparison of tokens/s.
Agent, what level of programming capacity?
One of the major challenges in the past in the proliferation of language models has been to maintain complex reasoning, code generation and command compliance while increasing the efficiency of generation. DiffuSpace's published assessment shows that Clover-35B-A3B-Preview achieved higher scores in multiple Agent baseline tests. Of these, SWE-bench Verified score 73.6, Terminal-Bench 2.1 51.7 and Overall 65.3.
According to comparative data provided by the team, Clover performed more than Mercury 2.5 and Qwen3.6-35B-A3B of the same magnitude of parameters. The tests concerned, respectively, the restoration of real software problems, task execution in the end environment and the processing of complex tasks with interactive tools. They are closer to the multi-step work that Agent needs to do than traditional knowledge questions and answers or code completion.
In terms of generic capability, Clover has also been tested for knowledge, mathematics, code, logical reasoning, command compliance and long context understanding. From the data published by the team, Clover exceeded the same size self-regression model on some of the indicators, but not all of the projects did. For example, other models remain in the lead in some mathematical and code assessments.
This also indicates that the ability to spread language models is not comprehensive and even. The advantages of the model structure will vary depending on the requirements of the different tasks for the sequence of generation, context modelling and reasoning processes.
Reconstruct GDN: How does a proliferation model combine long context and parallel decode?
Clover's more noteworthy technological change than the size of the parameter and the test score is an adjustment to the bottom network structure. Clover is not from zero training, but is based on Qwen3.6-35B-A3B, introducing a block proliferation generation mechanism based on the retention of the original mixed GDN/Attention MoE backbone, and a two-way adaptation of the Delta Network to the parallel noise process.
Traditional self-regression models usually rely on causal attention mechanisms, and currently Token only has access to historical information that has been generated. The proliferation language model, on the other hand, can handle multiple Tokens in one text block at the same time and evolves into a final output through an iterative update.
This type of generation brings with it different structural needs. On the one hand, the model would need to retain sufficient historical context to support long text understanding and multiple assignments; on the other hand, there would need to be a need to allow Token in the same generator block to fully exchange information and avoid semantic inconsistencies in parallel processes. The Clover design attempts to combine these two types of needs.
According to the team presentation, the model compresses historical context through a state of regression to reduce the computational costs of the duplicate processing of historical information; at the same time, a two-way focus is introduced within the current generation block so that Token at different locations can interact with more finer particles.
On this basis, the model also combines a learning-based block calibration mechanism and a two-way local characterization to accommodate different generation needs. At the core of this design is to allow the model to use different mechanisms when dealing with historical information and the currently generated block: historical information is kept in a compressed state, while the current block allows for two-way interaction and parallel updates.
This design has some relevance for the proliferation of language models. The calculation and storage costs may continue to rise with the increase in the length of context and in the size of the model, and the loss of context details for complex missions if historical information is over-compressed. Clover tried to balance the two through a hybrid structure.
Clover currently demonstrates a structure design approach that applies to a large-scale proliferation of language models, but its efficiency advantages need to be further tested for sustainability in different model scales and mission scenarios.
How far is the proliferation model from text generation to Agent?
Clover-35B-A3B-Preview ' s publish reflects the proliferation language model that is trying to generate high-speed text to expand to a more complex mission performance scenario. In the past, the most easily demonstrated advantage of dlm is the speed of generation. By updating several Tokens in parallel, it has the opportunity to reduce the serial decoder steps required for some text generation tasks.
In the Agent scene, however, the work the model has to do is much more than generate text. It needs to understand mission objectives, plan implementation steps, mobilize external tools and continuously adapt follow-up behaviour to implementation results. Some tasks also require that model outputs conform strictly to a particular format or that there be logical consistency between multiple relevant judgements. These demands also mean that the rate of generation is only part of the Agent efficiency.
DiffuSpace revealed that the next plan is to introduce decision models and multi-modular models based on Clover-35B-A3B-Preview. Of these, the decision-making model will be oriented towards a fast, structured task of judgement, and it is planned to support the joint treatment of up to eight interrelated issues during a single generation. This approach, as envisaged by the team, would allow the model to share context between multiple judgements and reduce the potential for logical conflict of individual decision-making. However, this capacity is still part of the successor product planning and the actual results are not yet known.
In terms of broader industry developments, diffuse language models are no longer merely the direction of research in laboratories. Google has launched the Gemini Diffusion, Inception Ongoing Mercury Series model, and open-source studies like LaDA are exploring the power boundaries of diffuse text generation. The technical routes of the different teams are not entirely consistent, but attempts are being made to break the efficiency limits of the traditional self-regression decode.
At the same time, self-regression models are continuously increasing their efficiency of generation through decodering, model structure optimization and improved reasoning engines. Therefore, the speed advantage of the proliferation model does not mean that it necessarily replaces the self-return architecture. The next two types of technical routes may create their respective advantages under different mandates and deployment conditions.
For developers, the more valuable question is: What options can accomplish the task at a lower cost under the same hardware, comparable model capabilities and acceptable quality constraints? This is also a question that must be answered when the proliferation of language models moves from research to a productive environment.
Currently, Clover-35B-A3B-Preview is officially open, with model weights and associated codes publish in Hugging Face, GitHub and ModelScope. The model uses the Apache-2.0 license and provides related support at the back end of the reasoning, such as SGLang, llama.cpp.
With more large-scale proliferation of open sources of language models, the industry will have the opportunity to further compare the actual efficiency of the two generation models under unified hardware and mission conditions. For Clover, 35B parameter size and 680 tokens/s velocity data are only starting points. The next more important test is whether this speed advantage can be stabilized and translated into cost and efficiency gains in the complex Agent mission.
Auto-fetched and translated by AI Gem Box. All rights belong to the original author.