图像生成正确率 54%，文字问答仅 14.57%。
作者丨郑佳美
编辑丨岑 峰
智谱内部的 GLM-Omni，先在一篇论文里露了面。这篇研究由清华大学与智谱 AI 联合完成，唐杰担任通讯作者。模型名称刚刚出现在论文中，社交平台上就有人把它与 GLM-5.5 联系起来。虽然论文并没有披露版本对应关系，但给出了 GLM-Omni 的技术底座，以及它在统一视觉理解和图像生成方面的一组实验结果。参考链接：https://arxiv.org/pdf/2610.21459其中一组数据尤其扎眼：经过图像生成训练，GLM-Omni 在数字任务中，把指定号码正确画在球衣上的比例达到 72%，研究人员拿走图片，改用文字询问球衣号码，正确率却只有 2%。模型已经能在画面里复现的知识，换成一句问话，竟然几乎答不上来。这恰好戳中了统一多模态模型的一处技术瓶颈。过去，图像理解负责看，图像生成负责画。如今两种能力逐渐进入同一套架构，模型在学习生成物体、地图和文字图案时，也会接触大量未被语言描述的视觉信息。可图像里学来的东西，究竟能有多少被语言推理使用，光看生成效果很难知道。而智谱的这篇文章正是试着沿着知识进入模型、再被调用的过程，把这个问题拆开来看。要确认生成训练有没有教给模型新知识，先得绕开它已有的记忆。让模型生成现实球队的球衣，即使号码画对了，也可能只是因为它早在文字资料中读到过。雷峰网研究人员因此搭建了一个虚构世界 MagicWorld，给从未存在过的实体安排颜色、号码和空间关系，让目标知识尽量只从此次训练的图片里进入模型。例如，一件虚构球衣叫 Arvenbrock，号码设为 10。训练时，文字提示只要求生成这件球衣，数字 10 只出现在目标图像中。等模型完成训练，研究人员再直接问它 Arvenbrock 的球衣号码。如果能回答 10，就有理由认为它把图像里的实体属性关联带进了文字回答。这套数据并不只有球衣。团队设计了颜色、数量、形状、空间排列、相对大小、数字和文字 7 类任务，每类 50 条知识，共 350 条，再为每个实体制作不同背景和视角的图像，筛选后留下 8552 组训练样本。训练文本还经过检查，尽量排除答案泄漏和属性分布不均造成的干扰。不同任务也能拉开知识类型的差别：形状与颜色往往属于有限类别，空间排列要求理解对象之间的位置关系，球衣号码和书籍标题则需要记住具体的字符序列有了这批受控知识，研究人员分别测试两条路径：一条让模型重新生成目标图片，再检查图中的属性，另一条不提供图片，只用文字提问。前者衡量模型能否在生成过程中复现关联，后者衡量它能否直接把关联转成语言答案。这样就能区分模型根本没学会某项属性，还是已经会画，却无法从文字端调用。参与实验的三种模型采用不同架构。BAGEL 将理解和生成放在相对独立的专家分支中，SenseNova-U1 直接建模图像像素，GLM-Omni 则从 GLM-4.1V-Thinking 的预训练模型出发，接入基于 X-Omni 的离散图像 Tokenizer。研究还加入 10000 条与目标知识无关的文字问答，让模型熟悉回答形式，却不提前得到答案。直接测试出来的差距，比球衣案例更普遍。BAGEL 的平均视觉正确率为 31.43%，文字回答只有 8.86%，GLM-Omni 分别为 54.00% 和 14.57%，SenseNova-U1 则为 61.14% 和 20.00%。SenseNova-U1 在号码任务中的视觉正确率甚至达到 88%，纯文字回答却是 0%。图像生成的训练目标，是让模型根据名称和提示重建视觉内容。只要实体与属性的关联能帮助还原图片，就可以通过训练得到强化，但文字问答还要沿着语言输出路径找到对应属性，并把它变成字符或词语。共享参数不意味着这些路径已经互相打通。还有些知识连生成路径也没学稳，例如 BAGEL 与 SenseNova-U1 在空间排列任务中的视觉正确率均为 0%，因此后续训练需要区分学习不足和读取困难。团队先采用 Textual Alignment，给部分实体补充文字问答，看看模型能否从少量样例学会怎样读取同类视觉属性。每类 50 个实体平分为两组：全部接受图像生成训练，其中 25 个实体再获得每个 10 条的文字问答示例，剩余 25 个不提供文字答案，留作测试。模型需要把在前一组学到的方式用到后一组身上，而不能直接记住测试答案。结果颇有分化。BAGEL 的平均文字正确率从 8.86% 升至 15.43%，SenseNova-U1 从 20.00% 升至 28.00%，GLM-Omni 却从 14.57% 降至 12.57%。SenseNova-U1 的形状任务从 20% 增至 68%，相对大小达到 100%，可三种模型在测试实体的数字与书籍标题任务中，正确率仍是 0%。这种差别与属性本身有关。形状、大小有可以迁移的分类和比较规则，其他实体的问答示例能够提供帮助，号码和书名却需要对应到每个实体独有的字符，学会回答句式远远不够。更关键的是，图像生成和文字问答仍被拆成两种训练样本，模型分别学会了画图与作答，却没有在同一条响应中练习从视觉内容走向语言答案。研究于是改变了两种监督的组织方式。Interleaved Image–Text Alignment（图文交错对齐）把图像生成和文字回答放进同一条训练序列。碰到球衣号码问题，模型先生成对应球衣，再根据画面里的号码组织文字答案。图像重建目标和语言监督围绕同一个实体属性发挥作用，比起两类任务各练各的，训练过程多了直接的衔接。雷峰网(公众号：雷峰网)评估条件却没有改变：训练时可以图文交错，测试仍要求直接文字回答，不能临时生成图片。GLM-Omni 的平均文字正确率由文本对齐阶段的 12.57% 提高到 26.86%，形状任务从 12% 升至 64%，空间排列从 0% 升至 40%。BAGEL 和 SenseNova-U1 的平均值也分别达到 17.14% 与 28.57%。不过，部分颜色和数量任务出现回落，交错训练还无法兼顾每种视觉属性。提升究竟来自图像监督，还是只是增加了文字推理内容？团队在 GLM-Omni 上做了消融：保留同样的交错响应、文字监督与独立图像生成训练，只屏蔽交错样本中的图像损失。平均文字正确率随即从 26.86% 跌到 13.71%。这说明图像重建的训练信号确实对语言调用产生了影响，而不只是给模型多看了一段文字。至于模型具体通过哪些参数共享这些信息，实验还没有定位。即使如此，号码、书名等知识依旧难以稳定地被直接说出来。研究人员索性换了条路：既然模型更擅长把答案画出来，回答之前能不能先画一遍？Visual Chain-of-Thought（Visual CoT）允许推理过程中生成目标图像，再从图像中识别属性，最后回答文字问题。这次 GLM-Omni 的平均正确率从 26.86% 升至 41.14%，BAGEL 从 17.14% 升至 20.57%，SenseNova-U1 从 28.57% 升至 34.86%。更明显的是数字任务：三种模型原先直接文字回答的正确率均为 0%，使用 Visual CoT 后分别达到 4%、36% 和 20%。论文中的一个案例里，SenseNova-U1 直接回答球衣号码时给出了错误的 69，生成球衣图像再读取后，答出了正确的 31。这也带来新的取舍。生成图像能够成为知识检索的中间载体，却需要额外计算，若图像内容出了错，答案还可能被带偏。书籍标题任务在 Visual CoT 下依然没有改善，复杂字符的生成和读取仍存在困难。加上实验知识来自受控虚构世界，这条路线在真实动态场景中的效果，还有待进一步检验。GLM-Omni 随论文露面，让智谱在统一视觉理解与生成方向上的内部探索有了具体的技术细节。研究中的几轮实验则把一个长期被生成效果掩盖的问题摆了出来：模型通过图像学到的东西，并不一定能直接用于语言回答。图文交错训练改善了部分知识的跨模态连接，Visual CoT 则让模型在需要时借助生成图像重新读取信息。对于继续走向统一架构的多模态模型，接下来的难题除了把图画好、把问题答对，还包括怎样让不同任务积累的知识有效互通。至于 GLM-Omni 是否会进入某个后续产品版本，仍需更多公开信息。
可独家畅览：
专家演讲PPT
大会报告全文
热门论文解读
学术新星访谈
扫描上方二维码
或点击「阅读原文」关注专区。
雷峰网原创文章，未经授权禁止转载。详情见转载须知。
本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
The image generation is 54 per cent correct and text Questions and Answers only 14.57 per cent.
Author Jung Jia-mi
Edit Peak
GLM-Omni inside Zhipu, first appeared in a paper. The study was carried out jointly by Tsinghua University and Zhipu AI, with Tang Jie as the author of the newsletter. The name of the model has just appeared in the paper, and it has been linked to GLM-5.5 on social platforms. Although the paper did not disclose a correspondence to the version, it gave the technical base of the GLM-Omni and a set of its experiments in harmonizing visual understanding and image generation. Reference link: https://arxiv.org/pdf/2610.214559, one of the data sets is particularly obtrusive: after training in image generation, the GLM-Omni in digital missions correctly painted 72 per cent of the assigned number on the jersey, researchers took pictures and asked for the jersey number in text, with a 2 per cent correct rate. The model is already re-emerging in the picture, and in exchange for a question, it barely answers. This coincides with a technical bottleneck in a unified multi-mode model. In the past, image understanding was responsible for seeing, and image generation was responsible for painting. Today, both capabilities are gradually entering the same set of structures, and models are also exposed to a large amount of visual information that is not described in language when learning to generate objects, maps and text patterns. But it's hard to know how much of what's learned in the image can be used for linguistic reasoning. This article by Zhipu is just an attempt to take this issue apart by following the knowledge into the model and being called again. To confirm whether or not the training has taught the model new knowledge, it is necessary to bypass its existing memory. Lets the model generate real-life team jerseys, even if the number is right, it could be just because it was read in the text. As a result, the Thunder Peak researcher built a fictional world, MagicWorld, arranging colours, numbers and spatial relationships for entities that never existed, and allowing target knowledge to enter the model as far as possible from the picture of the training. For example, a false jersey called Arvenbrock is set at 10. When you train, text tips only require the jersey to be generated, the number 10 only appears in the target image. When the model is finished, researchers will ask it the number of Arvenbrook's jersey. If you can answer 10, there is reason to believe that it brought the physical properties of the image into the text answer. This set of data is not just a jersey. The team designed the colour, number, shape, spatial organization, relative size, number and text type 7 tasks, 50 knowledges per category, 350, and produced images of different backgrounds and perspectives for each entity, leaving a selection of 8552 groups for training sample. The training text has also been checked to remove, as far as possible, the interference caused by leakage of answers and uneven distribution of attributes. Different tasks can also draw up differences in the type of knowledge: shapes and colours tend to belong to a limited category, spatial sequencing requires understanding of the location relationship between objects, and the number of the ballie and book titles need to remember that specific character sequences have this controlled knowledge, and researchers test two paths: one for the model to recreate the target picture, then check the properties in the picture, and the other does not provide a picture and only asks questions with words. The former measures whether the model reproduces linkages in the process of generation, while the latter measures whether it can directly translate linkages into language answers. This makes it possible to distinguish between a model that does not learn a certain attribute at all, or that can be painted but cannot be called from the text. The three models involved in the experiment have different structures. BAGEL places understanding and generation in relatively independent expert branches, SenseNova-U1 direct modelling image pixels, and GLM-Omni starts with the GLM-4.1V-Thinking pre-training model with access based on X-O Mni's discrete image of Tokenizer. The study also added 10000 textual questions and answers that are not related to target knowledge, so that the model is familiar with the form of the response but does not get answers in advance. The gaps directly tested are more common than in the case of jerseys. The average visual correctness rate for BAGEL was 31.43 per cent, with only 8.86 per cent text answers, 54.00 per cent and 14.57 per cent for GM-Omni, respectively, and 61.14 per cent and 20.00 per cent for Sensenova-U1. SenseNova-U1 even had 88 per cent visual correctness in the numbered task, while the answer was 0 per cent. The training objective of image generation is to allow the model to rebuild visual content by name and hint. As long as the connection between the entity and the attribute helps restore the picture, it can be enhanced through training, but text questions and answers also find the corresponding properties along the language output path and turn them into characters or words. Sharing parameters does not mean that these paths are already connected. There are also knowledge that is not even stable, such as BAGEL and Sensenova-U1, which have a zero per cent visual correctness in space scheduling tasks, so follow-up training requires a distinction between under-learning and difficulty of reading. The team uses Textual Support to add text questions and answers to some entities to see if the model can learn how to read similar visual properties from a small sample. Each of the 50 entities is divided into two groups: all were trained in image generation, 25 of which received an additional 10 text question and answer examples, while the remaining 25 did not provide text answers and left them for testing. Models need to apply the methods learned in the first group to the latter group rather than simply remember the test answer. The results were very divisive. The average text correct rate for BAGEL rose from 8.86% to 15.43%, Sensenova-U1 from 20.00% to 28.00%, while the GM-Omni dropped from 14.57% to 12.57%. The SenseNova-U1 shape mission increased from 20% to 68% and reached a relative size of 100%, but the correct rate for three models in testing entity numbers and book titles remains 0%. This difference relates to the attributes themselves. Shapes, sizes with transferable classification and comparison rules, examples of questions and answers from other entities can help, numbers and book names need to be matched to characters unique to each entity, and it is far from adequate to learn how to respond. More critically, image generation and text Query are still being broken down into two types of training sample, with models that learn to draw and answer, but do not practice from visual content to language answers in the same response. The study changed the way the two types of oversight were organized. Interleaved Image-Text Instruction (typographically aligned) places image generation and text answers in the same training sequence. When it comes to the number of the jersey, Mr. Model is the corresponding jersey, followed by the textual answer based on the number in the picture. Image reconstruction objectives and language monitoring function around the same entity attributes, with more direct interfaces in the training process than between the two types of tasks. The conditions for the assessment of the Ray Peak Network (WeChat Official Account:Ray Peak Network) have not changed: the training can be staggered, and the test still requires a direct text answer and cannot temporarily generate a picture. The average text correct rate for GM-Omni increased from 12.57 per cent to 26.86 per cent for text alignment, from 12 per cent to 64 per cent for shape tasks and from 0 per cent to 40 per cent for spatial sequencing. BAGEL and Sens The average of eNova-U1 also reached 17.14 per cent and 28.57 per cent respectively. However, partial colours and volume tasks have fallen and staggered training has not yet been able to accommodate each visual attribute. Does the upgrade come from image monitoring, or does it just add text reasoning? The team digested on GM-Omni: the same staggered response, text monitoring and independent image generation training was maintained, and only the image loss in the staggered sample was shielded. The average text correct rate fell from 26.86% to 13.71%. This suggests that the training signal for image reconstruction does have an impact on language calls, rather than just an extra text for the model. As to the parameters for which the model is to share this information, the experiment has not yet been located. Even so, the knowledge of numbers, book names and so forth is still difficult to tell with stability. Researchers have asked for a different way: since models are better at drawing answers, can you do it before you answer? Visual Chain-of-Thought (Visual Cot) allows target images to be generated during the reasoning process, then identifies properties from the images and finally answers text questions. The average correct rate for GM-Omni this time rose from 26.86 per cent to 41.14 per cent, BAGEL from 17.14 per cent to 20.57 per cent and Sensenova-U1 from 28.57 per cent to 34.86 per cent. More obvious is the digital task: the three models used the original direct text answers were 0 per cent correct, with the use of Visal Cot reaching 4 per cent, 36 per cent and 20 per cent, respectively. In a case in the paper, SenseNova-U1 gave the wrong 69 in reply to the jersey number, and once the jersey image was reread, the correct answer was 31. It also brings new trade-offs. Generating images can be an intermediate vehicle for knowledge retrieval, but additional calculations are required, and the answer may be biased if the image content is wrong. The book title job is still not improving under Vision Cot, and there are still difficulties in generating and reading complex characters. Together with the fact that experimental knowledge comes from a controlled fictional world, the effect of this route in a real dynamic scenario needs to be further tested. GLM-Omni shows up with papers and gives Zhipu specific technical details to explore in-house in the direction of unified visual understanding and generation. Several rounds of experiments in the study present an issue that has long been masked by the effects of generation: what models learn through images is not necessarily directly used in language answers. The staggered training improves the cross-modular connection of some of the knowledge, while the Visual Cot allows models to reread information by generating images when needed. For the multi-module model that continues to move towards a unified architecture, the next challenge is not only how to draw the picture and answer the questions, but also how to make the knowledge accumulated by the different mandates effectively interoperable. More information is needed on whether GM-Omni will enter a subsequent product version.
It's an exclusive view:
Expert presentations PPT
* Reissued for technical reasons.
The reading of popular papers
Academic New Star Interview
Scan 2D above
This post is part of our special coverage Syria Protests 2011.
Original Ray Peak article, unauthorized reproduction. For further information, see the information below.
Auto-fetched and translated by AI Gem Box. All rights belong to the original author.