真正的挑战，是让模型保留有效信息、识别可靠证据，并适应新的任务。
作者丨张 璐
编辑丨马晓宁
在具身智能、自动驾驶和智慧医疗等场景中，模型需要处理的数据远不止图像、文本和语音。雷达、触觉、心电、医学影像及各类传感器数据，都可能成为预测和决策的依据。然而，不同模态之间既存在关联，也包含彼此无法替代的信息；现实数据还常常面临模态缺失、噪声干扰等问题。如何在融合信息的同时保留模态差异？当不同来源的数据质量不一致时，模型又该如何判断哪些信息更可靠？如果每遇到一种新任务，都需要重新设计和训练专用模型，多模态技术的应用成本又该如何降低？10月21日，在 CNCC 2026 首日的Tutorial环节，天津大学人工智能学院英才教授、国家级青年人才张长青作了题为《广义多模态融合技术》的报告，介绍了团队围绕多模态表征、动态可信融合和通用预测基座开展的研究。从保留模态特有信息，到利用证据评估预测的不确定性，再到尝试构建能够适应不同模态组合与预测任务的基础模型，这些工作指向同一个问题：如何让多模态融合更好地适应真实世界的复杂性。多模态融合的价值，在于不同来源的信息能够相互补充。单一模态只能呈现对象或事件的部分特征，而不同模态的结合，有可能帮助模型形成更全面的判断。但模态数量增加，并不意味着预测准确率必然提高。新增信息可能包含噪声，也可能对当前任务产生误导。因此，多模态融合不仅要回答如何整合信息，还要回答如何保留有效信息、抑制无关干扰。张长青首先介绍了多模态表征中的两种思路：联合表征（Joint Representation）与协同表征（Coordinated Representation）。联合表征尝试将不同模态的信息整合到共同的表征中；协同表征则强调不同模态之间的关联。以典型相关分析（CCA）及其深度学习变体为例，相关方法会将不同模态映射到具有较强相关性的空间，以利用它们之间的共性信息。问题在于，如果过度强调共享信息，模态特有的信息可能难以得到充分保留。例如，在医学诊断中，病原学检测和 CT 影像能够从不同角度提供信息。前者可能帮助识别感染的病原体，后者能够显示肺部病灶的位置与范围。两类信息相互补充，却不能简单地相互替代。因此，多模态表征需要同时考虑共性与个性，而不是仅仅追求不同模态之间的一致性。张长青团队尝试改变表征学习的思路：不只从观测数据中提取共同特征，还通过重构不同模态的观测，学习能够容纳多种信息的潜在表征。在这一思路下，可以将每个样本背后的潜在表征视为需要学习的变量，再通过不同的映射机制重构各模态的观测数据。与单纯将不同模态映射到共同空间的做法相比，这种方式试图从重构目标出发，保留观测数据中包含的不同信息。张长青将这一思路与神经辐射场（NeRF）的立体结构恢复联系起来。NeRF 根据不同视角的观测学习场景表示，并据此生成相应视角的图像。两者在通过潜在表示解释多种观测这一思路上具有相似之处，但具体任务和技术机制并不相同。这一研究方向也与医疗、工业场景中的模态缺失问题有关。在医院中，不同患者接受的检查项目可能不同：有些患者做了 CT，有些患者没有；有些患者具有完整的化验数据，有些患者则缺少部分检查结果。如果算法必须先补齐所有缺失数据才能进行预测，就可能面临高维数据补全困难、补全结果不可靠等问题。针对这一问题，团队探索了基于潜在表征的学习方法，希望利用现有可观测数据进行建模，而不必对每一种缺失情况都进行人工补全。这类研究的意义在于，现实中的多模态数据并不总是完整、规整的。如何在模态缺失的条件下有效利用已有信息，仍是多模态学习需要解决的重要问题。即使不同模态都能够提供信息，它们在具体任务中的价值也可能随环境和样本发生变化。张长青在报告中用一个日常场景解释了这种动态性：如果人在户外被蚊子叮咬，天色较暗、看不清蚊子时，可能会根据皮肤发痒的位置拍打；如果光线充足，则可以先观察，再采取行动。任务没有改变，但人会根据当时能够获得的信息调整判断依据。对于机器学习模型而言，问题也类似。在自动驾驶等场景中，可见光图像和热成像可能提供互补信息，但不同模态在不同环境下的有效程度并不相同。如果模型始终过度依赖某一种模态，就可能无法充分利用其他信息。此前的一些研究尝试根据图像的光照质量等外部指标调整模态权重。但光照条件并不能完全代表图像对特定任务的判别价值。因此，仅凭预先设定的质量指标调整权重，并不一定能够解决动态融合问题。张长青介绍的另一条研究路线，是将不确定性估计引入多模态融合。在相关工作 Trusted Multi-View Classification with Dynamic Evidential Fusion 中，团队提出可信多视图分类方法 TMC。该方法利用变分狄利克雷分布刻画类别概率分布，并结合 Dempster–Shafer 证据理论，融合不同视图提供的证据，以估计预测的不确定性。与只输出一个分类结果的方式相比，这类方法还尝试回答：当前判断获得了多少证据支持？不同来源的信息是否一致？现有证据是否足以支持一个相对可靠的判断？这种思路尤其适用于信息质量不稳定的场景。模型不必将每一种输入都视为同样可靠，而是可以根据不同来源提供的证据，调整融合结果。报告还介绍了动态证据融合在晶体结构识别中的应用。利用电子显微镜获取晶体结构信息时，不同观测角度提供的信息量可能存在较大差异。如果将所有观测一概而论，再通过固定权重或简单投票整合结果，有效证据就可能受到大量低价值观测的干扰。动态融合则尝试根据每次观测提供的证据更新判断。在满足相应条件时，系统还可以根据不确定性是否达到设定阈值，决定是否继续采集数据。演讲中展示的实验结果显示，该晶体结构识别任务的分类准确率由约58%—59%提高至93.9%左右。具体结果仍应结合对应论文的实验设置理解，不能据此推断所有多模态任务都能取得相同幅度的提升。这类研究所关注的，不只是如何将不同模态的信息放在一起，更是如何根据证据质量形成判断，并在适当的条件下停止继续收集信息。前两部分讨论的是多模态融合中的具体问题：如何保留模态特有信息，以及如何在不同信息源之间进行动态、可信的决策。在此基础上，张长青进一步介绍了团队的通用多模态预测基座——Fusion Anything Model（FAM）。相关论文 Fusion Anything: A Generalized Multimodal Foundation Model 提出了一个更具普适性的研究问题：能否训练一个模型，使其不再局限于预先指定的模态组合和单一任务，而是能够利用上下文样例，适应不同的多模态预测任务？传统多模态模型通常针对特定的数据类型和任务进行设计。面对新的应用场景，研究者可能需要重新选择算法、整理数据、训练模型，并反复调整参数。这种方式能够针对具体任务进行优化，但也增加了迁移到新场景的成本。FAM 希望探索另一种路径：通过预训练学习可迁移的多模态关联模式，在面对新任务时，利用给定的上下文样例完成预测，而不必针对每个任务重新更新模型参数。这一目标面临的主要困难，是如何获得足够多样的训练数据。真实世界中的多模态数据往往难以覆盖所有可能的模态组合和关联模式。为此，团队采用结构多模态因果模型（Structural Multimodal Causal Models，SMCMs）生成合成多模态预测任务，通过改变潜在信息、共享信息、模态特有信息及其关联结构，构造具有不同相关模式的数据。
- 模态内注意力（Intra-modality Attention）：建模同一模态内部不同特征之间的关系。
- 跨模态注意力（Cross-modality Attention）：建模同一样本中不同模态之间的关系。
- 样本级注意力（Sample-level Attention）：利用上下文样例之间的信息，帮助模型根据已有样例预测新样本。
这三类注意力机制分别关注模态内部、模态之间和不同样本之间的关联，共同支持模型对多模态预测任务的处理。这里需要区分两个概念：无需针对新任务更新模型参数，并不意味着模型完全不需要任务样例。FAM 在推理阶段采用上下文学习（In-context Learning）：研究者提供一些带标签的样例，模型根据这些样例中的输入与目标值预测新的查询样本。模型不需要针对该任务进行额外的参数微调，但仍然需要上下文样例来提供任务信息。在论文报告的评估中，FAM 在18个真实数据集、12种模态和11类预测任务上进行了测试。结果显示，它在多种任务上取得了有竞争力的表现，论文报告的平均准确率较所比较的专用融合基线提高约6.9个百分点。不过，不同数据集上的表现并不完全一致，模型也并非在所有测试中都优于其他方法。除了论文中的公开评估，张长青还展示了团队在若干实际任务上的初步测试。在具身智能方向，团队将 FAM 用于视触觉抓取成功率预测。现场展示的结果中，FAM 的预测表现与一项专用方法接近。这说明通用预测基座有可能用于处理视觉和触觉等不同来源的信息，但预测抓取是否成功，与机器人实际执行抓取动作仍是不同的问题。在另一项具身智能相关测试中，团队尝试将模型用于强化学习中的评分任务，结合文本指令和视频观测，对不同动作轨迹进行评价。现场 PPT 展示的结果中，加入 FAM 后的方案优于所比较的基线方案。具体提升幅度仍需结合相应实验设置理解。在医疗方向，团队与湘雅医院皮肤科开展了多模态皮肤病诊断测试，涉及临床图像、皮肤镜图像和病史等信息。报告展示了 FAM 与 PanDerm 等方法在相应测试中的比较结果。这些结果可以作为探索通用多模态预测模型在医学任务中应用潜力的初步依据，但不能直接推导出它在所有皮肤病诊断场景中都优于专用模型。团队还展示了一项历史手稿识别测试。面对难以直接识别的手写文本，研究者利用视觉信息、前后文以及其他特征进行预测。该案例说明，通用预测框架不仅可以用于常见的图像和文本任务，也有可能整合不同类型的辅助信息。不过，这项工作仍需结合其具体数据、评价指标和实验设置进一步理解。从这些探索可以看到，Fusion Anything 的目标并不是用一个模型立即取代所有专用模型，而是尝试降低新任务的适配成本，让多模态融合能够更灵活地迁移到不同的应用场景。目前，这一方向仍有不少问题需要继续研究，包括模型的后训练、垂域适配、模态编码器的选择，以及如何将通用预测能力可靠地应用于真实的机器人系统。回顾整场报告，多模态融合研究涉及三个相互关联的层面：在表征阶段，如何兼顾不同模态的共性与个性；在决策阶段，如何利用证据评估预测的不确定性；在更通用的模型层面，如何让学到的关联模式适应新的模态组合和预测任务。这些问题尚未全部解决，但它们共同指向一个明确的研究方向：让多模态模型不仅能够整合更多信息，还能够更有效地利用信息，并在面对新的任务时减少重复设计和训练的成本。雷峰网(公众号：雷峰网)
CNCC 即将召开，欢迎参会的研究者、学生，以及关注计算机领域前沿进展的读者加入我们的 CNCC 交流社群。
在这里，你可以认识同行、交流研究，分享招生信息、科研合作、约饭、投稿、组队参会等，寻找潜在的合作伙伴。
大会期间，我们也会持续带来现场报道速递、嘉宾演讲解读、前沿方向梳理和论坛精华。
会场见，也在群里见。
? 进群传送门： 扫码进群或添加微信Qvv0909777，备注：CNCC + 单位/学校+姓名+方向。
可独家畅览：
专家演讲PPT
大会报告全文
热门论文解读
学术新星访谈
扫描上方二维码
或点击「阅读原文」关注专区。
雷峰网原创文章，未经授权禁止转载。详情见转载须知。
本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
The real challenge is for models to retain effective information, identify reliable evidence and adapt to new mandates.
Author Zhang Zhang Zhang
Editor Ma Xiaoning
In settings such as smart, autopilot and smart medicine, models need to process data that go beyond images, text and voice. Radars, tactiles, electrocardiograms, medical images and various sensor data can all form the basis for prediction and decision-making. However, there is a correlation between the different models and they contain information that cannot be replaced by one another; real-life data also often face problems such as the absence of the models and noise interference. How can a model difference be maintained while integrating information? How can the model judge what information is more reliable when the quality of data from different sources is not consistent? If, for each new mission, a specialized model for re-engineering and training is needed, how can the cost of applying multi-modular technologies be reduced? On 21 October, at the first day of CNCC 2026, Tutorial, a report entitled " Broad multi-modular integration techniques " was presented by Professor Zhang Zhang Zhang, Professor of Artificial Intelligence at the University of Tianjin, and a national young man, describing the team ' s research around multi-modular signs, dynamic credible integration and a generic prediction base. From the retention of model-specific information to the use of evidence to assess the uncertainties of predictions, to the attempt to construct a basic model that can be adapted to different model combinations and prediction tasks, these efforts point to the same issue: How to better adapt multi-modular integration to the complexity of the real world. The value of multidimensional integration lies in the complementarity of information from different sources. A single mosaic can present only part of the characteristics of the object or event, while a combination of different mosaics may help the model to form a more comprehensive judgement. However, the increase in the number of models does not mean that the accuracy of projections will necessarily increase. The additional information may contain noise or be misleading to the current task. Multi-modular integration, therefore, answers not only how to integrate information, but also how to retain valid information and discourage unconnected interference. Zhang Chang Qingqing began by introducing two lines of multi-state representation: Joint Presentation and Co-ordination. The joint representation attempts to integrate information on different modes into a common representation; the synergetic representation emphasizes the linkages between the different modes. In the case of typical relevant analyses (CAs) and their in-depth learning variants, methods can map different models into more relevant spaces to take advantage of common information among them. The problem is that, if too much emphasis is placed on sharing information, information that is unique to the model may not be fully preserved. For example, in medical diagnosis, pathogen tests and CT images can provide information from different perspectives. The former may help identify the infected pathogens, while the latter can show the location and extent of pulmonary stoves. The two types of information complement each other but cannot simply replace one another. Multi-modular expressions therefore need to take into account commonality and individuality at the same time, rather than simply seeking consistency between different modes. The Zhang Young-young team tried to change the idea of typologies: not only extracting common features from observations, but also reconfiguring observations of different mosaics, learning the potential signs of a multiplicity of information. In this line of thinking, the potential manifestations behind each sample can be considered as variables that need to be studied and then re-construct observations of the various modulations through different mapping mechanisms. This approach seeks to preserve the different information contained in the observation data from a reprogramming target, as compared to the simple mapping of different mosaics into a common space. Zhang Yongqing linked this idea to the restoration of the stereostructure of the NeRF. NeRF is expressed on the basis of observational learning scenarios from different perspectives, and from which images of the respective perspectives are generated. The two have similarities in the interpretation of multiple observations through potential expressions, but the specific tasks and technical mechanisms are not the same. The direction of the study is also missing from the medical, industrial landscape. The issue is relevant. In hospitals, different patients may have different screening programmes: some patients have CT and some patients do not; some patients have complete laboratory data and others lack some of the results. If algorithms have to complete all missing data in order to make projections, they may face high-dimensional data completion difficulties and unreliable completion results. In response, the team explored learning methods based on potential manifestations and wanted to model using available and observable data, without having to manually complete each of the deficiencies. The significance of such research is that the reality of multi-modular data is not always complete and organized. The effective use of available information in the absence of a model remains an important issue to be addressed by multi-modular learning. Even if different models can provide information, their value in a specific task may change with the environment and sample. Zhang Yangqing explains this dynamic with a daily scene: When a person is bitten by a mosquito outside the house, it is darker and can't see the mosquito, it is likely that the beating will be based on the position of the skin itching; if there is sufficient light, it can be observed and action taken. The mandate has not changed, but the basis of judgement will be adjusted on the basis of information available at the time. The problem is similar for machine learning models. Visible photo images and thermal imaging may provide complementary information in scenarios such as automatic driving, but different mosaics are not equally effective in different environments. If the model is always over-reliant on one model, other information may not be fully utilized. Some previous studies have attempted to adjust the modular weights to external indicators such as image photomass. The light conditions, however, do not fully represent the value of the image for a given mission. Thus, weighting on pre-set quality indicators alone does not necessarily solve the problem of dynamic integration. Another line of study presented by Zhang Yongqing is to introduce uncertainty estimates into multi-modular integration. In related work, the team presented a credible multi-view classification TMC. The method uses a variable Dilecre distribution of the probability distribution of the type, combined with the Dempster-Shafer evidence theory, to integrate evidence from different views to estimate the uncertainty of projections. Compared to the way in which only one classification result is exported, such methods also attempt to answer: How much evidence is there to support the current judgement? Is the information from different sources consistent? Is the available evidence sufficient to support a relatively reliable judgement? This approach applies in particular to situations where the quality of information is unstable. The model does not need to regard each input as equally reliable, but may adjust the integration results based on evidence from different sources. The report also describes applications of dynamic evidence integration in crystal structure recognition. The amount of information provided from different observations may vary considerably when information on crystal structure is obtained using an electronic microscope. If all observations are summarized and the results are consolidated through fixed weights or simple votes, effective evidence may be disturbed by a large number of low-value observations. Dynamic convergence attempts to update judgement based on the evidence provided by each observation. When the corresponding conditions are met, the system may also decide whether to continue the collection of data, depending on whether the uncertainty meets the set threshold. The results of the experiment shown in the presentation showed that the classification accuracy of the crystal structure recognition tasks increased from about 58 to 59 per cent to about 93.9 per cent. The results should still be understood in conjunction with the experimental set-up of the corresponding paper, on the basis of which it cannot be assumed that all multi-modular missions will achieve the same degree of improvement. These kinds of research are concerned not only with how to put information on different models together, but also with how. Based on the quality of the evidence, the continued collection of information is discontinued under appropriate conditions. The first two sections discuss specific issues in the multi-mode integration: how to preserve the model-specific information and how to make dynamic and credible decisions among different sources of information. On this basis, Zhang Yongqing further described the team's generic multi-modular projection base, Fusion Anything Model (FAM). The paper Fusion Anything: A Generalized Multimedia Foundation Model raises the more general question of whether a model can be trained to move away from pre-defined modulations and a single task, but can be adapted to different multi-modular prediction tasks using a context sample. Traditional multi-model models are usually designed for specific data types and tasks. In the face of new applications, researchers may need to re-select algorithms, collate data, train models and adjust parameters over and over again. This approach can be mission-specific and optimized, but also increases the cost of moving to new scenarios. FAM wishes to explore another path: to study in advance the transportable model of multi-modular linkages, using a given context sample to complete the projection in the face of new tasks, without having to re-engineer model parameters for each mission. The main challenge to this goal is to obtain sufficient diversity of training data. Multi-mode data in the real world often make it difficult to cover all possible modular combinations and associated patterns. To this end, the team uses a structured multimodular cause and effect model (SMCMs) to generate synthetic multimodular predictions that construct data with different related patterns by changing potential information, sharing information, modular specific information and its associated structure.
- Intra-modality Attention: Modelling the relationship between different characteristics within the same model.
- Cross-modality Attention: Modelling the relationship between the different forms in the same sample.
- sample Level of Attention (Sample-level Attention): Using information between context samples to help models predict new sample based on existing examples.
The three types of attention mechanisms focus on the linkages between the modulus, between the modulars and between the different samples, and together support the model ' s treatment of multi-modular prediction tasks. There is a need to distinguish between two concepts: the fact that model parameters need not be updated for new missions does not mean that the model does not require a mission sample at all. FAM uses context Learning (In-contextLearning) at the reasoning stage: Researchers provide some examples of labeled cases where models predict new queries from sample based on input and target values in these samples. The model does not need to fine-tune additional parameters for the mission, but a sample of context is still required to provide mission information. In the assessment reported in the paper, FAM tested 18 real data sets, 12 modulations and 11 types of prediction missions. The results showed that it had performed competitively on a wide range of missions, with the average accuracy rate of paper reports increasing by about 6.9 percentage points compared to the specific integration baseline compared with that used in the comparison. However, the performance of different data sets is not entirely consistent and models are not always better than other methods in all tests. In addition to the public assessment in the dissertation, Zhang Yongqing presented the team ' s initial tests on a number of practical tasks. In a smart direction, the team uses FAM to capture success rate predictions. In the results presented on the ground, FAM's predictions were close to a specific method. This suggests that the generic projection base may be used to handle information from different sources, such as visual and tactile, but the success of the projection remains a different issue from the actual implementation of the robotic capture action. In another smart-related test, the team tried to use the model to reinforce the scoring task in learning, and to evaluate the trajectory of different actions in conjunction with text instructions and video observations. In the results of the live PPT presentation, the after-FAM programme is better than the compared baseline programme. The specific increase still needs to be understood in conjunction with the corresponding experimental settings. In the medical direction, the team conducted a multi-modular skin disease diagnostic test with the dermatology section of the Jaia Hospital, which included information on clinical images, skin lens images and medical history. The report shows how FAM and PanDerm are compared in the corresponding tests. These results could serve as an initial basis for exploring the potential of generic multi-modular prediction models for application in medical missions, but could not be directly extrapolated from the fact that they were superior to specialized models in all skin disease diagnostic scenarios. The team also showed a history manuscript recognition test. In the face of handwritten texts that are difficult to identify directly, researchers use visual information, background and other features to predict. The case demonstrates that the generic forecasting framework can be used not only for common imagery and text missions, but also for the integration of different types of supporting information. However, this work needs to be further understood in the context of its specific data, evaluation indicators and experimental settings. As can be seen from these explorations, the goal of Fusion Anything is not to immediately replace all dedicated models with a model, but to try to reduce the adaptation costs of new missions, allowing for more flexible migration of multi-modular integration to different applications. At present, there are a number of problems in this direction that need to be studied further, including post-training of models, apogee adaptation, the choice of model encoders, and how to reliably apply generic prediction capabilities to real robotic systems. Looking back at the whole report, the multi-mode integration study has three interrelated dimensions: At the presentation stage, how to balance the commonalities and individuality of the different models; at the decision-making stage, how to use evidence to assess the uncertainty of predictions; and at the more generic model level, how to adapt the relevant models learned to the new model mix and prediction tasks. These issues have not yet been fully addressed, but together they point to a clear direction of research: to enable multi-modular models not only to integrate more information but also to use letters more effectively and reduce the cost of repetitive design and training in the face of new tasks. Thunder Peak Network (WeChat Official Account:Ray Peak Network)
The CNCC is about to be convened to welcome researchers, students and readers interested in the cutting edge of the computer field to our CCCC community.
Here, you can identify potential partners by knowing your peers, exchanging research, sharing information on enrolment, scientific cooperation, dinners, contributions, team participation, etc.
During the General Assembly, we will also continue to bring on-the-spot stories, guest speeches, front-line orientations and forum excellence.
I'll see you there, too.
♪ Into the Quarreau portal: Scrutinize or add WeChat Qvv0909977, note: CCC + unit/school + name + direction.
It's an exclusive view:
Expert presentations PPT
* Reissued for technical reasons.
The reading of popular papers
Academic New Star Interview
Scan 2D above
This post is part of our special coverage Syria Protests 2011.
Original Ray Peak article, unauthorized reproduction. For further information, see the information below.
Auto-fetched and translated by AI Gem Box. All rights belong to the original author.