Google希望视频AI了解人们在工作时需要什么。
一个由Google领导的研究团队拥有推出 StreamVibe，一个视频agent，旨在自行决定何时说话。 测试表明，对于今天的AI模型来说，这尤其困难。
来自Google，斯坦福，牛津，和多所美国大学的研究人员正在研究一个助手，看着你做饭或组装家具，并且只在需要的时候说话。 与StreamVibe合作，他们制作了一款agent的视频，旨在理解"如果遇到任何困难，请帮助我"等甚至模糊的请求。
据报纸报道，现有的视频助理是为"水从沸腾起就告诉我"等清晰的指令而建造的，因此他们很少发现不明的意图。 他们也很难追踪已经做的工作，这导致他们重复建议或回复太晚，当用户在中途更改请求时，助手们往往会坚持原指令。
一个更关心的助手的三个组成部分
为了衡量这个问题，团队从183个多小时的日常生活和软件录音中汇编出大约4100个基准测试的例子。 Gemini 2.5 (中文(简体) ). Flash建议何时提供帮助和说什么，而四名人类评估人员必须一致赞同每个例子。 测试包括三个阶段，包括明确的指示，模糊的目标，并要求随着时间推移而改变。 一个例子是，一个用户拆分一个玩具挖掘器，要求更新进度，然后在3分钟后切换到要求质量检查。
从那里，研究人员训练了StreamVibe-VL，这个模型显然是基于Alibaba的Qwen3.5-9B，它决定了每秒是保持沉默还是说话。 第二个训练阶段采用强化学习来使模型更被保留，以免不断被打断。 一个单独的内存系统跟踪那些等待处理的请求和哪些步骤是完整的，尽管只有模型才能决定是否说话和说什么。
所有测试过的模型在连续操作中都失去了对对话的跟踪
研究人员认为，StreamVibe-VL比其他模型提高了10.7%的响应时间和质量，而内存系统则旨在再促进12.3%的主动互动，并减少不必要的中断等。
StreamVibe-VL一般会以宽比来比起其基础模型和其他开源模型，尽管Google的专有Gemini 3.8 Flash在多起测试中得分较高。 报纸称，专业培训不会损害一般视频理解。
模型通常只在具体时刻才需要决定一个提示是否合适时才发出正确的呼声。 但当他们处理一个完整的视频，并且必须依靠自己先前的反应，在所有测试过的系统中，性能会因错误而急剧下降，而根据系统本身的历史，在超过5分钟的视频中，模型很少选择正确的时机，发现在很长的时间内追踪一个被改变的指令特别困难。
在将一张图片转换为黑白的GIMP教程中，在纸上和项目页面上记录的相片，对比模型JoyAI 称中，完成的图像已经保存了，尽管用户尚未打开，而StreamVibe-VL则在正确的时刻给出了下一步，如"Colors"和"Desaturate". 在用果酱做卷面包的分步指南中，其他模型只是描述了屏幕上发生的情况，比如面包被平地了，而StreamVibe-VL则建议先切地壳。
研究者承认，现实世界的情况可能比他们的测试设置更加复杂，而今后的工作将研究用户偏好和环境如何随着时间的推移相互影响。
Google一直致力于AI系统，目前积极分析视频。 9月，Google 推出 agent基于Gemini的视频分析，其中模型本身决定更仔细地检查哪些场景。 随着Gemini 4 Argon,Google最近也展示了AI如何追踪机器人视频中较长的动作序列和点出出错。 StreamVibe的助手旨在了解随时发生的事情，并识别何时实际需要它的帮助。
在"主动奔驰基准测试"上，AI模型经常猜测或拒绝回答，而不是在视觉信息缺失时提出具体问题。 一个单独的视频理解基准测试揭示出空间和时间推理的重大弱点。 而对于自主的AI agents来说，对话本身会伤害性能，正如Microsoft研究者所显示的那样。
AI News Without the Hype – 由人类治愈
订阅THE DECODER免费阅读，每周AI通讯，我们独家的"AI雷达"前沿报告每年6次，全面存档访问，访问我们的评论部分。
现在订阅
读取全相。
订阅无韵报道。
- 全面查阅THE DECODER的每篇文章
- 没有广告
- 加入评论和社区讨论
- 每周一次通过邮件转载AI新闻
- 6x/年："AI雷达"——对AI最重要的话题进行深潜。
- 每日AI新闻，总是最新的
- 我们完整的十年档案
- 由一个在AI工作10年以上的团队负责
订阅到 The Decoder
本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Google wants video AI to understand what people need as they work
A research team led by Google has introduced StreamVibe, a video agent designed to decide for itself when to speak up. The tests show above all how hard that remains for today's AI models.
Researchers from Google, Stanford, Oxford, and several U.S. universities are working on an assistant that watches you cook or assemble furniture and speaks up only when needed. With StreamVibe, they've built a video agent designed to understand even vague requests such as "Please help me if I run into any difficulties."
According to the paper, existing video assistants are built for clear-cut commands like "Let me know when the water starts boiling," so they rarely pick up on unspoken intentions. They also have a hard time tracking what's already been done, which causes them to repeat advice or respond too late, and when users change their requests midway through, the assistants often stick to the original instructions.
Three building blocks for a more attentive assistant
To measure the problem, the team compiled a benchmark of about 4,100 examples from more than 183 hours of everyday life and software recordings. Gemini 2.5 Flash suggested when to offer help and what to say, while four human evaluators had to unanimously approve each example. The test covers three stages including clear instructions, vague goals, and requests that change over time. In one example, a user taking apart a toy excavator asks for progress updates and then switches to requesting a quality check after three minutes.
From there, the researchers trained StreamVibe-VL, a model apparently based on Alibaba's Qwen3.5-9B that decides every second whether to stay quiet or speak. A second training stage uses reinforcement learning to make the model more reserved so it doesn't constantly interrupt. A separate memory system tracks which requests are pending and which steps are complete, though the model alone decides whether to speak and what to say.
All tested models lose track of the conversation in continuous operation
According to the researchers, StreamVibe-VL improves response timing and quality by 10.7 percent over other models, while the memory system is designed to boost proactive interaction by another 12.3 percent and cut down on unnecessary interruptions, among other things.
StreamVibe-VL generally beats its base model and other open-source models by a wide margin, though Google's proprietary Gemini 3.8 Flash scores higher in several tests. According to the paper, the specialized training doesn't hurt general video comprehension.
Models often make the right call when they only have to decide at specific moments whether a prompt is appropriate. But when they process an entire video and have to build on their own earlier responses, performance across all tested systems plummets as errors compound, and a key content metric for the responses drops by up to 61.5 percent based on the system's own history. In videos longer than five minutes, the models rarely pick the right moment and find it especially hard to track a changed instruction over a long stretch.
In a GIMP tutorial on converting a photo to black-and-white documented in the paper and on the project page, the comparison model JoyAI says the finished image has already been saved even though the user hasn't opened it yet, while StreamVibe-VL gives the next step at the right moment, such as "Colors" and then "Desaturate." In a step-by-step guide for making rolled toast with jam, other models just describe what's happening on screen, such as the bread being flattened, while StreamVibe-VL advises cutting off the crust first.
The researchers acknowledge that real-world situations are likely more complex than their test setup, and future work will examine how user preferences and the environment influence each other over time.
Google has been working on AI systems that actively analyze videos for a while now. In September, Google introduced agent-based video analysis for Gemini, where the model itself decides which scenes to examine more closely. With Gemini 4 Argon, Google also recently showed how AI can track longer action sequences in robot videos and spot errors. StreamVibe's assistant is designed to understand what's happening at any moment and recognize when its help is actually needed.
On the ProactiveBench benchmark, AI models often guess or refuse to answer rather than ask specific questions when visual information is missing. A separate video comprehension benchmark revealed major weaknesses in spatial and temporal reasoning. And for autonomous AI agents, the conversation itself can hurt performance, as Microsoft researchers have shown.
AI News Without the Hype – Curated by Humans
Subscribe to THE DECODER for ad-free reading, a weekly AI newsletter, our exclusive "AI Radar" frontier report six times a year, full archive access, and access to our comment section.
Subscribe now
Read on for the full picture.
Subscribe for hype-free coverage.
- Full access to every article on THE DECODER
- No ads
- Join the comments and community discussions
- A weekly AI news recap via mail
- 6x/year: "AI Radar" — deep dives on the AI topics that matter most
- Daily AI news, always up to date
- Our full ten-year archive
- Covered by a team with 10+ years in AI
Subscribe to The Decoder
Auto-fetched and translated by AI Gem Box. All rights belong to the original author.