AI 基本要素
什么是计算机Using Agents? AI 如何看到、点击、类型和导航软件
使用计算机的agents通过屏幕，cursors操作软件，键盘也和人们一样多。 这种通俗性是强大的，但视觉上的不确定性，长期的任务范围，以及广泛的权限，使得谨慎的控制至关重要。
在 Google 上的首选来源中添加 Unite.AI
使用计算机的agents是AI系统，可以通过点击，打字，滚动，拖取等动作来感知软件接口并运行。 它们不要求每个应用程序都有一个专门的API，而是通过人们使用的许多相同的视觉界面工作。
这种能力使agents在遗留软件、网站和桌面应用程序中具有广泛的影响力。 这也是推出不寻常的风险。 一个小的视觉误解可以重定向一个很长的任务，而接口访问可能暴露了被签名的用户能够看到或改变的一切。
为什么计算机使用事项
世界上许多重要的软件都是为人设计的，而不是agents. 一个应用程序可能没有公开的API，在程序上只暴露其部分功能，或者需要跨多个接口的工作。
传统的机器人流程自动化可以操作用户界面，但通常取决于预定义的选择器和工作流程。 一个使用计算机的agent 新增一个模型，可以解释不熟悉的屏幕，决定需要什么步骤，并在接口改变或出现出乎意料的对话框时进行调整。
这使得计算机使用通用工具。 同样的感知和动作循环可以潜在地导航电子表格，客户关系管理器，设计应用程序，浏览器，或内部门户。
电脑用户Agent如何工作
铺地是困难的中间步骤。 agent必须将“开出最新发票”等概念性指示与特定的可见行或控制相连接。 文本标签和可访问元数据可以有所帮助，但动态布局，基于canvas的界面，相重叠的元素，和模棱两可的图标可能迫使模型从像素中推理。
基本循环类似于一个人如何与软件互动。
- 观察：捕捉截图、无障碍树、文件结构、cursor位置和其他环境状态。
- 解释：识别可见的控制，内容，警告，以及当前屏幕与任务之间的关系。
- 选择动作：决定单击、键入、滚动、按键、等待或请求帮助。
- 执行：运行时间通过操作系统、浏览器或虚拟机接口执行动作。
- 验证：抓取已更改的屏幕并确定动作是否具有预期效果。
- 重复或停止：继续朝向目标，从错误中恢复，或在受保护的边界上暂停。
OpenAI对计算机使用agent的描述将能力设定为将视觉感知与推理和鼠标键盘动作相结合。 关键不是一次点击预测；而是感知、行动和新观察到状态之间的闭合循环。
Agent 能“看到”什么
计算机使用系统可以接收接口的不同表示。
屏幕截图提供一般像素级视图，即使在没有结构化的接口元数据时也能工作。 它们可以捕捉图表，图像，canvas应用，以及视觉布局。 然而，小目标，隐藏状态，和模棱两可的图标可能难以被解释。
无障碍树或文件结构以更结构化的形式暴露出控制和文字。 它们可以使元素选择更加精确，但可能会省略视觉信息，或者在应用程序中行为不一致。
混合系统结合了像素，结构元素，应用状态，有时还直接的API. 这往往比把每一个问题都当作纯粹的视觉模仿来对待更强大。
计算机使用对浏览器自动化
常规浏览器自动化使用代码和稳定选择器来进行已知的序列。 当页面结构得到控制时，这是快速和可预测的。 一个使用计算机的agent能动态地解释接口，可以决定要遵循的序列。
这些做法是相辅相成的。 一个强大的系统可能让agent决定必须做什么，然后通过确定自动化或API执行众所周知的操作。 像素级交互最有价值的地方是没有结构化接口或者无法事先指定路径。
计算机使用与API和工具调用
这种混合方法既能减少出错又能降低成本。 agent可能通过API来取取数据，打开桌面应用程序只是为了完成遗留表，然后通过另一个结构化的查询来验证最终记录。 将视觉控制作为几个工具中的一个，避免强迫每一个步骤通过最不可靠的界面。
API提供面向机器的接口，具有明确的操作和数据类型。 通常比点击屏幕更快、更准确、也更容易验证。 计算机的使用以更大的模糊性为代价提供了更广泛的相容性。
例如，agent可以通过API查询存货，但使用供应商的门户网站处理API不支持的例外。 工具调用可以将两种能力都暴露在模型上，而运行时间则倾向于最可靠的路由。
常用案例
- 后台业务：通过门户传送数据，处理例行表格并准备记录供批准。
- 研究：浏览网站、过滤结果、收集证据并更新工作文件。
- 软件测试：跟踪用户旅行，记录故障，检查视觉变化。
- 信息技术支助：检查设置、收集诊断材料并指导或执行核准的修正。
- 个人生产力：协调文件、网络应用程序、日历和文件。
- 可获取性：帮助用户操作本来难以导航的界面，但需遵守严格的隐私控制。
为什么计算机使用任务仍然困难
用户界面包含动态菜单，延迟加载，弹出，滚动区域，小目标，以及看起来相似但产生不同后果的状态。 一项任务可能需要数十个正确的步骤，因此即使是微小的每步出错率复合。
基准测试如OSWorld评价agents跨越了真实的计算机环境，而不是孤立地点击。 OSWorld 2.0将这一想法扩展至更长的工作流程，并随着状态变化和更大的变化。 Stanford的2026 AI Index技术性能分会报告快速改善，同时仍然显示人类在OSWorld上的性能仍有差距，强调令人印象深刻的演示并不等同于普遍可靠性。
主要失败模式
- 视觉定位错误：点击附近的控制器或外观类似的控制器。
- 状态混淆：假设一个已装入的页面，保存成功，或者未装入时活动窗口已更改。
——"长相漂移"：经过多起小动作后失去任务目标。
- 隐藏副作用：在agent只打算预览时提交表格或购买。
- 提示词注入：遵循嵌入在网页、文档或消息中的恶意指示。
- 信誉暴露：披露或滥用在签约会议上提供的信息。
- 不可扭转的错误：删除、出版、发送或购买，没有经过充分的审查。
安全和管制原则
界面内容不可信。 网页，电子邮件或文档可以包含旨在重定向agent的指示，恶意对话框可以模仿熟悉的确认。 该系统应将页面内容与政策相隔离，限制目的地和全权证书，并在出现意想不到的许可请求或相应确认时暂停。
计算机的使用应该在能够完成任务的最狭窄环境中运行。 专用浏览器配置，虚拟机，或沙盒限制访问无关的账户和文件。 会话应当使用特定任务证书和最小权限，而不是继承用户的整个桌面权限。
影响大的行动需要明确的确认界限。 系统应该在其提交，发送，收购，删除，发布，或改变敏感设置之前立即暂停，并向用户显示准确的目标和效果。
Agents还应将屏幕内容视为不信任的数据。 网页中遇到的指令不得超越系统政策或用户的任务。 下载，复制命令，对外通信需要单独验证。
如何评价计算机 Using Agent
任务的成功是必要的，但还不够。 各小组应衡量步骤的准确性、从意外状态中恢复、不必要的行动、时间、费用、敏感数据曝光、批准遵守情况以及反复运行的一致性。
评价环境应改变屏幕分辨率、布局、数据、时间和应用状态。 否则，agent可以在日常软件所发现的变异上失败的同时，过度地适应被脚本化的基准测试。
如何读取计算机使用图
读取从编号输入到可见结果的计算机使用图。 比较将区分出计算机使用与其最近捷径的边界隔离出，而控制路径则标记在现实世界的后果之前必须检查权威或不确定性。 这些标签仍然是普通的可响应的页面文本，这样它们就可以在电话上翻译，包装和回放。
电脑Using Agents是什么？AI Sees, Clicks, Types, and Navigates Software，视觉是一张地图而不是一个保证。 真正的实施可以结合阶段，重复步骤，或者增加人审批界限。 重要的是团队能否在部署的系统中识别出相同的功能，为每个边界指派一个所有者，并测试它们之间的过渡。 如果无法观察或测量某一阶段，故障将难以诊断。
计算机用户Agents
使用计算机的agents将人机软件接口作为一般工具提供。 它们可以到达缺乏适当API的系统，并适应固定自动化难以轻易预料的各种任务。
这种笼统性带有模糊性和广泛的潜在影响。 最强的设计更喜欢在可用时结构化的API，隔离计算机环境，验证每一个状态的变化，让人们控制相应的动作。
本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
AI Fundamentals
What Are Computer-Using Agents? How AI Sees, Clicks, Types, and Navigates Software
Computer-using agents operate software through screens, cursors, and keyboards much like people do. That generality is powerful, but visual uncertainty, long task horizons, and broad permissions make careful controls essential.
Add Unite.AI to your preferred sources on Google
Computer-using agents are AI systems that can perceive a software interface and operate it through actions such as clicking, typing, scrolling, dragging, and keyboard shortcuts. Instead of requiring a dedicated API for every application, they work through many of the same visual interfaces people use.
This ability gives agents broad reach across legacy software, websites, and desktop applications. It also introduces unusual risk. A small visual misunderstanding can redirect a long task, and interface access may expose everything the signed-in user can see or change.
Why Computer Use Matters
Much of the world’s important software was designed for people, not agents. An application may have no public API, expose only part of its functionality programmatically, or require work across several interfaces.
Traditional robotic process automation can operate user interfaces, but it usually depends on predefined selectors and workflows. A computer-using agent adds a model that can interpret unfamiliar screens, decide what step is needed, and adapt when the interface changes or an unexpected dialog appears.
That makes computer use a general-purpose tool. The same perception-and-action loop can potentially navigate a spreadsheet, customer relationship manager, design application, browser, or internal portal.
How a Computer-Using Agent Works
Grounding is the difficult middle step. The agent must connect a conceptual instruction such as “open the latest invoice” to a particular visible row or control. Text labels and accessibility metadata can help, but dynamic layouts, canvas-based interfaces, overlapping elements, and ambiguous icons may force the model to reason from pixels.
The basic loop resembles how a person interacts with software.
- Observe: capture a screenshot, accessibility tree, document structure, cursor position, and other environmental state.
- Interpret: identify visible controls, content, warnings, and the relationship between the current screen and the task.
- Choose an action: decide to click, type, scroll, press a key, wait, or ask for help.
- Execute: the runtime performs the action through an operating-system, browser, or virtual-machine interface.
- Verify: capture the changed screen and determine whether the action had the intended effect.
- Repeat or stop: continue toward the goal, recover from an error, or pause at a protected boundary.
OpenAI’s description of a computer-using agent frames the capability as combining visual perception with reasoning and mouse-and-keyboard actions. The key is not a single click prediction; it is the closed loop between perception, action, and a newly observed state.
What the Agent Can “See”
Computer-use systems can receive different representations of an interface.
Screenshots provide a general pixel-level view and work even when no structured interface metadata is available. They can capture charts, images, canvas applications, and visual layout. However, small targets, hidden state, and ambiguous icons can be difficult to interpret.
Accessibility trees or document structure expose controls and text in a more structured form. They can make element selection more precise but may omit visual information or behave inconsistently across applications.
Hybrid systems combine pixels, structured elements, application state, and sometimes direct APIs. This is often stronger than treating every problem as pure visual imitation.
Computer Use vs. Browser Automation
Conventional browser automation uses code and stable selectors to perform a known sequence. It is fast and predictable when the page structure is controlled. A computer-using agent interprets the interface dynamically and can decide which sequence to follow.
The approaches are complementary. A robust system may let the agent decide what must be done, then execute well-known operations through deterministic automation or APIs. Pixel-level interaction is most valuable where a structured interface is unavailable or the path cannot be specified in advance.
Computer Use vs. APIs and Tool Calling
This hybrid approach reduces both error and cost. The agent might retrieve data through an API, open a desktop application only to complete a legacy form, then verify the final record through another structured query. Treating visual control as one tool among several avoids forcing every step through the least reliable interface.
An API provides a machine-oriented interface with explicit operations and data types. It is generally faster, more precise, and easier to validate than clicking through a screen. Computer use provides broader compatibility at the cost of greater ambiguity.
For example, an agent might query inventory through an API but use the vendor’s web portal to handle an exception that the API does not support. Tool calling can expose both capabilities to the model while the runtime prefers the most reliable route.
Common Use Cases
- Back-office operations: transferring data across portals, handling routine forms, and preparing records for approval.
- Research: navigating websites, filtering results, gathering evidence, and updating a working document.
- Software testing: following user journeys, recording failures, and checking visual changes.
- IT support: inspecting settings, collecting diagnostics, and guiding or executing approved fixes.
- Personal productivity: coordinating files, web applications, calendars, and documents.
- Accessibility: helping users operate interfaces that are otherwise difficult to navigate, subject to strong privacy controls.
Why Computer-Use Tasks Remain Difficult
User interfaces contain dynamic menus, delayed loading, pop-ups, scrolling regions, small targets, and states that look similar but have different consequences. A task may require dozens of correct steps, so even a modest per-step error rate compounds.
Benchmarks such as OSWorld evaluate agents across real computer environments rather than isolated clicks. OSWorld 2.0 extends that idea to longer workflows with state changes and greater variation. Stanford’s 2026 AI Index technical performance chapter reports rapid improvement while still showing a remaining gap from human performance on OSWorld, underscoring that impressive demos do not equal universal reliability.
Major Failure Modes
- Visual grounding errors: clicking a nearby or look-alike control.
- State confusion: assuming a page loaded, a save succeeded, or the active window changed when it did not.
- Long-horizon drift: losing the task objective after many small actions.
- Hidden side effects: submitting a form or purchase when the agent intended only to preview it.
- Prompt injection: following malicious instructions embedded in a webpage, document, or message.
- Credential exposure: revealing or misusing information available in a signed-in session.
- Irreversible mistakes: deleting, publishing, sending, or purchasing without adequate review.
Security and Control Principles
Interface content is untrusted. A webpage, email, or document can contain instructions intended to redirect the agent, and a malicious dialog can imitate a familiar confirmation. The system should isolate page content from policy, restrict destinations and credentials, and pause when an unexpected permission request or consequential confirmation appears.
Computer use should run in the narrowest environment that can complete the task. A dedicated browser profile, virtual machine, or sandbox limits access to unrelated accounts and files. Sessions should use task-specific credentials and least-privilege permissions rather than inheriting the user’s entire desktop authority.
High-impact actions need a clear confirmation boundary. The system should pause immediately before it submits, sends, buys, deletes, publishes, or changes sensitive settings, and show the user the exact target and effect.
Agents should also treat screen content as untrusted data. Instructions encountered inside a webpage must not override system policy or the user’s task. Downloads, copied commands, and external communications require separate validation.
How to Evaluate a Computer-Using Agent
Task success is necessary but insufficient. Teams should measure step accuracy, recovery from unexpected states, unnecessary actions, time, cost, sensitive-data exposure, approval compliance, and consistency across repeated runs.
Evaluation environments should vary screen resolution, layout, data, timing, and application state. Otherwise an agent can overfit a scripted benchmark while failing on the variation found in everyday software.
How to Read the Computer Use Diagrams
Read the Computer Use diagrams from the numbered input toward the observable outcome. The comparison isolates the boundary that distinguishes Computer Use from its closest shortcut, while the control path marks where authority or uncertainty must be checked before a real-world consequence. The labels remain ordinary responsive page text so they can translate, wrap, and reflow on a phone.
For What Are Computer-Using Agents? How AI Sees, Clicks, Types, and Navigates Software, the visual is a map rather than a guarantee. A real implementation may combine stages, repeat a step, or add a human approval boundary. What matters is whether the team can identify the same functions in the deployed system, assign an owner to each boundary, and test the transition between them. If a stage cannot be observed or measured, failures there will be difficult to diagnose.
What to Remember About What Are Computer-Using Agents
Computer-using agents make the human software interface available as a general tool. They can reach systems that lack suitable APIs and adapt across tasks that fixed automation cannot easily anticipate.
That generality comes with ambiguity and broad potential impact. The strongest designs prefer structured APIs when available, isolate the computer environment, verify every state change, and keep people in control of consequential actions.
Auto-fetched and translated by AI Gem Box. All rights belong to the original author.