Vercel实验室有发布 agent-desktop，一个开源 CLI，用于在macOS,Windows和Linux上使用AI agent计算机。 agent-desktop通过操作系统的无障碍树来读取每个应用程序，而不是通过截图来读取。 agent得到一个JSON快照，上面有它能够作用的每个元素的回放。 1.0.2版本于2026年10月30日发货，在Apache-2.0下。 对于开发者来说，任何可 shell-cap可编码的agent现在都可以在没有视觉模型的情况下驱动本地应用。
TL; DR 数据
- 图书馆：Rust CLI和一个C图书馆。 60个指挥名称，56个行动
-运行于：macOS 13+,Windows 10 1809+或Server 2019+,Ubuntu 24.04+，在x64和ARM64上。 Plain CPU，无GPU.
- 性能：决定性、安全性。 还没有独立的计算机用途基准测试。
- 最佳：密集的Electron应用软件通过骨架转盘(自我报告)的信号减少78%至96%。
agent台式电脑是什么？
agent-desktop是一个来自Vercel Labs的Rust CLI和C库，用于阅读和操作桌面应用。 它是agents调用的工具，而不是agent. 它嵌入了没有LLM并且没有GUI. 每个命令都打印出一个JSON信封和出入口。 观察、操作和验证循环停留在你的agent。
agent-台式电脑是如何工作的？
每个命令都通过固定的链条流出。 CLI或C库称为"agent-desktop-core"，它只能通过平台Adapter特质到达OS. 一个平台板块然后与本地无障碍API交谈：
- macOS: AX API
- Windows:UI自动化
- Linux: AT-SPI(英语：
Core从不进口一个平台箱。 每个二进制被编译为驱动完全一个OS.
截图和校对
一个快照指定了合格的参考文件，如 @s8f3k2p9:e8. 裁判内部的快照ID使其自我描述。 以 INVALID ARGS 拒绝一个赤色的 @e8。 当 UI 更改时，一个动作返回 STALE REF 或 AMBIGUUS TARGET。 agent然后取出新的快照。
每一个错误都包含一个处理。 Agents只应在称安全时再试。 参考解析度很严格，没有关闭而不是猜测。
滑石和钻地
腾讯应用可以曝光上千个无障碍节点。 A-skeleton快照只读取了3个深等。 切开容器报告儿童数量，并获得一个钻探员。 即时快照 -- root 然后将一个区域读入同一即时快照。
建设者报告，Slack,VS代码和Notion中，78%至96%的地标切出与全树堆放。 在"Apple地圖"中，他报告了每张相貌的3,811个令牌下降到了273个。 一个模拟的52步任务从198克降到了31克。 这些是自报的数字，不是独立的基准测试。
默认无头
校正动作通过可访问性 API 不移动指针或聚焦。 这让agents在继续使用机器的同时工作。 全球-头旗允许在必要时进行物理输入。 每个回放动作还自动等取5000毫秒，以达到一个可见的，能动的，稳定的目标。
它与以像素为基础的计算机使用有何比较？
模型-侧式计算机使用工具的工作来自截图和像素坐标。 兹克 1Z的docs说每个截图花费大约1000到1800个输入符。 它的工具集定义新增每个请求约4500个输入符。 兹克 2Z 工具同样返回单击、 键入和滚动操作，您的代码执行。
agent-desktop是模型不可知和返回结构，而不是图像。 兹克 1Z的UFO2也使用UI自动化，但舰只则作为完整的Windows agent.
如何安装和使用 agent 桌面？
安装 npm 安装 - g agent 桌面。 Bun, Yarn和pnpm的工作也一样。 包将本地二进制捆绑为x64和ARM64.
- macOS：以agent-台式计算机权限授予无障碍权限 -- -- 向导。
Windows：您的终端完整性水平的应用程序不需要赠款。
- Linux：需要一个以-spi2-core的图形会话。
Agents通过被捆绑的docs通过agent-desktop技能来学习工具得到桌面。 技能也列在技能.sh上。 C库让Python,Swift,Go,Ruby和Node主机在进程中调用。 对于Electron网页内容，发布-cdp打开了用于agent-浏览器的DevTools端口。
关键外卖
- agent-desktop通过OS访问树给予任何agent计算机的使用。
- 在Apache-2.0下运行macOS,Windows和Linux,x64和ARM64.
- 无头反弹的动作从不移动指针或偷取焦点
- Skeleton快照将信使切为78%至96%(自报)。
- 没有无障碍树的Apps仍然无法到达。
请检查date=中的日期值 (帮助) GitHub repo, Docs and npm 包。 并加入我们的通讯。 等等，你在电报里吗？现在你也可以加入我们的电报。
需要和我们合作推广您的 GitHub Repo OR Hugging Face Page OR Product Release OR Webinar等吗？联系我们
米哈尔·萨特(Michal Sutter)是一名数据科学专业人员，拥有帕多瓦大学的数据科学硕士。 在统计分析、机器学习和数据工程的坚实基础上，Michal出色地将复杂的数据集转化为可操作的见解。
本文由 AI宝库自动抓取并翻译，版权归原作者所有；如需查看原文请点下方链接。
Vercel Labs has released agent-desktop, an open-source CLI for AI agent computer use on macOS, Windows and Linux. agent-desktop reads each app through the operating system’s accessibility tree, not through screenshots. An agent gets a JSON snapshot with a ref for every element it can act on. Version 1.0.2 shipped on October 30, 2026, under Apache-2.0. For developers, any shell-capable coding agent can now drive native apps without a vision model.
TL;DR
- Library: Rust CLI and a C library. 60 command names, 56 operational.
- Runs on: macOS 13+, Windows 10 1809+ or Server 2019+, Ubuntu 24.04+, on x64 and ARM64. Plain CPU, no GPU.
- Performance: Deterministic, retry-safe ref actions. No independent computer-use benchmark yet.
- Best: 78% to 96% fewer tokens on dense Electron apps via skeleton traversal (self-reported).
What is agent-desktop?
agent-desktop is a Rust CLI and C library from Vercel Labs for reading and operating desktop apps. It is a tool that agents call, not an agent. It embeds no LLM and has no GUI. Every command prints one JSON envelope and exits. The observe, act and verify loop stays in your agent.
How does agent-desktop work?
Each command flows through a fixed chain. The CLI or C library calls agent-desktop-core, which reaches the OS only through a PlatformAdapter trait. One platform crate then talks to the native accessibility API:
- macOS: AX API
- Windows: UI Automation
- Linux: AT-SPI
Core never imports a platform crate. Each binary is compiled to drive exactly one OS.
Snapshots and refs
A snapshot assigns qualified refs such as @s8f3k2p9:e8. The snapshot ID inside the ref makes it self-describing. A bare @e8 is rejected with INVALID_ARGS. When the UI changes, an action returns STALE_REF or AMBIGUOUS_TARGET. The agent then takes a fresh snapshot.
Every error carries a disposition. Agents should retry only when it says safe. Ref resolution is strict and fails closed rather than guessing.
Skeleton and drill-down
Dense apps can expose thousands of accessibility nodes. A --skeleton snapshot reads only 3 levels deep. Cut containers report a children_count and get a drill ref. snapshot --root then reads one region into the same snapshot.
The builder reported a 78% to 96% token cut versus full-tree dumps in Slack, VS Code and Notion. In Apple Maps, he reported 3,811 tokens per look falling to 273. A simulated 52-step task dropped from 198k to 31k tokens. These are self-reported figures, not an independent benchmark.
Headless by default
Ref actions go through the accessibility API without moving the pointer or taking focus. That is what lets agents work while you keep using the machine. The global --headed flag allows physical input when needed. Every ref action also auto-waits up to 5,000 ms for a visible, enabled, stable target.
How does it compare with pixel-based computer use?
Model-side computer use tools work from screenshots and pixel coordinates. Anthropic’s docs say each screenshot costs roughly 1,000 to 1,800 input tokens. Its toolset definition adds about 4,500 input tokens per request. OpenAI’s tool similarly returns click, type and scroll actions your code executes.
agent-desktop is model-agnostic and returns structure, not images. Microsoft’s UFO² also uses UI Automation, but ships as a full Windows agent.
How do you install and use agent-desktop?
Install with npm install -g agent-desktop. Bun, Yarn and pnpm work too. The package bundles native binaries for x64 and ARM64.
- macOS: grant Accessibility permission with agent-desktop permissions --guide.
- Windows: no grant needed for apps at your terminal’s integrity level.
- Linux: needs a graphical session with at-spi2-core.
Agents learn the tool from bundled docs via agent-desktop skills get desktop. A skill is also listed on skills.sh. The C library lets Python, Swift, Go, Ruby and Node hosts call it in process. For Electron web contents, launch --cdp opens a DevTools port for agent-browser.
Key Takeaways
- agent-desktop gives any agent computer use through OS accessibility trees.
- It runs on macOS, Windows and Linux, x64 and ARM64, under Apache-2.0.
- Headless ref actions never move your pointer or steal focus.
- Skeleton snapshots cut tokens 78% to 96% (self-reported).
- Apps without an accessibility tree remain out of reach.
Check out the GitHub repo, docs and npm package. Also, feel free to follow us on Twitter and don’t forget to join our 150k+ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.
Need to partner with us for promoting your GitHub Repo OR Hugging Face Page OR Product Release OR Webinar etc.? Connect with us
Michal Sutter is a data science professional with a Master of Science in Data Science from the University of Padova. With a solid foundation in statistical analysis, machine learning, and data engineering, Michal excels at transforming complex datasets into actionable insights.
Auto-fetched and translated by AI Gem Box. All rights belong to the original author.