Vercel Labs open-sources agent-desktop, a Rust CLI that gives any AI agent computer use through accessibility trees

What happened

Vercel Labs released agent-desktop 1.0.2 on October 30, 2026, under the Apache-2.0 license. The tool is a Rust CLI and C library that lets any AI agent control desktop applications on macOS, Windows, and Linux by reading each app through the operating system's accessibility tree rather than analyzing screenshots.

The release follows an earlier 0.x series and introduces Linux support with the 1.0.0 milestone on October 27, then 1.0.2 on October 30 with improved installation help. The npm package installs native binaries for x64 and ARM64 on all three platforms.

How the accessibility-tree approach works

Agent-desktop gives an agent a JSON-formatted snapshot of a desktop application's interface. Every actionable element gets a ref — an identifier like @s8f3k2p9:e8 that combines a snapshot ID with the element's position in the tree. The agent takes a snapshot, acts on a ref (click, type, scroll, etc.), and snapshots again to verify.

A ref is qualified by its snapshot ID, so a bare @e8 is rejected. When the UI changes between actions, an action returns STALE_REF or AMBIGUOUS_TARGET, and the agent takes a new snapshot to get fresh refs. Errors carry a disposition field that tells the agent whether a retry is safe.

The tool works headless by default — ref actions go through the accessibility API without moving the pointer or taking focus. A global --headed flag allows physical mouse and keyboard input. On macOS, CLI calls reuse a private command host that retains native accessibility objects so a removed row cannot silently resolve to a surviving row with the same identifier.

Supported platforms and commands

Agent-desktop runs on macOS 13+, Windows 10 1809+ or Server 2019+, and Ubuntu 24.04+ (glibc 2.35 or later), on both x64 and ARM64 architectures. It needs no GPU. macOS requires Accessibility permission granted through agent-desktop permissions --guide. Windows needs no grant for apps at the terminal's integrity level. Linux needs a graphical session with at-spi2-core installed.

The tool ships 60 command names, 56 of which are operational. The command groups include: snapshot, find, get, is, and screenshot for observation; click, type, select, toggle, scroll, and scroll-to for acting on refs; press, hover, drag, and mouse-wheel for keyboard and mouse; launch, close-app, list-apps, and list-windows for app management; clipboard-get and clipboard-set for clipboard access; and system commands including status, permissions, version, batch, and skills.

Why the approach matters for AI agents

Most agent computer-use tools today rely on screenshots and vision models to identify interface elements. This approach is expensive in tokens and latency, fragile to visual changes, and requires a vision-capable model for every action. Agent-desktop bypasses all three problems by reading the accessibility tree that the OS already maintains for assistive technologies.

The tool reports 78% to 96% fewer tokens on dense Electron apps through skeleton traversal, and actions are deterministic rather than probabilistic. An agent does not need a vision model to click a button — it reads the button's ref from the tree and acts on it. The agent loop stays in the agent's own code; agent-desktop only provides the observation and action layer.

How it compares to existing computer-use tools

The closest comparison is OpenAI's Computer Use model (GPT-6.1 Terra), which uses screenshots and a vision model to determine interface actions. Agent-desktop takes the opposite approach: no vision model, no screenshots, just the OS accessibility tree. Each approach has tradeoffs. Pixel-based computer use works on any application that renders to the screen, including games and custom UI. Accessibility-tree-based computer use only works on applications that publish an accessibility tree, which most native desktop apps and Chromium-based apps do.

A Chromium or Electron application may take 10 to 25 seconds to expose its web tree on cold start. Applications that do not publish an accessibility tree at all remain invisible to the tool. Agent-desktop's advantage is cost: the tool itself is free and open source, requires no API calls for observation, and the deterministic ref system eliminates the need for a vision model in the loop.

Installation and integration

Installation is a single npm command: npm install -g agent-desktop. The package bundles native binaries and platform helpers. Bun, Yarn, and pnpm can install the same package. For direct execution without a JavaScript launcher, GitHub Releases provide native archives.

Every agent-desktop command prints one JSON envelope with ok and either data or an error carrying a code, disposition, and suggestion. The consistent JSON output makes integration straightforward for any agent harness that can parse JSON from a subprocess. A C library (libagent_desktop_ffi) lets programs call the tool in-process. The tool's own skills.sh integration provides the loop, ref usage, verification, and retry rules that an agent needs.

Caveats

Agent-desktop can only see applications that publish an accessibility tree. Cold-starting a Chromium or Electron app can take 10 to 25 seconds before its tree becomes available. Snapshots, clipboard reads, and screenshots can contain private desktop content, and the documentation recommends keeping them out of public logs. The tool has no independent computer-use benchmark yet — the 78% to 96% token reduction figure is Vercel Labs' own.