[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M71SPKHPTS6V08DBWMDF49GE":3,"skill-same:01M71SPKHPTS6V08DBWMDF49GE":139},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":16,"sourceHtmlUrl":17,"categories":18,"keywords":27,"publishedAt":48,"linkOnly":49,"markdownContent":50,"markdownBody":51,"frontMatter":52,"refName":78,"commitSha":79,"skillDirectory":80,"forksCount":81,"repositoryHtmlUrl":82,"repositoryHomepageUrl":6,"ownerHtmlUrl":83,"ownerBlogUrl":6,"ownerDescription":84,"updatedAt":85,"install":86,"resources":99},"01M71SPKHPTS6V08DBWMDF49GE","cua-driver",null,"Drive a native GUI app (macOS, Windows, Linux) via the cua-driver CLI (default) or MCP server; snapshot its accessibility tree, act through snapshot-bound element tokens, native menu paths, exact window geometry, or pixel coordinates, and verify from fresh state. Use when the user asks you to operate, drive, automate, or perform a GUI task in a real application on the host, or to continue, resume, or recall recent Cua activity.","sanurb\u002F.dotfiles","sanurb","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F60886336?v=4","01M71SKJS17F5YPRE27CVMEH1X","Santiago U","01M71SKJSTGEXEQ82HMAQP4ZAF",2,"MIT","https:\u002F\u002Fgithub.com\u002Fsanurb\u002F.dotfiles\u002Fblob\u002Fmain\u002Fconfig\u002Fpi\u002Fagent\u002Fskills\u002Fcua-driver\u002FLICENSE.md","https:\u002F\u002Fgithub.com\u002Fsanurb\u002F.dotfiles\u002Fblob\u002Fmain\u002Fconfig\u002Fpi\u002Fagent\u002Fskills\u002Fcua-driver\u002FSKILL.md",[19,24],{"categoryNo":20,"categoryName":21,"taxonomyType":22,"taxonomyTypeDesc":23},"01JBSEEDCAT000000000000010","后端与接口","SCENARIO","使用场景",{"categoryNo":25,"categoryName":26,"taxonomyType":22,"taxonomyTypeDesc":23},"01JBSEEDCAT000000000000023","Skill与智能体开发",[28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47],"cua","cua driver","driver","fresh","mode","wayland","you","port","paths","element","host","MCP","app","act","geometry","local","snapshot","server","windows","macos","2026-11-10 13:02:02",false,"---\nname: cua-driver\ndescription: Drive a native GUI app (macOS, Windows, Linux) via the cua-driver CLI (default) or MCP server; snapshot its accessibility tree, act through snapshot-bound element tokens, native menu paths, exact window geometry, or pixel coordinates, and verify from fresh state. Use when the user asks you to operate, drive, automate, or perform a GUI task in a real application on the host, or to continue, resume, or recall recent Cua activity.\nversion: 0.24.0 # x-release-please-version\nmetadata:\n  openclaw:\n    requires:\n      bins:\n        - cua-driver\n    envVars:\n      - name: CUA_DRIVER_EMBEDDED\n        required: false\n        description: Set to 1 when a macOS host app launches the driver in embedded mode.\n      - name: CUA_DRIVER_HOST_BUNDLE_ID\n        required: false\n        description: Bundle identifier of the macOS host app in embedded mode.\n      - name: CUA_DRIVER_PATH\n        required: false\n        description: Optional path to a cua-driver binary used by an embedding host.\n      - name: CUA_DRIVER_RS_ENABLE_WAYLAND\n        required: false\n        description: Set to 1 to enable the native Wayland backend.\n      - name: CUA_DRIVER_RS_MCP_HTTP_PORT\n        required: false\n        description: Optional port for the local MCP HTTP endpoint.\n      - name: CUA_DRIVER_RS_MCP_HTTP_TOKEN\n        required: false\n        description: Required host-generated bearer token when the local MCP HTTP endpoint is enabled.\n    homepage: https:\u002F\u002Fcua.ai\u002Fdocs\u002Fcua-driver\n---\n\n# cua-driver\n\n## Dotfiles integration — read first\n\nRead [LOCAL.md](LOCAL.md) before the upstream guidance below. This setup uses\nthe persistent `computer` MCP server for GUI workflows. LOCAL.md documents the\nrepo-owned Pi configuration and overrides upstream transport defaults. Installed\ntool schemas take precedence over stale examples.\n\nOrchestrates cross-platform app automation via `cua-driver`. Whenever\na user asks to drive a native app, follow the loop in this skill\nrather than calling tools ad-hoc — the snapshot-before-action\ninvariant is not optional and silently breaks if you skip it.\n\n## Consult recent Cua activity only for continuation\n\nWhen both `history_status` and `history_query` are advertised and the user asks\nto continue, resume, or recall prior Cua work, call `history_status` first. If\nhistory is healthy and access is admitted, make one bounded initial\n`history_query` before broad application or window discovery. Treat returned\nmetadata only as a lead and verify current state through the least intrusive\nappropriate source. Content, geometry, arguments, results, and user intent\nomitted from the metadata remain unknown.\n\nMake another bounded query only when the initial slice exposes a relevant\nsession or sequence boundary; never broaden a query to reconstruct excluded\nfields.\n\nContinue without history when either tool is absent, access is denied, the\nquery is empty, or history is unhealthy. Do not query history for unrelated\ntasks merely because the tools are advertised, and never mutate history\nlifecycle or settings.\n\n## Platform-specific reading — read this first\n\nThis file is the **cross-platform core**: snapshot invariant, CLI vs\nMCP choice, tool surface naming, behavior matrix, canonical loop,\npixel-click contract, common failure modes. The platform-specific\nmaterial (forbidden-list, accessibility tree implementation, launch\nsemantics, click dispatch) lives in companion files in this same\ndirectory:\n\n- **macOS** — read `MACOS.md` (no-foreground contract, forbidden\n  `open`\u002F`osascript`\u002F`cliclick` invocations, AXMenuBar navigation,\n  SkyLight pixel-click dispatch).\n- **Windows** — read `WINDOWS.md` (UIA tree vs AX, UWP \u002F\n  ApplicationFrameHost hosting, layered UIA+PostMessage click chain,\n  Session 0 isolation, Windows-specific focus-steal vectors).\n- **Linux** — read `LINUX.md` (X11 background input via AT-SPI +\n  XSendEvent and compositor-specific Wayland capabilities).\n\nCross-cutting topics also have their own files:\n\n- `BROWSER.md` — exact native-window binding, explicit browser preparation,\n  typed Chromium\u002FElectron page tools, input trust classes, and native\n  fallbacks for browser chrome and unsupported engines.\n- `RECORDING.md` — session recording + `replay_trajectory`.\n\nUse whichever combination matches the host. When in doubt, run\n`cua-driver doctor` — it reports the platform and the right entry\npoint.\n\n## Start with the narrowest semantic route\n\nBefore opening or operating an application, name the desired postcondition and\nuse the first applicable route below. Verify the result in the same domain\nbefore stopping or advancing:\n\n0. **Caller-provided headless\u002Fbackground operation for a non-GUI outcome.**\n   Prefer an exact application API\u002FSDK, service or database client, CLI, or\n   filesystem operation over imitating a user. This includes batch-safe file\n   moves, renames, copies, directory creation, archive extraction, data\n   conversion, and process inspection. Read the resulting semantic state back;\n   a zero exit status alone is not proof.\n1. **Typed Cua operation for an application or window outcome.** Use\n   `set_window_frame` for exact geometry, `invoke_menu` for a known native\n   application-menu path, typed browser tools for supported page content, and\n   clipboard tools for clipboard state. Verify with\n   `list_windows`, `get_browser_state`, or `clipboard_read`, respectively.\n2. **Background accessibility action.** Use a fresh AX\u002FUIA\u002FAT-SPI target.\n3. **Background pixel action.** Use the pixels from the same state snapshot.\n4. **Foreground delivery.** Retry only the action that evidence says could not\n   land in the background.\n5. **Desktop fallback.** Select an exact desktop target for that call only.\n   Later calls may return to an exact window target in the same session.\n\nUse Cua Driver when the outcome lives in an application's UI or window state,\nor when the user explicitly asks to operate that GUI. Once the task crosses\nthat boundary, do not replace Cua's targeted and verified actions with shell\nscripts that mutate the app UI. A shell is a capability of the calling agent,\nnot of the Cua Driver MCP server; an MCP-only client must not assume one exists.\n\n### Filesystem outcomes and GUI fallbacks\n\nWhen the requested outcome is a filesystem change and the caller has a\nheadless filesystem or command capability, keep it on rung 0. Enumerate the\nexact source set, decide the destination-conflict policy before changing\nanything, perform one batch-safe operation, then independently read back both\nsource and destination manifests. Do not open a file manager merely to mimic a\nmove, copy, or rename that the caller can execute and verify directly.\n\nIf the caller has no such capability, use the file manager as a GUI fallback\nand keep each claim narrow:\n\n1. After entering an inline rename and setting its value, commit it with the\n   platform's confirmation key, then take a fresh snapshot. Value readback from\n   the inline editor proves only that the editor changed; it does not prove the\n   filesystem rename committed.\n2. For a multi-selection, use the platform modifier (`cmd` on macOS, `ctrl` on\n   Windows\u002FLinux). On macOS and Windows, issue that modified click with\n   `delivery_mode:\"foreground\"` so the target observes physical modifier state;\n   a refused background attempt is an escalation signal, not a failed action to\n   trust or repeat. Re-snapshot before the next operation. Continue only when\n   every intended item is selected and the prior selection was preserved.\n3. After a cross-window drag or paste, verify the destination contains the\n   complete expected set and the source reflects copy-versus-move semantics.\n   A delivered drag, keypress, or menu action is not file-operation proof.\n4. If a destination conflict presents an unrecognized policy or ambiguous\n   partial result, stop that GUI path and surface the unresolved state instead\n   of retrying blindly.\n\n### Clipboard outcomes and GUI fallbacks\n\nWhen the requested postcondition is an exact value on the system clipboard,\nrather than the literal gesture of selecting and copying it, keep the operation\nsemantic. Read the value from the narrowest typed source, call\n`clipboard_write`, then prove the real clipboard state with `clipboard_read`.\nFor browser content, this means reading the page with `get_browser_state` and\nwriting the exact observed text; a passive page-text ref does not need to be\nclicked first.\n\nUse visual selection followed by the platform copy hotkey only when the user\nexplicitly asks for that gesture, the source cannot expose the value\nsemantically, or direct clipboard tools are unavailable. Treat that as a GUI\nfallback: re-snapshot before acting, verify the selected range when the\napplication exposes it, and escalate only the delivery step that cannot land\nin the background.\n\n## The no-foreground principle (window phase)\n\nDuring window-targeted background actions, **the user's frontmost app MUST NOT\nchange.** Every platform\nhas its own list of forbidden commands:\n\n- macOS: any `open` invocation, any `osascript` that mutates GUI\n  state, `cliclick`, `cghidEventTap` writes targeting another app's\n  window. Full list in `MACOS.md`.\n- Windows: any `Start-Process` that triggers a `ShowWindow`\u002F`SetForegroundWindow`\n  on the target, `WScript.Shell.AppActivate`, attaching to the\n  foreground thread for input forwarding. Full list in `WINDOWS.md`.\n\nIf you reach for a command that says \"activate\", \"foreground\",\n\"raise\", or \"make key\", stop and translate to the cua-driver tool\nthat does the same intent without focus-stealing.\n\n`delivery_mode:\"foreground\"` is a user-visible takeover boundary, not an\nautomatic retry. Use it only when the user already authorized foreground\ncontrol for this workflow or after asking for approval. It may change focus,\nworkspace, and the compositor cursor while the action runs; restoration is\nbest-effort. If background delivery is unavailable and foreground control is\nnot authorized, stop with the driver's refusal instead of silently escalating.\n\nA desktop target is an explicit per-call choice to operate the visible desktop\nand therefore uses foreground\u002Fsystem input. Use it only after the narrower\nwindow ladder has been attempted and verified. Permission policy must still\nadmit the display resource. Never infer desktop permission from a failed action\nor a public session label.\n\n## GUI transport defaults — prefer cua-driver over GUI shell shims\n\n**Default transport is the `cua-driver` CLI for one-off calls** — `Bash`\nshelling out to `cua-driver \u003Ctool-name> '\u003CJSON-args>'`. Each CLI invocation\nowns a disposable transport session that is cleaned up after its response.\nUse one persistent `cua-driver mcp` connection for a multi-call GUI workflow\nthat needs shared cursor, recording, browser, or named-session state. A public\nsession label is not a credential and a later one-shot process cannot adopt\nthe previous process's lifecycle merely by repeating that label.\n\nCLI wins for isolated inspection and management because it picks up rebuilds\ninstantly, failures are easier to diagnose, and there's no per-tool\nschema-load overhead. Persistent MCP wins for an ordered action loop.\n\nEvery reference to `click(...)`, `get_window_state(...)` etc. in this\nskill means `cua-driver click '{...}'` — translate to MCP form only\nwhen MCP is requested.\n\n### Claude Code computer-use compatibility flag\n\nFor normal Claude Code use, keep the default CLI or `cua-driver` MCP\nserver path above. If the user explicitly wants Claude Code's\nvision\u002Fcomputer-use-style flow, they can register:\n\n```bash\ncua-driver mcp-config --client claude   # then paste + run the printed line\n```\n\nThe compatibility flag is retained for old setup snippets, but the standalone\n`screenshot` tool was removed. It does not add or replace tools. Use\n`get_window_state({pid, window_id})` for a window-local accessibility snapshot\nand PNG, or `get_desktop_state()` for an explicitly authorized desktop capture.\n\n## Using cua-driver from the shell\n\nTool names are `snake_case`, management subcommands are\n`kebab-case` — no ambiguity. Tools invoked as `cua-driver\n\u003Ctool-name> '\u003CJSON-args>'`. Management subcommands:\n\n- `cua-driver serve` — start an explicit persistent service when short-lived\n  clients must share runtime state or a platform identity. Bare MCP owns its\n  runtime directly on Windows\u002FLinux and uses the signed app service on macOS;\n  `cua-driver mcp --socket \u003Cendpoint>` selects a service explicitly.\n  One-shot CLI tool calls still use the service path. macOS users: see\n  `MACOS.md` for the LaunchServices-routed launch form.\n- `cua-driver stop` \u002F `status`\n- `cua-driver list-tools`, `describe \u003Ctool>`\n- `cua-driver recording start|stop|status` — see `RECORDING.md`\n- `cua-driver check-update [--json] [--no-cache]` — read-only \"is a newer release available?\" probe. Same payload as the `check_for_update` MCP tool; pair with `cua-driver update --apply` to install.\n\nCanonical multi-step workflow within one persistent MCP connection (example\nshape — platform-specific launch idioms in the per-OS companion file):\n\n```bash\n# Start the service once, then connect one MCP client with `cua-driver mcp`.\n# The calls below are tool calls on that same connection, not separate shell\n# invocations of `cua-driver \u003Ctool>`.\nlaunch_app({\"bundle_id\":\"...\"})\n# → {pid: 844, windows: [{window_id: 10725, ...}]}\nget_window_state({\"pid\":844,\"window_id\":10725})\n# Use the returned structuredContent.elements[].element_token:\nclick({\"pid\":844,\"element_token\":\"s0000002a:14\"})\nverify_state({\"pid\":844,\"window_id\":10725,\"expect\":[{\"element\":{\"selector\":{\"label_contains\":\"Saved\"},\"exists\":true}}]})\n```\n\nFor Chromium page content, keep the same native window selection but switch to\nthe browser capability loop: use one lifecycle session, bind `(pid, window_id)` with\n`get_browser_state`, snapshot the returned tab, then use `browser_click`,\n`browser_type`, or `browser_navigate`. Read `BROWSER.md` before using this\nroute. Browser target ids, tab ids, and refs are session-scoped and stale refs\nmust be replaced by a fresh snapshot.\n\n## Agent cursor overlay\n\nVisual cursor overlay for demos and screen recordings. It initializes on the\nfirst cursor-bearing action, including `move_cursor`, and follows the\ntransport's implicit or named lifecycle session. Toggle a named cursor with\n`set_agent_cursor_enabled` to hide or re-show it. The embedded\n`cua.default` theme uses a session-colored pointer over a larger,\ncursor-shaped glow in the same session color. The glow fades to transparent\naround the full silhouette. Action marks use the same\nsession-colored center and white-outline treatment, plus a tighter, softer\nglow. This pairing preserves contrast across varied backgrounds. It provides animations for\nidle, observe, click, drag, scroll, text, key, navigation, app, transfer,\nrecording, and system activity. Motion knobs:\n`set_agent_cursor_motion` takes any subset of `start_handle`,\n`end_handle`, `arc_size`, `arc_flow`, `spring` — tuneable at runtime,\npersisted to config.\n\n**Agent-control safety.** Keep this overlay enabled whenever the agent is\ncontrolling pointer or keyboard input. Cursor-bearing and keyboard actions\nautomatically re-show their session cursor, even if it was hidden while idle.\nOn Linux, ordinary `move_cursor({x,y})` moves only this synthetic cursor.\n`move_cursor({x,y,scope:\"desktop\"})` is the explicit escape hatch that moves\nthe user's compositor cursor and must not be used unless the user asked for\ndesktop-pointer control. Raw Wayland input that requires\n`delivery_mode:\"foreground\"` crosses the same user-visible takeover boundary.\n\nDelivery and target context is shown as host-owned chips inside the session\nbadge. Themes own the twelve action animations only. The session name and\ncontext chips fade independently, so an active tool can show its execution\ncontext without revealing a session name that has already faded.\n\n**Per-session cursors.** Each MCP session automatically owns its own\ncursor, keyed by the session's id (the proxy mints one session id per\nMCP connection and the daemon scopes the cursor, config overrides, and\nrecording to it). The CLI and SDK contracts take the declared `session`\nexplicitly. Cursor-theme controls no longer accept `cursor_id` or the legacy\nshape\u002Fcolor\u002Fimage fields. Input-delivery tools may still use `cursor_id` to\nname a virtual pointer; it never selects artwork. The default cursor is Cua\nblue, while each named session receives a stable fill from the built-in\npalette. Select only preinstalled\nthemes with `set_agent_cursor_theme`; theme source paths and inline animation\ndata are never accepted through an agent tool. Use the trusted local\n`cua-driver cursor-theme` workflow to validate, compile, preview, install,\nlist, or remove custom themes.\n\n**Visibility caveat (AX runs).** On a pure accessibility-action run\n(clicking by `element_index`), the first action **seeds the cursor\non-screen a short distance from the target and plays a brief glide +\npulse** — not the long Bezier sweep a cursor already on-screen would\ntrace from its previous spot. It's subtle and easy to miss in a\nrecording. If you want a clearly _gliding_ cursor for a demo or screen\nrecording, do a pixel click (`click({pid,x,y})`) or a `move_cursor`\nfirst to put the cursor on-screen; subsequent AX actions then glide the\nfull path normally.\n\nPixel `click` already glides the overlay. Do not call `move_cursor`\nimmediately before `click` on the same target; that plays two glides.\nUse `move_cursor` to place the overlay without clicking, or as the\none-time seed above before AX actions.\n\nRequires a suitable UI event loop. Service and private-worker runtimes provide\none. On macOS, a same-process SDK runtime or `cua-driver mcp --direct` without\na certified host main-thread adapter returns a structured\n`facility_unavailable` result for overlay operations; do not treat that as a\nsuccessful cursor move. One-shot CLI adapters do not own an overlay\nthemselves.\n\n## The core invariant — snapshot before and verify after every action\n\n**Every action MUST be bracketed by observation for the session's effective\nscope.** Use `get_window_state(pid, window_id)` before a window action (or\n`get_desktop_state(session)` in desktop scope), then use `verify_state` for an\nexpressible window-scoped postcondition. In effective desktop scope,\n`verify_state` is intentionally refused with `window_scope_disabled`; verify\nwith a fresh `get_desktop_state` result and agent-owned visual\u002Fsemantic reading.\n\n- **Before** — the pre-action snapshot resolves the `element_index`\n  you're about to use. Indices from previous turns are stale; the\n  server replaces the element index map on every snapshot, keyed\n  on `(pid, window_id)`. Indices from turn N don't resolve in turn\n  N+1, and indices from window A don't resolve against window B of\n  the same app. Skip this and element-indexed actions fail with\n  `No cached AX state`.\n- **After** — `verify_state(pid, window_id, expect)` checks a bounded,\n  deterministic postcondition. Results are `satisfied`, `unsatisfied`, or\n  `unknown`; `unknown` never means success. Set `include_screenshot:true` when\n  the outcome also needs visual reading. The driver returns that final image\n  without interpreting it. A multimodal agent harness reads the image and owns\n  the stop\u002Fretry\u002Fladder decision.\n\n`unknown_reason` distinguishes invalid\u002Funsupported predicates, untrusted web\ncontent, ambiguous matches, missing targets, unavailable observations, and\n`stability_unproven`. A positive final sample that was not observed for the\nrequested consecutive sample count is `stability_unproven`, not success.\nNegative element existence is conservative: when an accessibility projection\ncannot prove its search domain exhaustive, absence remains `unknown`.\n\nDo not make the driver invent task meaning or retry actions automatically.\nFor postconditions not expressible by `verify_state`, take a fresh state\nsnapshot and let the agent judge the tree and\u002For image explicitly. This applies\nto pixel clicks and desktop actions too.\n\n### Read action facts without confusing them with task success\n\nA successful action returns `effect` and `route`, with optional typed\n`delivery`, `evidence`, and `escalation`. These fields describe the actuator;\nthey do not declare the user's task complete.\n\n- `confirmed` means the driver has publishable value readback or window-change\n  evidence for that action.\n- `partial` means only `delivery.delivered_count` was delivered.\n- `unverifiable` means the driver cannot prove the effect.\n- `suspected_noop` means available evidence suggests no useful change.\n- `refused` means the selected route deliberately did not deliver.\n\nThe route vocabulary is intentionally cross-platform:\n`accessibility`, `synthetic_events`, `global_input`, `dom`, and\n`trusted_input`. Do not branch on private OS transport names.\n\nAn optional escalation is a harness instruction, never an automatic retry:\n\n- `pixel`: refresh visual state and choose an exact pixel target;\n- `foreground`: explicitly select foreground delivery if the authorization\n  stack admits the tool and exact target;\n- `page`: bind the native window to a supported browser page route;\n- `session`: a legacy compatibility signal from an older capture-scope daemon;\n  current callers choose a desktop target on the specific action instead.\n\nBranch on the closed reason vocabulary:\n`route_unavailable`, `delivery_failed`, `effect_unconfirmed`,\n`suspected_noop`, and `permission_required`.\n\nAfter any action, keep using `verify_state` or a fresh state snapshot for the\nactual task postcondition. The multimodal harness owns visual reading and the\ndecision to stop, retry, or advance the ladder.\n\n## Choose the target on each action\n\nA session owns lifecycle, cursor, recording, cleanup, and telemetry state. It\ndoes not store the current capture modality. Select an exact target on each\naction:\n\n```jsonc\n{\"target\":{\"kind\":\"window\",\"pid\":844,\"window_id\":10725}}\n{\"target\":{\"kind\":\"desktop\",\"display_id\":\"primary\"}}\n```\n\nThe window target uses window-local coordinates and the background\u002Fforeground\ndelivery ladder. The desktop target uses screen coordinates and foreground\ndelivery. A desktop action does not disable window tools for later calls.\n\n`start_session` is optional. For a multi-call run, prefer a short public\n`session` label and pass the same label on every call that accepts it. The label\nis call-scoped: if a later call omits it, that call uses the authenticated\ntransport's implicit session instead. Unnamed calls on one transport reuse that\nimplicit identity. The default idle TTL is five minutes. Call\n`start_session(session)` to name or configure a run before acting, or to revive\nan ended name.\n\nDo not use `config set capture_scope` or `set_config`; that key is retired and\nstale values on disk are ignored. `start_session.capture_scope`,\n`get_session_state`, and `escalate_session` are deprecated compatibility\nsurfaces. There is no `deescalate_session`. Reserved fields such as\n`_session_id` are transport metadata and cannot create authority.\n\n## Keep authorization separate from sessions\n\nThe trusted host selects one permission profile at startup. `standard` keeps\nthe normal profile behavior and residual approval requirements, `bounded`\nrequires a reviewed capability manifest and has no runtime approval path, and\n`unrestricted` bypasses Cua approval prompts after explicit risk acceptance.\nHard invariants plus managed and user policy remain binding in every profile.\n\nAn optional capability manifest is a deny-by-default ceiling in `standard` and\n`unrestricted`; `bounded` requires one. It can remove tools or typed resources\nfrom the selected profile, but it cannot grant a tool, resource, or approval\nbypass that another authorization layer denies. Approval is considered only\nafter the tool and every adapter-attested resource are inside manifest scope.\n\nUse the canonical startup pair together:\n\n```bash\ncua-driver mcp \\\n  --permission-mode standard \\\n  --capability-manifest .\u002Fcapabilities.yaml \\\n  --approve-capability-manifest\n```\n\nCapability manifest v3 omits file-level `mode` and `ask.tools`. Its\n`allow.tools` list is nonempty. Lifetime fields are optional in `standard` and\n`unrestricted`; `bounded` requires both `expires_after` and `idle_timeout`.\nThe older `--session-policy` names remain compatibility aliases and must not be\nused in new configurations.\n\nStarting, ending, naming, reconnecting, or omitting a session never changes\npermission authority. A public session label is lifecycle metadata, never a\ngrant, caller identity, or bearer credential.\n\n### Why window selection is the caller's job now\n\n`get_app_state` used to pick a window for you via a max-area heuristic\nthat returned the wrong surface on apps with large off-screen utility\npanels. Concrete reproducer: IINA's OpenSubtitles helper (600×432\noff-screen) out-area'd the visible 320×240 player window, so\n`get_app_state(pid)` screenshot'd the invisible panel and clicks landed\nthere silently. The new `get_window_state(pid, window_id)` makes the\ncaller name the window explicitly — the driver validates that the\nwindow belongs to the pid and is on the current Space\u002Fdesktop, then\nsnapshots exactly what was asked for. Enumerate candidates via\n`list_windows` or read the `windows` array `launch_app` already\nreturns.\n\n## Behavior matrix\n\n### Perception is mode-agnostic — `get_window_state` returns BOTH\n\n`get_window_state(pid, window_id)` **returns both the accessibility\ntree AND a screenshot by default.** There is no capture mode to pick\nand nothing to configure — you ground on the tree and the screenshot\ntogether, and you cross-check one against the other. This matters\nbecause the tree **lies** on some surfaces:\n\n- **Electron** echo-confirms a `set_value` \u002F `type_text` against the AX\n  shim while the rendered text view never changed.\n- **Catalyst** (iOSAppOnMac) exposes null \u002F placeholder `AXValue`s.\n- **Virtualized \u002F off-viewport list rows** report bogus frames (an\n  `h:1` height, an off-screen origin) for rows that aren't actually\n  laid out.\n\nA grounding screenshot is present by default, so when the tree looks\nwrong you look at the pixels **in the same response** — no second\ncapture, no mode flip.\n\n> **Perf opt-out — `include_screenshot`.** `include_screenshot`\n> (boolean, default `true`) is the one knob, and it is a **perf** knob,\n> not a modality choice. Default returns both (grounding-first). Pass\n> `include_screenshot:false` to skip the screen grab and get the tree\n> only — the cheap path when you're just **re-indexing before an\n> element ax action** and don't need to re-ground on pixels. The\n> `ax`\u002F`px` decision still lives at action time, not here.\n\n> **`capture_mode` is DEPRECATED and ignored.** It is still _accepted_\n> on `get_window_state` so old callers don't error, but it has **no\n> effect** — both the tree and the screenshot come back regardless of\n> what you pass (`ax`, `vision`, `som`, anything). There is no\n> `ax`\u002F`vision`\u002F`som` capture choice anymore. Drop the word \"vision\"\n> for perception entirely. (The tool named `screenshot` is separate —\n> raw PNG, no AX walk — and unrelated.)\n\n### The modality is chosen at ACTION time — `ax` vs `px`\n\nYou don't pick a capture mode; you pick **how you address the target**\non the action call, and that one choice selects the rung:\n\n- **element ax action** — pass `element_token` (preferred), or the exact\n  `element_index` + `snapshot_id` pair from the same response.\n  Dispatches through the **accessibility rung**: AXPress (macOS) \u002F UIA\n  Invoke (Windows) \u002F AT-SPI `doAction` (Linux). Backgroundable,\n  z-order-independent, and the only **driver-verifiable** rung.\n- **element px action** — pass `x`, `y`. Dispatches through the **pixel\n  rung**, reading the coordinate straight off the screenshot that's\n  already in the `get_window_state` response. Best-effort; the caller\n  confirms the effect.\n\n`ax`↔`element_index`, `px`↔pixel `x,y`. We retired the word \"vision\"\nfor the _dispatch_ path — it conflated perception with dispatch.\nPerception is always both; dispatch is `ax` or `px`.\n\n**The keyboard family has both forms too.** `type_text`, `press_key`,\nand `hotkey` take a snapshot-bound element target (ax) **or** `x,y` (px) — mutually\nexclusive, same as the pointer tools. The px form **pixel-clicks at\n`(x,y)` to establish real renderer focus, then delivers the\nkeystroke(s)** to the now-focused element (it reuses `click`'s\ncoordinate translation + `delivery_mode`). That gives e.g.\n`type_text({pid, window_id, x, y, text})` as a one-call focus-then-type\nfor Chromium\u002FElectron inputs the AX path can't reach, and\n`hotkey({pid, x, y, keys:[\"cmd\",\"v\"]})` to paste into a specific field.\n\n**Typing default (the ladder).** Call `type_text` directly with\n`element_token` (ax) — it targets the field, no pre-click. On\nElectron\u002FCatalyst the AX layer echoes the write without rendering it,\nso the driver returns `effect:\"unverifiable\"` with\n`escalation.target:\"pixel\"` there (never a false `effect:\"confirmed\"`) —\nfollow it, and cross-check the\nscreenshot in the response (the only ground truth). Escalate to the px\nform — `type_text({pid, window_id, x, y, text})` — which pixel-clicks\nto focus, then types. **If the target control is closed** (a search\nbutton, a collapsed field), AX-press to open it first (AX actions work\nin the background): a px focus-click won't reliably open _and_ focus a\nclosed control, so the text leaks into whatever's already focused.\nEscalate to `delivery_mode:\"foreground\"` only if it still drops.\n\n**`set_value` stays AX-only by design** — use it when the intent is to\nreplace a control's whole value: dropdowns, checkboxes, sliders, steppers,\nand native text fields such as Finder's inline rename editor. Use\n`type_text` when the intent is to insert text at the current selection or\ncursor. Its pixel counterpart is a `click`\u002F`drag` on the control, not a\n\"set value at a pixel.\" So: insert text → `type_text` (ax+px); replace a\nsurfaced native value → `set_value`; pixel-manipulate a control →\n`click`\u002F`drag`.\n\n**Action responses carry closed action facts**\n\nUse the `effect`, `route`, optional `delivery`, `evidence`, and\n`escalation` rules in “Read action facts without confusing them with task\nsuccess” above. The old `verified`, `path`, coordinates, scope, and\n`escalation.recommended` response fields no longer exist.\nThe full wire contract and 0.14 migration notes are in\n`..\u002F..\u002F..\u002Fdocs\u002Faction-result-contract.md`.\n\nA successful accessibility value write can still return\n`effect:\"unverifiable\"` when the provider publishes its new value only after\nthe action call unwinds. Take a fresh snapshot before retrying; an immediate\nretry can duplicate text. An explicit pixel escalation is reserved for a web\nsurface whose accessibility layer echoed the write without proving that the\nrenderer observed it.\n\n`get_window_state` itself, when the AX tree comes back empty (a non-AX\nsurface like Electron\u002FChromium\u002Fcanvas), returns `degraded: true`\nplus an observation-specific escalation hint — normally pointing at pixels (you\nstill have the screenshot from the same call to click off).\n\n**Platform nuance for action escalation.** On **Wayland** an unfocused\nwindow cannot be pixel-targeted in the background (libei →\n`background_unavailable`), so the action target is\n**`foreground`, not `pixel`**. macOS, X11, and most Windows surfaces\ncan pixel-target in the background, so they target `pixel`. See\n`LINUX.md` \u002F `WINDOWS.md`.\n\n## The verify-then-escalate ladder (algorithm)\n\nEvery snapshot already hands you both the tree and the screenshot, so\nverifying never means \"go take a screenshot\" — it means cross-check\nthe tree against the pixels you already have, and only change\n_dispatch rung_ on a real signal. Walk the rungs:\n\n```\n# Routes 0–1 — resolve non-GUI, exact geometry, and supported page outcomes first\n# Use a caller-provided semantic operation for a non-GUI outcome, then read it back.\n# For exact window geometry: set_window_frame(...), then list_windows(...) readback.\n# For a known native menu command: invoke_menu(pid, window_id, path), then verify its effect.\n# For supported page content: get_browser_state(...), typed browser action, refresh refs.\n# Continue below only when the postcondition actually requires native UI interaction.\n\n# Route 2 — element AX\u002FUIA\u002FAT-SPI action, backgrounded\nget_window_state(pid, window_id)            # tree + screenshot, both, always\nresp = click(pid, element_token)            # or type_text \u002F set_value \u002F press_key\ncheck = verify_state(                       # bounded structured read-back\n    pid, window_id,\n    expect=[...],\n    include_screenshot=true                 # optional evidence for multimodal harness\n)\n\nif check.status == \"satisfied\":\n    done                                    # driver-verified\n\nif check.status == \"unknown\" and check has an image:\n    harness reads the image                  # model-owned visual interpretation\n    if visual outcome is satisfied: done\n\n# escalate only on a real signal\nif resp.effect == \"suspected_noop\"\n   or resp.escalation.target == \"pixel\"\n   or get_window_state.degraded            # empty tree → non-AX surface\n   or check.status != \"satisfied\"\n   or the tree looks wrong vs the screenshot:   # e.g. an h:1 \u002F off-viewport row\n\n    # Route 3 — element px action off the SAME screenshot\n    pick the target pixel from the screenshot already in the response\n    click(pid, x, y)                        # background pixel — still no foreground\n    verify_state(..., include_screenshot=true)\n    if it landed: done\n\n# Route 4 — background delivery was dropped (insert\u002Fclick never arrived)\nif resp.escalation.target == \"foreground\"\n   or the px action still did nothing:\n    require existing user authorization for visible foreground control\n    otherwise stop and ask; do not retry automatically\n    re-call the same action with delivery_mode:\"foreground\"\n    # on Wayland this is the ONLY escalation — px-bg can't target an\n    # unfocused window there; see LINUX.md\n    verify again\n\n# Route 5 — per-call desktop fallback\n# Reach this only after semantic, AX, window-pixel, and foreground-window\n# delivery have all been exhausted and verified ineffective.\nget_desktop_state()                         # full primary display\ndesktop_action(target={kind:\"desktop\", display_id:\"primary\"}, ...)\nget_desktop_state()                         # verify in the same coordinate frame\n```\n\nThe two ideas to hold onto: (1) the AX tree **lies** on canvas \u002F web \u002F\nCatalyst \u002F virtualized surfaces, so an unchanged-or-bogus tree plus\n`suspected_noop`\u002F`degraded` — or a tree that simply disagrees with the\nscreenshot — is your cue to do an **element px action** off the\nscreenshot you already have; (2) `px` is a _conscious_ switch to the\npixel addressing path, not a different capture.\n\n**Window state → what works**\n\n| state                      | `get_window_state`                                                                             | element-index click (AX\u002FUIA) | `press_key` commit                                    | pixel click                    |\n| -------------------------- | ---------------------------------------------------------------------------------------------- | ---------------------------- | ----------------------------------------------------- | ------------------------------ |\n| frontmost                  | ✅                                                                                             | ✅                           | ✅                                                    | ✅                             |\n| backgrounded \u002F visible     | ✅                                                                                             | ✅                           | ✅                                                    | ✅                             |\n| **minimized**              | ✅                                                                                             | ✅ (actions fire in place)   | ❌ silent no-op — use `set_value` or click equivalent | ❌ no on-screen bounds         |\n| hidden                     | ✅                                                                                             | ✅                           | depends                                               | ❌                             |\n| on another desktop \u002F Space | ⚠️ tree may be stripped on some apps — response carries `off_space: true` so you can detect it | ✅                           | ✅                                                    | ❌ not in current-desktop list |\n\n**Critical cell — minimized + keyboard commit.** The keystroke\nreaches the app but accessibility focus doesn't propagate to renderer\nfocus on a minimized window. Workarounds in order of preference:\n`set_value` to write the field's entire value directly, or\nelement-index-click a commit-equivalent button (Go, Submit,\ncheckbox). Tell the user the window needs to un-minimize only as a\nlast resort.\n\n## The canonical loop\n\n```\n# for multi-call work, repeat the same session label on every call that accepts it\nlaunch_app(target, session)\n  → pick window_id from the returned `windows` array\n    (or call list_windows(pid) separately)\n  → get_window_state(pid, window_id)\n    → [act]  # pass target={kind:\"window\", pid, window_id}\n  → verify_state(pid, window_id, expect)  # structured check; optional image\nend_session(session?)             # optional explicit cleanup\n```\n\nAn existing-profile `end_session` restores the Chromium browser's\nremote-debugging toggle when Cua enabled it and no other Cua session still uses\nthat browser process and dismisses the exact native consent prompt. If exact\ncleanup cannot be proven, session cleanup fails closed and a later\n`end_session` retries it.\n\nFor screen-absolute work, replace the window portion with\n`get_desktop_state() → action(target={kind:\"desktop\",display_id:\"primary\"}, ...)\n→ get_desktop_state()`. Desktop actions use coordinates from that exact\nfull-display image.\n\n`launch_app` now returns a `windows` array alongside the pid, so the\ncommon case collapses to two calls (`launch_app` → `get_window_state`)\nwithout a separate `list_windows` hop.\n\n**Prefer a named session for multi-call work.** Choose a short label (for\nexample, `session: \"research-1\"`) and pass the same value on every call that\naccepts it. Passing it once is not sticky: a later call that omits `session`\nuses the transport's implicit session. Call `start_session(session)` when you\nneed to name or configure the run before acting, or to revive a name after\n`end_session`. For one-off or deliberately unlabeled work, omission is valid\nand the transport still gets one private lifecycle identity and visible agent\ncursor. A public label makes inspection and cleanup easier, but it is not a\ncredential. End with `end_session` when useful; transport close or the\nfive-minute idle TTL also reclaims it.\n\n**Concurrent runs\u002Fsubagents:** each transport gets its own implicit session.\nAlso,\n`launch_app` is idempotent — two runs that\nlaunch the same app get the **same** instance (and on single-instance apps\nlike Calculator, the same window), so they clobber each other. Give each run\nits **own connection** (for independent lifecycle\u002Fcursor ownership) AND pass\n`creates_new_application_instance: true` to `launch_app` (→ its own window).\nThe element cache is keyed on `(pid, window_id)` and the cursor on the private\nlifecycle session, so distinct instances and transports keep the runs isolated.\n\n**Parallelism vs. ordering.** Distinct sessions give distinct _cursors_, not\ndistinct _connections_. Subagents that share one `cua-driver mcp` (stdio)\nconnection have their tool calls **serialized** by the transport — they take\nturns, not run in parallel. That's not a correctness problem (session + window\nisolation means they can't collide), just a throughput one. For genuinely\nparallel agents, give each its **own connection**: separate `cua-driver mcp`\nprocesses, or point each agent's MCP client at the daemon's HTTP endpoint.\nSet `CUA_DRIVER_RS_MCP_HTTP_PORT` and a host-generated\n`CUA_DRIVER_RS_MCP_HTTP_TOKEN` of at least 32 characters, then send\n`Authorization: Bearer \u003Ctoken>` to `POST http:\u002F\u002F127.0.0.1:\u003Cport>\u002Fmcp`. The daemon\nserves connections concurrently; per-connection ordering keeps each agent's own\nsequence (e.g. `3 → + → 1 → =`) correct.\n\n`list_apps` is for app-level discovery (answering \"what's installed \u002F\nrunning \u002F frontmost?\") — not part of the core action loop. Skip it\nin the loop. For **window-level** questions — \"does this app have a\nvisible window?\", \"which desktop is this window on?\", \"which of this\npid's windows is the main one?\" — call `list_windows` instead; the\napp record doesn't carry window state on purpose. In the common\nsingle-window case you can skip `list_windows` entirely and read the\n`windows` array that `launch_app` already returned.\n\n### Snapshot and act with a snapshot-bound target\n\nCall `get_window_state({pid, window_id})` with the `window_id` from\n`launch_app`'s `windows` array (or a fresh `list_windows({pid})` if\nyou're interacting with a long-lived process). It returns **the tree\nand the screenshot together** by default, so you can both dispatch by\n`element_token` and ground on pixels from one call — no config change,\nno mode flip. When you're just re-indexing before an element ax action\nand don't need fresh pixels, pass `include_screenshot:false` to skip\nthe grab (a perf knob, not a modality choice).\n\nThe response carries:\n\n- `tree_markdown` — every actionable element tagged `[N]`; the structured row\n  with the same `element_index` carries its opaque `element_token`. The tree can be very large (Finder is\n  ~1600 elements, ~190 KB); when it exceeds token limits the MCP\n  harness saves it to a file and returns the path. Use `Bash` +\n  `jq -r '.tree_markdown'` + `grep` to pull the section you need.\n- `effect` \u002F `escalation` \u002F `degraded` — the verify-then-escalate\n  signals (see the behavior matrix above): `degraded: true` means the\n  tree came back empty (non-AX surface), so you act by **`px`** off the\n  screenshot in the same response.\n- `screenshot_file_path` — present when the screenshot was written to\n  disk instead of inlined (you passed `screenshot_out_file`, or the\n  context-saving CLI path); otherwise the frame is inlined.\n- `screenshot_width` \u002F `_height` \u002F `_scale_factor` — dimensions of\n  the captured image. Present whenever a screenshot was taken (i.e.\n  unless you passed `include_screenshot:false`).\n\n**Getting the screenshot as a file (CLI and context-constrained agents):**\n\n```bash\n# write to file — stdout stays readable (AX\u002FUIA tree \u002F summary only, no base64)\ncua-driver get_window_state '{\"pid\":N,\"window_id\":W,\"screenshot_out_file\":\"\u002Ftmp\u002Fshot.png\"}'\n\n# CLI --screenshot-out-file flag is equivalent\ncua-driver get_window_state '{\"pid\":N,\"window_id\":W}' --screenshot-out-file \u002Ftmp\u002Fshot.png\n```\n\nPass `screenshot_out_file` when using `get_window_state` via CLI or\nfrom an agent whose context window can't absorb ~31 KB of inline\nbase64 (e.g. OpenCode with a local Ollama model). The MCP image\ncontent block is omitted from the response when this param is set —\nthe model receives only the tree and `screenshot_file_path`, then\nreads the image from disk.\n\n**The tree and the screenshot are complementary, not redundant — and\nthey come from the _same_ call.** Each half carries signal the other\ncan't, which is exactly why you cross-check them:\n\n- The **tree** tells you _what's clickable_ — roles, labels,\n  snapshot-bound element handles, advertised actions, parent-child\n  structure. This is the ground truth for an **element ax action**.\n- The **screenshot** tells you _which one_ — the tree often has many\n  buttons with similar or empty labels (\"Delete\", \"OK\", anonymous\n  UUID-labeled buttons, repeated static-text), and visual context\n  disambiguates. Captions, colors, layout relationships visible in\n  pixels often don't show up in the tree at all (especially in\n  Chromium \u002F Electron \u002F web content) — and the screenshot is where you\n  catch the tree _lying_ (an `h:1`\u002Foff-viewport row, a Catalyst null\n  value).\n\nDefault to dispatching by `element_token` (the **element ax action**) —\nit's the verifiable, backgroundable rung. Do an **element px action**\n(`x,y` off the same screenshot) when the tree can't disambiguate\n(repeated\u002Fempty labels), when it's empty (`degraded` — non-AX\nsurface), when an action came back `suspected_noop`, or when the tree\ndisagrees with the pixels. You never re-capture to switch — the\nscreenshot is already there; you just change _how you address_ the\ntarget.\n\nReach for pixel coordinates only when the target is a canvas \u002F\nvideo \u002F WebGL \u002F custom-drawn surface that isn't in the tree\n(see \"Pixel-coordinate clicks\" below).\n\nThe `actions=[...]` list on each element is **advisory**, not\nauthoritative. cua-driver does not pre-flight check against it —\n`click({pid, element_token})` always attempts the default action (or\nthe action you pass) and surfaces whatever the target returns. **Try\nthe click first** — pivot only on the returned error code.\n\n### Tool dispatch table\n\nEvery row assumes a fresh `get_window_state`. Prefer its opaque\n`element_token`. If a client uses the visible integer instead, it must send\nthe response's `snapshot_id` with `element_index`; bare indices fail closed in\n0.17. Pixel-only forms remain independent of snapshot handles.\n\n| Intent                           | Tool                                                                                                            | Notes                                                                                                                                                                                                                 |\n| -------------------------------- | --------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| List an app's windows            | `list_windows({pid})`                                                                                           | returns `window_id`, `title`, `bounds`, `z_index`, `is_on_screen`, `on_current_space`. Already included in `launch_app`'s response — only call this for long-lived pids                                               |\n| Set an exact window frame        | `set_window_frame({pid, window_id, x, y, width, height})`                                                       | uses the platform window manager and returns `confirmed` only after geometry readback; inspect `list_windows` again before continuing when the result is not confirmed                                                |\n| Invoke a native application menu | `invoke_menu({pid, window_id, path:[\"Window\",\"Arrange\",\"Left\"]})`                                              | resolves exact immediate-child labels from live native state at every hop; refuses missing, ambiguous, or disabled segments and never falls back to pixels; verify the command's semantic effect afterward          |\n| Snapshot a window                | `get_window_state({pid, window_id})`                                                                            | returns `tree_markdown` + `screenshot_*`; populates the `(pid, window_id)` element_index cache                                                                                                                        |\n| Verify a postcondition           | `verify_state({pid, window_id, expect, include_screenshot?})`                                                   | polls bounded structured predicates; returns `satisfied`, `unsatisfied`, or `unknown`. Optional final image is interpreted by the agent harness, never by the driver                                                 |\n| Left click                       | `click({pid, element_token})` or `click({pid, window_id, element_index, snapshot_id})`                          | default `action: \"press\"`. Pixel form: `click({pid, x, y})` (window_id optional) — `modifier: [\"cmd\"\\|\"ctrl\"]`                                                                                                        |\n| Double-click \u002F open              | `double_click({pid, element_token})`                                                                            | Default action when the element advertises one (Open on Finder items \u002F openable rows), else stamped pixel double-click at the element's center                                                                        |\n| Right click \u002F context menu       | `right_click({pid, element_token})` or `click({pid, element_token, action:\"show_menu\"})`                       | Browser page content should use the typed route where available; see `BROWSER.md`                                                                                                                                     |\n| Type at cursor                   | `type_text({pid, text, element_token})` (ax) or `type_text({pid, text, window_id, x, y})` (px)                  | ax focuses the element then writes via the platform's text-set primitive; **px** pixel-clicks `(x,y)` to focus the renderer, then types — the one-call fix for Chromium\u002FElectron inputs the AX path can't reach       |\n| Set whole non-text control value | `set_value({pid, element_token, value})`                                                                         | **AX-only by design** — dropdown\u002F`AXPopUpButton`, checkbox, slider, stepper; **also the keyboard-commit workaround on minimized windows.** For text use `type_text`; to pixel-manipulate a control use `click`\u002F`drag` |\n| Scroll                           | `scroll({pid, direction, amount, by, element_token})`                                                           | synthesizes per-pid PageUp\u002FPageDown\u002Farrows                                                                                                                                                                            |\n| Focus + send key                 | `press_key({pid, key, element_token, modifiers})` (ax) or `press_key({pid, key, x, y})` (px)                    | ax targets the element before posting the key; **px** pixel-clicks `(x,y)` to focus, then sends the key                                                                                                               |\n| Send key to pid                  | `press_key({pid, key, modifiers})`                                                                              | no focus change; key goes to pid's current focus                                                                                                                                                                      |\n| Modifier combo                   | `hotkey({pid, keys})` (no focus) or `hotkey({pid, x, y, keys})` (px)                                            | e.g. `[\"cmd\",\"c\"]` \u002F `[\"ctrl\",\"c\"]`; posted per-pid, not HID tap. **px** pixel-clicks `(x,y)` to focus a field first, e.g. `[\"cmd\",\"v\"]` to paste into it                                                             |\n\n`list_windows.z_index` uses one portable convention: higher integer\nvalues are closer to the front. Select a frontmost candidate with the\nmaximum non-null value. If all values are `null` (as they can be on\nnative Wayland), use an explicit fallback; never treat `null` as zero\nor infer stacking from array order. The `windows` records returned by\n`launch_app` use the same convention.\n\nIn effective desktop scope, the foreground\u002Fsystem equivalents omit\n`pid`\u002F`window_id` and pass `scope:\"desktop\"`: `click`, `scroll`, `drag`,\n`move_cursor`, `type_text`, `press_key`, and `hotkey`. Coordinates are\nscreen-absolute pixels from the latest `get_desktop_state` image.\n\n**Window-scope keyboard\u002Ftext primitives require `pid`.** They use the named\ntarget's per-pid event-post path. Only a strict\u002Feffective desktop session may\nomit `pid`, and it intentionally routes keyboard input to the current\nforeground application.\n\n**Why the snapshot-bound element target is the primary path:** works on hidden \u002F\noccluded \u002F off-desktop windows, avoids focus steal, and fails closed after a\ntree rebuild instead of silently retargeting a reused index. Labels tell you\nwhat you're clicking. Reach for pixel\ncoordinates only when the accessibility tree can't.\n\n## Cross-platform parameter contract\n\nThe capture, dispatch, and addressing params — `session`,\n`delivery_mode`, `capture_mode` (deprecated\u002Fignored — see the behavior\nmatrix; still in the schema only so old callers don't error), `scope`,\n`modifier`, `button`, `element_index`, `snapshot_id`, `element_token` — are a **shared\nschema contract**: identical _shape_ (`type`\u002F`enum`\u002F`items`) on macOS,\nWindows, and Linux.\nThey compose from canonical fragments in\n`cua-driver-core::tool_schema` (+ `capture_mode`), and a CI gate\n(`schema_consistency_test`) runs every tool's live `tools\u002Flist` through a\nstructural checker on each platform, so the three surfaces can't\nsilently drift. _Contributor note:_ when you add or edit one of these\nshared params on a tool, pull from the fragment — don't re-hand-write the\nJSON, or the gate fails. (Descriptions may legitimately vary per tool;\nthe gate compares shape, not prose.)\n\nTwo consequences for callers:\n\n- **`session` is accepted on every action and cursor tool, on all three\n  platforms.** It's cursor-wired where the platform glides a cursor and\n  schema-accepted everywhere else — so the same `session` you pass on\n  macOS is no longer _rejected_ by Windows\u002FLinux, which previously\n  refused unknown keys via `additionalProperties:false`.\n- **`delivery_mode` (`\"background\"` default \u002F `\"foreground\"`) is on the\n  whole input family** — `click`, `double_click`, `right_click`, `drag`,\n  `scroll`, `type_text`, `press_key`, `hotkey` — uniformly. The\n  `foreground` rung briefly fronts the target, acts, then restores the\n  prior frontmost: the explicit last resort when a background attempt\n  didn't land. **`foreground` is a reaction and a user-authorization gate,\n  never a prediction.** Always\n  fire the `background` default first and let the driver tell you it\n  can't (a `background_unavailable` error with\n  `escalation.recommended == \"foreground\"`, or a successful action result\n  with `escalation.target == \"foreground\"`) — or observe a confirmed no-op —\n  _before_ you consider escalation. Then use it only when the user already\n  authorized visible foreground control or after asking for approval.\n  Do **not** reason \"it's a GTK\u002FChromium\u002FElectron app, so background will\n  drop, so I'll front up-front\": the toolkit lists in the tool schemas\n  are the _driver's_ internal detectors, not a checklist for you to front\n  on a guess. (Concretely: GIMP's GTK toolbox accepts background pixel\n  clicks fine — a preemptive foreground click there just steals the\n  user's focus for nothing.) What each platform's _background_ rung can\n  actually carry differs (e.g. a Windows background click can't carry\n  `modifier` state — see `WINDOWS.md`); the schema is uniform, the\n  residual limits are per-OS.\n\n**Required-set contract.** `click` requires nothing (`required:[]`),\n`scroll` requires `[\"direction\"]`, `zoom` requires\n`[\"window_id\",\"x1\",\"y1\",\"x2\",\"y2\"]` — same on every platform. `pid` is\n**conditionally** required (needed unless a windowless desktop-scope\ncall) and validated in code with a clear error, NOT pinned in the schema\n— so omitting `pid` for a desktop-scope action is no longer\nschema-rejected.\n\nGenuinely platform-specific params stay OUT of the shared contract by\ndesign (launch-app identifiers, the Windows-only `debug_window_info`, the\nmacOS-only status-only `check_permissions.prompt`). The per-OS files list the\nresiduals that matter when you drive on that platform.\n\n## Pixel-coordinate clicks\n\nThe pixel path (`click({pid, x, y})`) is for surfaces the\naccessibility tree doesn't reach — canvases, video players, WebGL,\ncustom-drawn controls. Coords are **window-local screenshot pixels**\n(same space as the PNG `get_window_state` returns). Top-left origin,\ny-down. The driver handles screen-point conversion internally.\nPassing `window_id` alongside `x, y` is optional but recommended —\nit pins the coordinate conversion to the window whose screenshot\nproduced the pixel.\n\nPNGs returned by `get_window_state` are capped at **1568 px long-side\nby default** (`max_image_dimension` config), matching Anthropic's\nmultimodal-vision downsampling limit. The image the model reasons\nover and the image the click tool's coordinate system lives in are\nthe **same resolution** — just look at the PNG, pick a pixel, click\nat that pixel. No scaling math.\n\nThis is the default because the mismatch between \"rendered\nthumbnail\" and \"native PNG\" was a recurring coord-estimation\nfootgun. If you opt out (explicit `max_image_dimension=0` for\npixel-perfect verification flows), the old rule applies: don't\neyeball coords from whatever your client renders — it may be\n2-4× smaller than the PNG on disk, and a 2% error in thumbnail\nspace becomes ~80 px in the real image.\n\nFor precise targeting on small \u002F dense UIs:\n\n1. `get_window_state({pid, window_id})` → image capped at 1568\n   long-side plus `screenshot_width` \u002F `screenshot_height`. Write to\n   disk via `--screenshot-out-file \u003Cpath>`.\n2. Look at the PNG. Since it matches what you see, pick the target\n   pixel directly.\n3. When precision matters, draw a crosshair on the image (do\n   **not** crop — cropping loses the coordinate system) and verify\n   before clicking:\n\n```python\nfrom PIL import Image, ImageDraw\nimg = Image.open('\u002Ftmp\u002Fshot.png')\ndraw = ImageDraw.Draw(img)\nx, y = \u003Cyour_coordinate>\nr = 18\ndraw.ellipse([x-r, y-r, x+r, y+r], outline='red', width=4)\ndraw.line([x-30, y, x+30, y], fill='red', width=3)\ndraw.line([x, y-30, x, y+30], fill='red', width=3)\nimg.save('\u002Ftmp\u002Fshot_annotated.png')\n```\n\n4. Only dispatch the click after the user (or your own re-read of\n   the annotated image) confirms the crosshair is on target.\n\nAddressing variants:\n\n- `click({pid, x, y})` — single left-click.\n- `click({pid, x, y, count: 2})` — double-click.\n- `click({pid, x, y, modifier: [\"cmd\"\\|\"ctrl\"]})` — modifier click.\n  Accepts any subset of `cmd\u002Fshift\u002Foption\u002Falt\u002Fctrl`.\n- `right_click({pid, x, y})` — also takes `modifier`.\n\nThe pixel path animates the agent cursor overlay but never warps\nthe real cursor (the per-pid event paths the driver uses on macOS\nand Windows route around HID synthesis). If the pid has no on-screen\nwindow the call errors with `pid X has no on-screen window` — you\nneed a visible window to anchor the conversion. Dispatch details\n(SkyLight on macOS, layered UIA+PostMessage on Windows) are in the\nper-OS companion files.\n\n## Web-rendered apps (browsers, Electron, Tauri)\n\nFor Chromium-family browsers and Electron, use the exact, session-scoped\nbrowser capability workflow in **`BROWSER.md`**. It keeps native\n`(pid, window_id)` selection as the entry point, makes setup explicit through\n`browser_prepare`, and distinguishes trusted browser input from an explicitly\nrequested synthetic DOM event.\n\nUse the native `get_window_state` and AX\u002FPX action ladder for browser chrome,\npermission prompts, downloads, file pickers, Safari, Firefox, Tauri, and any\nembedded webview for which exact browser binding is unavailable. The legacy\n`page` tool remains a compatibility surface; do not use it as the starting\npoint for new browser workflows.\n\n## Verify after every action — mandatory\n\n**Always** verify after an action. Prefer\n`verify_state({pid, window_id, expect})` for structured state such as a\nwindow's existence\u002Fbounds or a semantic element's existence, value, enabled\nstate, or selected state. Use its bounded poll and stable-sample requirement\ninstead of hand-written sleeps. `unknown` means the driver could not establish\nthe predicate; it is not success. Once a session has effective desktop scope,\nuse a fresh `get_desktop_state(session)` result instead—window-scoped\n`verify_state` is denied by that capture policy.\n\nPass `include_screenshot:true` when visual evidence is useful. The same result\nthen contains a fresh final window image. The driver still evaluates only the\nstructured predicates; the multimodal agent harness reads the pixels and\ndecides whether to stop, retry, or advance the ladder. For a postcondition not\nexpressible by the tool, explicitly take a fresh `get_window_state` snapshot\nand have the harness judge its tree and image.\n\nSwitch to an **element px action** only on a real signal: the action\nresponse carried `effect:\"suspected_noop\"`, verification returned\n`unsatisfied`\u002F`unknown`, the snapshot came back `degraded` (empty tree →\nnon-AX surface), the tree looks unchanged\u002Funreadable or disagrees with the screenshot, or\n`escalation.target` points you there (`pixel`). That's the\nverify-then-escalate ladder in the behavior-matrix section. If the tree\nis unchanged AND the screenshot confirms nothing moved, the action\nlikely failed silently — **tell the user what you attempted and what\nyou observed**, don't paper over with \"done\" language (and consider\n`delivery_mode:\"foreground\"` when `escalation.target ==\n\"foreground\"`). Agents that skip this step report success on\nsilently-dropped actions — the single most common failure mode.\n\n## Recording trajectories\n\nSession-scoped action recording + replay, for demos, regressions,\nand training data. Only invoke when the user explicitly asks to\nrecord a session — the skill does not auto-enable this. CLI surface:\n`cua-driver recording start|stop|status`; raw tools:\n`start_recording` \u002F `stop_recording`. Video capture (main display →\n`recording.mp4`) is on by default; pass `record_video: false` to opt out.\n\nSee **`RECORDING.md`** for the full flow: enable\u002Fdisable, turn folder\ncontents, replay via `replay_trajectory`, and the element_index\ndoesn't-survive-across-sessions caveat.\n\n## Common error patterns (cross-platform)\n\n| Error text                                                                         | Meaning                                                                                                                                                                          | Fix                                                                                                                                                                                                                  |\n| ---------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| `No cached AX state for pid X window_id W`                                         | You either skipped `get_window_state` this turn, or passed a different `window_id` to the click than the one the snapshot cached against                                         | Call `get_window_state({pid: X, window_id: W})` first — the same window_id you intend to click in                                                                                                                    |\n| `snapshot_id_required` \u002F `stale_element_token`                                    | A bare index was supplied, or a newer snapshot superseded this target                                                                                                            | Re-run `get_window_state`; use the new `element_token`, or send its `snapshot_id` with the matching integer                                                                                                          |\n| `window_id W belongs to pid P, not …`                                              | Passed a window_id that's owned by a different process                                                                                                                           | Use `list_windows({pid: X})` to enumerate this pid's own windows                                                                                                                                                     |\n| `ambiguous_window_target`                                                          | A PID-only window action matched multiple eligible top-level windows                                                                                                              | Use the returned candidates or `list_windows({pid: X})`, select the intended sibling, and retry with its explicit `window_id`                                                                                       |\n| `AX action … failed with code …` \u002F `UIA invoke failed`                             | Element doesn't support the default action                                                                                                                                       | Try `show_menu`, `confirm`, `cancel`, `pick`, or fall through to a pixel click on the element's center                                                                                                               |\n| `The user doesn't want to proceed with this tool use. The tool use was rejected …` | The harness uses this _exact_ string for BOTH a permission-prompt denial AND a manual interrupt (Esc \u002F stop) of a running tool — they are indistinguishable from the tool result | Treat as \"tool canceled, no result, await the user.\" Do NOT paraphrase (\"you stopped me\") — quote the literal message and name the canceled tool + its args, so the user can tell what was in flight vs. what landed |\n\nPlatform-specific errors (TCC dialogs on macOS, Session 0 \u002F UAC\nprompts on Windows, AT-SPI bus issues on Linux) live in their\nrespective companion files.\n\n## Things to avoid\n\n- **Never** reuse an element target across a re-snapshot of the same window.\n  A new snapshot invalidates older tokens immediately. Bare `element_index`\n  input is rejected; use `element_token` or `element_index` + `snapshot_id`.\n- **Don't conflate the two addressing modes.** The tree gives you\n  `element_index` handles; the screenshot (same call) gives you the\n  pixel frame. An **element ax action** addresses by index, an\n  **element px action** by `x,y`. Default to `element_index` and only\n  do a px action on a real signal (`suspected_noop` \u002F `degraded` \u002F\n  repeated labels \u002F tree-disagrees-with-pixels). Don't pass an\n  `element_index` you read off the screenshot, and don't pixel-click a\n  coordinate you computed from the tree's (possibly lying) frame\n  without checking it against the image.\n- **Prefer accessibility actions over pixels.** `click({pid, x, y})`\n  works for canvas \u002F WebView regions, but it lands blindly on raw\n  coordinates. Exhaust accessibility paths (menu bars, cmd-k palettes,\n  toolbar items, keyboard shortcuts) before dropping to coordinates.\n  (The AX path does **not** skip the agent-cursor overlay — it seeds and\n  pulses the session cursor and draws a focus rect on the targeted\n  element; it just doesn't play a long glide on the very first action.\n  See \"Agent cursor overlay\" for the demo-recording caveat.)\n- **Never** drive destructive actions (delete files, close unsaved\n  documents, send messages, submit forms) without explicit user\n  intent for that specific destructive step.\n- **Never** launch apps autonomously; confirm with the user first\n  unless their original request clearly implies the launch.\n\n## Example end-to-end task\n\n**User:** \"Open the Downloads folder in the system file manager.\"\n\n1. `launch_app({bundle_id: \"com.apple.finder\", urls: [\"~\u002FDownloads\"]})`\n   on macOS, or `launch_app({name: \"explorer\", args: [\"%USERPROFILE%\\\\Downloads\"]})`\n   on Windows. Returns `{pid, windows: [{window_id, title, ...}]}`.\n   Idempotent launch; the driver opens a hidden window via the\n   platform's launch primitive — zero activation, no focus steal.\n2. `get_window_state({pid, window_id})` → verify the expected window\n   title is present with a populated tree (sidebar, list view, files).\n3. Done.\n\nPlatform-specific examples and edge cases (Finder menu navigation,\nExplorer ribbon, GNOME Files) live in the per-OS companion files.\n","# cua-driver\n\n## Dotfiles integration — read first\n\nRead [LOCAL.md](LOCAL.md) before the upstream guidance below. This setup uses\nthe persistent `computer` MCP server for GUI workflows. LOCAL.md documents the\nrepo-owned Pi configuration and overrides upstream transport defaults. Installed\ntool schemas take precedence over stale examples.\n\nOrchestrates cross-platform app automation via `cua-driver`. Whenever\na user asks to drive a native app, follow the loop in this skill\nrather than calling tools ad-hoc — the snapshot-before-action\ninvariant is not optional and silently breaks if you skip it.\n\n## Consult recent Cua activity only for continuation\n\nWhen both `history_status` and `history_query` are advertised and the user asks\nto continue, resume, or recall prior Cua work, call `history_status` first. If\nhistory is healthy and access is admitted, make one bounded initial\n`history_query` before broad application or window discovery. Treat returned\nmetadata only as a lead and verify current state through the least intrusive\nappropriate source. Content, geometry, arguments, results, and user intent\nomitted from the metadata remain unknown.\n\nMake another bounded query only when the initial slice exposes a relevant\nsession or sequence boundary; never broaden a query to reconstruct excluded\nfields.\n\nContinue without history when either tool is absent, access is denied, the\nquery is empty, or history is unhealthy. Do not query history for unrelated\ntasks merely because the tools are advertised, and never mutate history\nlifecycle or settings.\n\n## Platform-specific reading — read this first\n\nThis file is the **cross-platform core**: snapshot invariant, CLI vs\nMCP choice, tool surface naming, behavior matrix, canonical loop,\npixel-click contract, common failure modes. The platform-specific\nmaterial (forbidden-list, accessibility tree implementation, launch\nsemantics, click dispatch) lives in companion files in this same\ndirectory:\n\n- **macOS** — read `MACOS.md` (no-foreground contract, forbidden\n  `open`\u002F`osascript`\u002F`cliclick` invocations, AXMenuBar navigation,\n  SkyLight pixel-click dispatch).\n- **Windows** — read `WINDOWS.md` (UIA tree vs AX, UWP \u002F\n  ApplicationFrameHost hosting, layered UIA+PostMessage click chain,\n  Session 0 isolation, Windows-specific focus-steal vectors).\n- **Linux** — read `LINUX.md` (X11 background input via AT-SPI +\n  XSendEvent and compositor-specific Wayland capabilities).\n\nCross-cutting topics also have their own files:\n\n- `BROWSER.md` — exact native-window binding, explicit browser preparation,\n  typed Chromium\u002FElectron page tools, input trust classes, and native\n  fallbacks for browser chrome and unsupported engines.\n- `RECORDING.md` — session recording + `replay_trajectory`.\n\nUse whichever combination matches the host. When in doubt, run\n`cua-driver doctor` — it reports the platform and the right entry\npoint.\n\n## Start with the narrowest semantic route\n\nBefore opening or operating an application, name the desired postcondition and\nuse the first applicable route below. Verify the result in the same domain\nbefore stopping or advancing:\n\n0. **Caller-provided headless\u002Fbackground operation for a non-GUI outcome.**\n   Prefer an exact application API\u002FSDK, service or database client, CLI, or\n   filesystem operation over imitating a user. This includes batch-safe file\n   moves, renames, copies, directory creation, archive extraction, data\n   conversion, and process inspection. Read the resulting semantic state back;\n   a zero exit status alone is not proof.\n1. **Typed Cua operation for an application or window outcome.** Use\n   `set_window_frame` for exact geometry, `invoke_menu` for a known native\n   application-menu path, typed browser tools for supported page content, and\n   clipboard tools for clipboard state. Verify with\n   `list_windows`, `get_browser_state`, or `clipboard_read`, respectively.\n2. **Background accessibility action.** Use a fresh AX\u002FUIA\u002FAT-SPI target.\n3. **Background pixel action.** Use the pixels from the same state snapshot.\n4. **Foreground delivery.** Retry only the action that evidence says could not\n   land in the background.\n5. **Desktop fallback.** Select an exact desktop target for that call only.\n   Later calls may return to an exact window target in the same session.\n\nUse Cua Driver when the outcome lives in an application's UI or window state,\nor when the user explicitly asks to operate that GUI. Once the task crosses\nthat boundary, do not replace Cua's targeted and verified actions with shell\nscripts that mutate the app UI. A shell is a capability of the calling agent,\nnot of the Cua Driver MCP server; an MCP-only client must not assume one exists.\n\n### Filesystem outcomes and GUI fallbacks\n\nWhen the requested outcome is a filesystem change and the caller has a\nheadless filesystem or command capability, keep it on rung 0. Enumerate the\nexact source set, decide the destination-conflict policy before changing\nanything, perform one batch-safe operation, then independently read back both\nsource and destination manifests. Do not open a file manager merely to mimic a\nmove, copy, or rename that the caller can execute and verify directly.\n\nIf the caller has no such capability, use the file manager as a GUI fallback\nand keep each claim narrow:\n\n1. After entering an inline rename and setting its value, commit it with the\n   platform's confirmation key, then take a fresh snapshot. Value readback from\n   the inline editor proves only that the editor changed; it does not prove the\n   filesystem rename committed.\n2. For a multi-selection, use the platform modifier (`cmd` on macOS, `ctrl` on\n   Windows\u002FLinux). On macOS and Windows, issue that modified click with\n   `delivery_mode:\"foreground\"` so the target observes physical modifier state;\n   a refused background attempt is an escalation signal, not a failed action to\n   trust or repeat. Re-snapshot before the next operation. Continue only when\n   every intended item is selected and the prior selection was preserved.\n3. After a cross-window drag or paste, verify the destination contains the\n   complete expected set and the source reflects copy-versus-move semantics.\n   A delivered drag, keypress, or menu action is not file-operation proof.\n4. If a destination conflict presents an unrecognized policy or ambiguous\n   partial result, stop that GUI path and surface the unresolved state instead\n   of retrying blindly.\n\n### Clipboard outcomes and GUI fallbacks\n\nWhen the requested postcondition is an exact value on the system clipboard,\nrather than the literal gesture of selecting and copying it, keep the operation\nsemantic. Read the value from the narrowest typed source, call\n`clipboard_write`, then prove the real clipboard state with `clipboard_read`.\nFor browser content, this means reading the page with `get_browser_state` and\nwriting the exact observed text; a passive page-text ref does not need to be\nclicked first.\n\nUse visual selection followed by the platform copy hotkey only when the user\nexplicitly asks for that gesture, the source cannot expose the value\nsemantically, or direct clipboard tools are unavailable. Treat that as a GUI\nfallback: re-snapshot before acting, verify the selected range when the\napplication exposes it, and escalate only the delivery step that cannot land\nin the background.\n\n## The no-foreground principle (window phase)\n\nDuring window-targeted background actions, **the user's frontmost app MUST NOT\nchange.** Every platform\nhas its own list of forbidden commands:\n\n- macOS: any `open` invocation, any `osascript` that mutates GUI\n  state, `cliclick`, `cghidEventTap` writes targeting another app's\n  window. Full list in `MACOS.md`.\n- Windows: any `Start-Process` that triggers a `ShowWindow`\u002F`SetForegroundWindow`\n  on the target, `WScript.Shell.AppActivate`, attaching to the\n  foreground thread for input forwarding. Full list in `WINDOWS.md`.\n\nIf you reach for a command that says \"activate\", \"foreground\",\n\"raise\", or \"make key\", stop and translate to the cua-driver tool\nthat does the same intent without focus-stealing.\n\n`delivery_mode:\"foreground\"` is a user-visible takeover boundary, not an\nautomatic retry. Use it only when the user already authorized foreground\ncontrol for this workflow or after asking for approval. It may change focus,\nworkspace, and the compositor cursor while the action runs; restoration is\nbest-effort. If background delivery is unavailable and foreground control is\nnot authorized, stop with the driver's refusal instead of silently escalating.\n\nA desktop target is an explicit per-call choice to operate the visible desktop\nand therefore uses foreground\u002Fsystem input. Use it only after the narrower\nwindow ladder has been attempted and verified. Permission policy must still\nadmit the display resource. Never infer desktop permission from a failed action\nor a public session label.\n\n## GUI transport defaults — prefer cua-driver over GUI shell shims\n\n**Default transport is the `cua-driver` CLI for one-off calls** — `Bash`\nshelling out to `cua-driver \u003Ctool-name> '\u003CJSON-args>'`. Each CLI invocation\nowns a disposable transport session that is cleaned up after its response.\nUse one persistent `cua-driver mcp` connection for a multi-call GUI workflow\nthat needs shared cursor, recording, browser, or named-session state. A public\nsession label is not a credential and a later one-shot process cannot adopt\nthe previous process's lifecycle merely by repeating that label.\n\nCLI wins for isolated inspection and management because it picks up rebuilds\ninstantly, failures are easier to diagnose, and there's no per-tool\nschema-load overhead. Persistent MCP wins for an ordered action loop.\n\nEvery reference to `click(...)`, `get_window_state(...)` etc. in this\nskill means `cua-driver click '{...}'` — translate to MCP form only\nwhen MCP is requested.\n\n### Claude Code computer-use compatibility flag\n\nFor normal Claude Code use, keep the default CLI or `cua-driver` MCP\nserver path above. If the user explicitly wants Claude Code's\nvision\u002Fcomputer-use-style flow, they can register:\n\n```bash\ncua-driver mcp-config --client claude   # then paste + run the printed line\n```\n\nThe compatibility flag is retained for old setup snippets, but the standalone\n`screenshot` tool was removed. It does not add or replace tools. Use\n`get_window_state({pid, window_id})` for a window-local accessibility snapshot\nand PNG, or `get_desktop_state()` for an explicitly authorized desktop capture.\n\n## Using cua-driver from the shell\n\nTool names are `snake_case`, management subcommands are\n`kebab-case` — no ambiguity. Tools invoked as `cua-driver\n\u003Ctool-name> '\u003CJSON-args>'`. Management subcommands:\n\n- `cua-driver serve` — start an explicit persistent service when short-lived\n  clients must share runtime state or a platform identity. Bare MCP owns its\n  runtime directly on Windows\u002FLinux and uses the signed app service on macOS;\n  `cua-driver mcp --socket \u003Cendpoint>` selects a service explicitly.\n  One-shot CLI tool calls still use the service path. macOS users: see\n  `MACOS.md` for the LaunchServices-routed launch form.\n- `cua-driver stop` \u002F `status`\n- `cua-driver list-tools`, `describe \u003Ctool>`\n- `cua-driver recording start|stop|status` — see `RECORDING.md`\n- `cua-driver check-update [--json] [--no-cache]` — read-only \"is a newer release available?\" probe. Same payload as the `check_for_update` MCP tool; pair with `cua-driver update --apply` to install.\n\nCanonical multi-step workflow within one persistent MCP connection (example\nshape — platform-specific launch idioms in the per-OS companion file):\n\n```bash\n# Start the service once, then connect one MCP client with `cua-driver mcp`.\n# The calls below are tool calls on that same connection, not separate shell\n# invocations of `cua-driver \u003Ctool>`.\nlaunch_app({\"bundle_id\":\"...\"})\n# → {pid: 844, windows: [{window_id: 10725, ...}]}\nget_window_state({\"pid\":844,\"window_id\":10725})\n# Use the returned structuredContent.elements[].element_token:\nclick({\"pid\":844,\"element_token\":\"s0000002a:14\"})\nverify_state({\"pid\":844,\"window_id\":10725,\"expect\":[{\"element\":{\"selector\":{\"label_contains\":\"Saved\"},\"exists\":true}}]})\n```\n\nFor Chromium page content, keep the same native window selection but switch to\nthe browser capability loop: use one lifecycle session, bind `(pid, window_id)` with\n`get_browser_state`, snapshot the returned tab, then use `browser_click`,\n`browser_type`, or `browser_navigate`. Read `BROWSER.md` before using this\nroute. Browser target ids, tab ids, and refs are session-scoped and stale refs\nmust be replaced by a fresh snapshot.\n\n## Agent cursor overlay\n\nVisual cursor overlay for demos and screen recordings. It initializes on the\nfirst cursor-bearing action, including `move_cursor`, and follows the\ntransport's implicit or named lifecycle session. Toggle a named cursor with\n`set_agent_cursor_enabled` to hide or re-show it. The embedded\n`cua.default` theme uses a session-colored pointer over a larger,\ncursor-shaped glow in the same session color. The glow fades to transparent\naround the full silhouette. Action marks use the same\nsession-colored center and white-outline treatment, plus a tighter, softer\nglow. This pairing preserves contrast across varied backgrounds. It provides animations for\nidle, observe, click, drag, scroll, text, key, navigation, app, transfer,\nrecording, and system activity. Motion knobs:\n`set_agent_cursor_motion` takes any subset of `start_handle`,\n`end_handle`, `arc_size`, `arc_flow`, `spring` — tuneable at runtime,\npersisted to config.\n\n**Agent-control safety.** Keep this overlay enabled whenever the agent is\ncontrolling pointer or keyboard input. Cursor-bearing and keyboard actions\nautomatically re-show their session cursor, even if it was hidden while idle.\nOn Linux, ordinary `move_cursor({x,y})` moves only this synthetic cursor.\n`move_cursor({x,y,scope:\"desktop\"})` is the explicit escape hatch that moves\nthe user's compositor cursor and must not be used unless the user asked for\ndesktop-pointer control. Raw Wayland input that requires\n`delivery_mode:\"foreground\"` crosses the same user-visible takeover boundary.\n\nDelivery and target context is shown as host-owned chips inside the session\nbadge. Themes own the twelve action animations only. The session name and\ncontext chips fade independently, so an active tool can show its execution\ncontext without revealing a session name that has already faded.\n\n**Per-session cursors.** Each MCP session automatically owns its own\ncursor, keyed by the session's id (the proxy mints one session id per\nMCP connection and the daemon scopes the cursor, config overrides, and\nrecording to it). The CLI and SDK contracts take the declared `session`\nexplicitly. Cursor-theme controls no longer accept `cursor_id` or the legacy\nshape\u002Fcolor\u002Fimage fields. Input-delivery tools may still use `cursor_id` to\nname a virtual pointer; it never selects artwork. The default cursor is Cua\nblue, while each named session receives a stable fill from the built-in\npalette. Select only preinstalled\nthemes with `set_agent_cursor_theme`; theme source paths and inline animation\ndata are never accepted through an agent tool. Use the trusted local\n`cua-driver cursor-theme` workflow to validate, compile, preview, install,\nlist, or remove custom themes.\n\n**Visibility caveat (AX runs).** On a pure accessibility-action run\n(clicking by `element_index`), the first action **seeds the cursor\non-screen a short distance from the target and plays a brief glide +\npulse** — not the long Bezier sweep a cursor already on-screen would\ntrace from its previous spot. It's subtle and easy to miss in a\nrecording. If you want a clearly _gliding_ cursor for a demo or screen\nrecording, do a pixel click (`click({pid,x,y})`) or a `move_cursor`\nfirst to put the cursor on-screen; subsequent AX actions then glide the\nfull path normally.\n\nPixel `click` already glides the overlay. Do not call `move_cursor`\nimmediately before `click` on the same target; that plays two glides.\nUse `move_cursor` to place the overlay without clicking, or as the\none-time seed above before AX actions.\n\nRequires a suitable UI event loop. Service and private-worker runtimes provide\none. On macOS, a same-process SDK runtime or `cua-driver mcp --direct` without\na certified host main-thread adapter returns a structured\n`facility_unavailable` result for overlay operations; do not treat that as a\nsuccessful cursor move. One-shot CLI adapters do not own an overlay\nthemselves.\n\n## The core invariant — snapshot before and verify after every action\n\n**Every action MUST be bracketed by observation for the session's effective\nscope.** Use `get_window_state(pid, window_id)` before a window action (or\n`get_desktop_state(session)` in desktop scope), then use `verify_state` for an\nexpressible window-scoped postcondition. In effective desktop scope,\n`verify_state` is intentionally refused with `window_scope_disabled`; verify\nwith a fresh `get_desktop_state` result and agent-owned visual\u002Fsemantic reading.\n\n- **Before** — the pre-action snapshot resolves the `element_index`\n  you're about to use. Indices from previous turns are stale; the\n  server replaces the element index map on every snapshot, keyed\n  on `(pid, window_id)`. Indices from turn N don't resolve in turn\n  N+1, and indices from window A don't resolve against window B of\n  the same app. Skip this and element-indexed actions fail with\n  `No cached AX state`.\n- **After** — `verify_state(pid, window_id, expect)` checks a bounded,\n  deterministic postcondition. Results are `satisfied`, `unsatisfied`, or\n  `unknown`; `unknown` never means success. Set `include_screenshot:true` when\n  the outcome also needs visual reading. The driver returns that final image\n  without interpreting it. A multimodal agent harness reads the image and owns\n  the stop\u002Fretry\u002Fladder decision.\n\n`unknown_reason` distinguishes invalid\u002Funsupported predicates, untrusted web\ncontent, ambiguous matches, missing targets, unavailable observations, and\n`stability_unproven`. A positive final sample that was not observed for the\nrequested consecutive sample count is `stability_unproven`, not success.\nNegative element existence is conservative: when an accessibility projection\ncannot prove its search domain exhaustive, absence remains `unknown`.\n\nDo not make the driver invent task meaning or retry actions automatically.\nFor postconditions not expressible by `verify_state`, take a fresh state\nsnapshot and let the agent judge the tree and\u002For image explicitly. This applies\nto pixel clicks and desktop actions too.\n\n### Read action facts without confusing them with task success\n\nA successful action returns `effect` and `route`, with optional typed\n`delivery`, `evidence`, and `escalation`. These fields describe the actuator;\nthey do not declare the user's task complete.\n\n- `confirmed` means the driver has publishable value readback or window-change\n  evidence for that action.\n- `partial` means only `delivery.delivered_count` was delivered.\n- `unverifiable` means the driver cannot prove the effect.\n- `suspected_noop` means available evidence suggests no useful change.\n- `refused` means the selected route deliberately did not deliver.\n\nThe route vocabulary is intentionally cross-platform:\n`accessibility`, `synthetic_events`, `global_input`, `dom`, and\n`trusted_input`. Do not branch on private OS transport names.\n\nAn optional escalation is a harness instruction, never an automatic retry:\n\n- `pixel`: refresh visual state and choose an exact pixel target;\n- `foreground`: explicitly select foreground delivery if the authorization\n  stack admits the tool and exact target;\n- `page`: bind the native window to a supported browser page route;\n- `session`: a legacy compatibility signal from an older capture-scope daemon;\n  current callers choose a desktop target on the specific action instead.\n\nBranch on the closed reason vocabulary:\n`route_unavailable`, `delivery_failed`, `effect_unconfirmed`,\n`suspected_noop`, and `permission_required`.\n\nAfter any action, keep using `verify_state` or a fresh state snapshot for the\nactual task postcondition. The multimodal harness owns visual reading and the\ndecision to stop, retry, or advance the ladder.\n\n## Choose the target on each action\n\nA session owns lifecycle, cursor, recording, cleanup, and telemetry state. It\ndoes not store the current capture modality. Select an exact target on each\naction:\n\n```jsonc\n{\"target\":{\"kind\":\"window\",\"pid\":844,\"window_id\":10725}}\n{\"target\":{\"kind\":\"desktop\",\"display_id\":\"primary\"}}\n```\n\nThe window target uses window-local coordinates and the background\u002Fforeground\ndelivery ladder. The desktop target uses screen coordinates and foreground\ndelivery. A desktop action does not disable window tools for later calls.\n\n`start_session` is optional. For a multi-call run, prefer a short public\n`session` label and pass the same label on every call that accepts it. The label\nis call-scoped: if a later call omits it, that call uses the authenticated\ntransport's implicit session instead. Unnamed calls on one transport reuse that\nimplicit identity. The default idle TTL is five minutes. Call\n`start_session(session)` to name or configure a run before acting, or to revive\nan ended name.\n\nDo not use `config set capture_scope` or `set_config`; that key is retired and\nstale values on disk are ignored. `start_session.capture_scope`,\n`get_session_state`, and `escalate_session` are deprecated compatibility\nsurfaces. There is no `deescalate_session`. Reserved fields such as\n`_session_id` are transport metadata and cannot create authority.\n\n## Keep authorization separate from sessions\n\nThe trusted host selects one permission profile at startup. `standard` keeps\nthe normal profile behavior and residual approval requirements, `bounded`\nrequires a reviewed capability manifest and has no runtime approval path, and\n`unrestricted` bypasses Cua approval prompts after explicit risk acceptance.\nHard invariants plus managed and user policy remain binding in every profile.\n\nAn optional capability manifest is a deny-by-default ceiling in `standard` and\n`unrestricted`; `bounded` requires one. It can remove tools or typed resources\nfrom the selected profile, but it cannot grant a tool, resource, or approval\nbypass that another authorization layer denies. Approval is considered only\nafter the tool and every adapter-attested resource are inside manifest scope.\n\nUse the canonical startup pair together:\n\n```bash\ncua-driver mcp \\\n  --permission-mode standard \\\n  --capability-manifest .\u002Fcapabilities.yaml \\\n  --approve-capability-manifest\n```\n\nCapability manifest v3 omits file-level `mode` and `ask.tools`. Its\n`allow.tools` list is nonempty. Lifetime fields are optional in `standard` and\n`unrestricted`; `bounded` requires both `expires_after` and `idle_timeout`.\nThe older `--session-policy` names remain compatibility aliases and must not be\nused in new configurations.\n\nStarting, ending, naming, reconnecting, or omitting a session never changes\npermission authority. A public session label is lifecycle metadata, never a\ngrant, caller identity, or bearer credential.\n\n### Why window selection is the caller's job now\n\n`get_app_state` used to pick a window for you via a max-area heuristic\nthat returned the wrong surface on apps with large off-screen utility\npanels. Concrete reproducer: IINA's OpenSubtitles helper (600×432\noff-screen) out-area'd the visible 320×240 player window, so\n`get_app_state(pid)` screenshot'd the invisible panel and clicks landed\nthere silently. The new `get_window_state(pid, window_id)` makes the\ncaller name the window explicitly — the driver validates that the\nwindow belongs to the pid and is on the current Space\u002Fdesktop, then\nsnapshots exactly what was asked for. Enumerate candidates via\n`list_windows` or read the `windows` array `launch_app` already\nreturns.\n\n## Behavior matrix\n\n### Perception is mode-agnostic — `get_window_state` returns BOTH\n\n`get_window_state(pid, window_id)` **returns both the accessibility\ntree AND a screenshot by default.** There is no capture mode to pick\nand nothing to configure — you ground on the tree and the screenshot\ntogether, and you cross-check one against the other. This matters\nbecause the tree **lies** on some surfaces:\n\n- **Electron** echo-confirms a `set_value` \u002F `type_text` against the AX\n  shim while the rendered text view never changed.\n- **Catalyst** (iOSAppOnMac) exposes null \u002F placeholder `AXValue`s.\n- **Virtualized \u002F off-viewport list rows** report bogus frames (an\n  `h:1` height, an off-screen origin) for rows that aren't actually\n  laid out.\n\nA grounding screenshot is present by default, so when the tree looks\nwrong you look at the pixels **in the same response** — no second\ncapture, no mode flip.\n\n> **Perf opt-out — `include_screenshot`.** `include_screenshot`\n> (boolean, default `true`) is the one knob, and it is a **perf** knob,\n> not a modality choice. Default returns both (grounding-first). Pass\n> `include_screenshot:false` to skip the screen grab and get the tree\n> only — the cheap path when you're just **re-indexing before an\n> element ax action** and don't need to re-ground on pixels. The\n> `ax`\u002F`px` decision still lives at action time, not here.\n\n> **`capture_mode` is DEPRECATED and ignored.** It is still _accepted_\n> on `get_window_state` so old callers don't error, but it has **no\n> effect** — both the tree and the screenshot come back regardless of\n> what you pass (`ax`, `vision`, `som`, anything). There is no\n> `ax`\u002F`vision`\u002F`som` capture choice anymore. Drop the word \"vision\"\n> for perception entirely. (The tool named `screenshot` is separate —\n> raw PNG, no AX walk — and unrelated.)\n\n### The modality is chosen at ACTION time — `ax` vs `px`\n\nYou don't pick a capture mode; you pick **how you address the target**\non the action call, and that one choice selects the rung:\n\n- **element ax action** — pass `element_token` (preferred), or the exact\n  `element_index` + `snapshot_id` pair from the same response.\n  Dispatches through the **accessibility rung**: AXPress (macOS) \u002F UIA\n  Invoke (Windows) \u002F AT-SPI `doAction` (Linux). Backgroundable,\n  z-order-independent, and the only **driver-verifiable** rung.\n- **element px action** — pass `x`, `y`. Dispatches through the **pixel\n  rung**, reading the coordinate straight off the screenshot that's\n  already in the `get_window_state` response. Best-effort; the caller\n  confirms the effect.\n\n`ax`↔`element_index`, `px`↔pixel `x,y`. We retired the word \"vision\"\nfor the _dispatch_ path — it conflated perception with dispatch.\nPerception is always both; dispatch is `ax` or `px`.\n\n**The keyboard family has both forms too.** `type_text`, `press_key`,\nand `hotkey` take a snapshot-bound element target (ax) **or** `x,y` (px) — mutually\nexclusive, same as the pointer tools. The px form **pixel-clicks at\n`(x,y)` to establish real renderer focus, then delivers the\nkeystroke(s)** to the now-focused element (it reuses `click`'s\ncoordinate translation + `delivery_mode`). That gives e.g.\n`type_text({pid, window_id, x, y, text})` as a one-call focus-then-type\nfor Chromium\u002FElectron inputs the AX path can't reach, and\n`hotkey({pid, x, y, keys:[\"cmd\",\"v\"]})` to paste into a specific field.\n\n**Typing default (the ladder).** Call `type_text` directly with\n`element_token` (ax) — it targets the field, no pre-click. On\nElectron\u002FCatalyst the AX layer echoes the write without rendering it,\nso the driver returns `effect:\"unverifiable\"` with\n`escalation.target:\"pixel\"` there (never a false `effect:\"confirmed\"`) —\nfollow it, and cross-check the\nscreenshot in the response (the only ground truth). Escalate to the px\nform — `type_text({pid, window_id, x, y, text})` — which pixel-clicks\nto focus, then types. **If the target control is closed** (a search\nbutton, a collapsed field), AX-press to open it first (AX actions work\nin the background): a px focus-click won't reliably open _and_ focus a\nclosed control, so the text leaks into whatever's already focused.\nEscalate to `delivery_mode:\"foreground\"` only if it still drops.\n\n**`set_value` stays AX-only by design** — use it when the intent is to\nreplace a control's whole value: dropdowns, checkboxes, sliders, steppers,\nand native text fields such as Finder's inline rename editor. Use\n`type_text` when the intent is to insert text at the current selection or\ncursor. Its pixel counterpart is a `click`\u002F`drag` on the control, not a\n\"set value at a pixel.\" So: insert text → `type_text` (ax+px); replace a\nsurfaced native value → `set_value`; pixel-manipulate a control →\n`click`\u002F`drag`.\n\n**Action responses carry closed action facts**\n\nUse the `effect`, `route`, optional `delivery`, `evidence`, and\n`escalation` rules in “Read action facts without confusing them with task\nsuccess” above. The old `verified`, `path`, coordinates, scope, and\n`escalation.recommended` response fields no longer exist.\nThe full wire contract and 0.14 migration notes are in\n`..\u002F..\u002F..\u002Fdocs\u002Faction-result-contract.md`.\n\nA successful accessibility value write can still return\n`effect:\"unverifiable\"` when the provider publishes its new value only after\nthe action call unwinds. Take a fresh snapshot before retrying; an immediate\nretry can duplicate text. An explicit pixel escalation is reserved for a web\nsurface whose accessibility layer echoed the write without proving that the\nrenderer observed it.\n\n`get_window_state` itself, when the AX tree comes back empty (a non-AX\nsurface like Electron\u002FChromium\u002Fcanvas), returns `degraded: true`\nplus an observation-specific escalation hint — normally pointing at pixels (you\nstill have the screenshot from the same call to click off).\n\n**Platform nuance for action escalation.** On **Wayland** an unfocused\nwindow cannot be pixel-targeted in the background (libei →\n`background_unavailable`), so the action target is\n**`foreground`, not `pixel`**. macOS, X11, and most Windows surfaces\ncan pixel-target in the background, so they target `pixel`. See\n`LINUX.md` \u002F `WINDOWS.md`.\n\n## The verify-then-escalate ladder (algorithm)\n\nEvery snapshot already hands you both the tree and the screenshot, so\nverifying never means \"go take a screenshot\" — it means cross-check\nthe tree against the pixels you already have, and only change\n_dispatch rung_ on a real signal. Walk the rungs:\n\n```\n# Routes 0–1 — resolve non-GUI, exact geometry, and supported page outcomes first\n# Use a caller-provided semantic operation for a non-GUI outcome, then read it back.\n# For exact window geometry: set_window_frame(...), then list_windows(...) readback.\n# For a known native menu command: invoke_menu(pid, window_id, path), then verify its effect.\n# For supported page content: get_browser_state(...), typed browser action, refresh refs.\n# Continue below only when the postcondition actually requires native UI interaction.\n\n# Route 2 — element AX\u002FUIA\u002FAT-SPI action, backgrounded\nget_window_state(pid, window_id)            # tree + screenshot, both, always\nresp = click(pid, element_token)            # or type_text \u002F set_value \u002F press_key\ncheck = verify_state(                       # bounded structured read-back\n    pid, window_id,\n    expect=[...],\n    include_screenshot=true                 # optional evidence for multimodal harness\n)\n\nif check.status == \"satisfied\":\n    done                                    # driver-verified\n\nif check.status == \"unknown\" and check has an image:\n    harness reads the image                  # model-owned visual interpretation\n    if visual outcome is satisfied: done\n\n# escalate only on a real signal\nif resp.effect == \"suspected_noop\"\n   or resp.escalation.target == \"pixel\"\n   or get_window_state.degraded            # empty tree → non-AX surface\n   or check.status != \"satisfied\"\n   or the tree looks wrong vs the screenshot:   # e.g. an h:1 \u002F off-viewport row\n\n    # Route 3 — element px action off the SAME screenshot\n    pick the target pixel from the screenshot already in the response\n    click(pid, x, y)                        # background pixel — still no foreground\n    verify_state(..., include_screenshot=true)\n    if it landed: done\n\n# Route 4 — background delivery was dropped (insert\u002Fclick never arrived)\nif resp.escalation.target == \"foreground\"\n   or the px action still did nothing:\n    require existing user authorization for visible foreground control\n    otherwise stop and ask; do not retry automatically\n    re-call the same action with delivery_mode:\"foreground\"\n    # on Wayland this is the ONLY escalation — px-bg can't target an\n    # unfocused window there; see LINUX.md\n    verify again\n\n# Route 5 — per-call desktop fallback\n# Reach this only after semantic, AX, window-pixel, and foreground-window\n# delivery have all been exhausted and verified ineffective.\nget_desktop_state()                         # full primary display\ndesktop_action(target={kind:\"desktop\", display_id:\"primary\"}, ...)\nget_desktop_state()                         # verify in the same coordinate frame\n```\n\nThe two ideas to hold onto: (1) the AX tree **lies** on canvas \u002F web \u002F\nCatalyst \u002F virtualized surfaces, so an unchanged-or-bogus tree plus\n`suspected_noop`\u002F`degraded` — or a tree that simply disagrees with the\nscreenshot — is your cue to do an **element px action** off the\nscreenshot you already have; (2) `px` is a _conscious_ switch to the\npixel addressing path, not a different capture.\n\n**Window state → what works**\n\n| state                      | `get_window_state`                                                                             | element-index click (AX\u002FUIA) | `press_key` commit                                    | pixel click                    |\n| -------------------------- | ---------------------------------------------------------------------------------------------- | ---------------------------- | ----------------------------------------------------- | ------------------------------ |\n| frontmost                  | ✅                                                                                             | ✅                           | ✅                                                    | ✅                             |\n| backgrounded \u002F visible     | ✅                                                                                             | ✅                           | ✅                                                    | ✅                             |\n| **minimized**              | ✅                                                                                             | ✅ (actions fire in place)   | ❌ silent no-op — use `set_value` or click equivalent | ❌ no on-screen bounds         |\n| hidden                     | ✅                                                                                             | ✅                           | depends                                               | ❌                             |\n| on another desktop \u002F Space | ⚠️ tree may be stripped on some apps — response carries `off_space: true` so you can detect it | ✅                           | ✅                                                    | ❌ not in current-desktop list |\n\n**Critical cell — minimized + keyboard commit.** The keystroke\nreaches the app but accessibility focus doesn't propagate to renderer\nfocus on a minimized window. Workarounds in order of preference:\n`set_value` to write the field's entire value directly, or\nelement-index-click a commit-equivalent button (Go, Submit,\ncheckbox). Tell the user the window needs to un-minimize only as a\nlast resort.\n\n## The canonical loop\n\n```\n# for multi-call work, repeat the same session label on every call that accepts it\nlaunch_app(target, session)\n  → pick window_id from the returned `windows` array\n    (or call list_windows(pid) separately)\n  → get_window_state(pid, window_id)\n    → [act]  # pass target={kind:\"window\", pid, window_id}\n  → verify_state(pid, window_id, expect)  # structured check; optional image\nend_session(session?)             # optional explicit cleanup\n```\n\nAn existing-profile `end_session` restores the Chromium browser's\nremote-debugging toggle when Cua enabled it and no other Cua session still uses\nthat browser process and dismisses the exact native consent prompt. If exact\ncleanup cannot be proven, session cleanup fails closed and a later\n`end_session` retries it.\n\nFor screen-absolute work, replace the window portion with\n`get_desktop_state() → action(target={kind:\"desktop\",display_id:\"primary\"}, ...)\n→ get_desktop_state()`. Desktop actions use coordinates from that exact\nfull-display image.\n\n`launch_app` now returns a `windows` array alongside the pid, so the\ncommon case collapses to two calls (`launch_app` → `get_window_state`)\nwithout a separate `list_windows` hop.\n\n**Prefer a named session for multi-call work.** Choose a short label (for\nexample, `session: \"research-1\"`) and pass the same value on every call that\naccepts it. Passing it once is not sticky: a later call that omits `session`\nuses the transport's implicit session. Call `start_session(session)` when you\nneed to name or configure the run before acting, or to revive a name after\n`end_session`. For one-off or deliberately unlabeled work, omission is valid\nand the transport still gets one private lifecycle identity and visible agent\ncursor. A public label makes inspection and cleanup easier, but it is not a\ncredential. End with `end_session` when useful; transport close or the\nfive-minute idle TTL also reclaims it.\n\n**Concurrent runs\u002Fsubagents:** each transport gets its own implicit session.\nAlso,\n`launch_app` is idempotent — two runs that\nlaunch the same app get the **same** instance (and on single-instance apps\nlike Calculator, the same window), so they clobber each other. Give each run\nits **own connection** (for independent lifecycle\u002Fcursor ownership) AND pass\n`creates_new_application_instance: true` to `launch_app` (→ its own window).\nThe element cache is keyed on `(pid, window_id)` and the cursor on the private\nlifecycle session, so distinct instances and transports keep the runs isolated.\n\n**Parallelism vs. ordering.** Distinct sessions give distinct _cursors_, not\ndistinct _connections_. Subagents that share one `cua-driver mcp` (stdio)\nconnection have their tool calls **serialized** by the transport — they take\nturns, not run in parallel. That's not a correctness problem (session + window\nisolation means they can't collide), just a throughput one. For genuinely\nparallel agents, give each its **own connection**: separate `cua-driver mcp`\nprocesses, or point each agent's MCP client at the daemon's HTTP endpoint.\nSet `CUA_DRIVER_RS_MCP_HTTP_PORT` and a host-generated\n`CUA_DRIVER_RS_MCP_HTTP_TOKEN` of at least 32 characters, then send\n`Authorization: Bearer \u003Ctoken>` to `POST http:\u002F\u002F127.0.0.1:\u003Cport>\u002Fmcp`. The daemon\nserves connections concurrently; per-connection ordering keeps each agent's own\nsequence (e.g. `3 → + → 1 → =`) correct.\n\n`list_apps` is for app-level discovery (answering \"what's installed \u002F\nrunning \u002F frontmost?\") — not part of the core action loop. Skip it\nin the loop. For **window-level** questions — \"does this app have a\nvisible window?\", \"which desktop is this window on?\", \"which of this\npid's windows is the main one?\" — call `list_windows` instead; the\napp record doesn't carry window state on purpose. In the common\nsingle-window case you can skip `list_windows` entirely and read the\n`windows` array that `launch_app` already returned.\n\n### Snapshot and act with a snapshot-bound target\n\nCall `get_window_state({pid, window_id})` with the `window_id` from\n`launch_app`'s `windows` array (or a fresh `list_windows({pid})` if\nyou're interacting with a long-lived process). It returns **the tree\nand the screenshot together** by default, so you can both dispatch by\n`element_token` and ground on pixels from one call — no config change,\nno mode flip. When you're just re-indexing before an element ax action\nand don't need fresh pixels, pass `include_screenshot:false` to skip\nthe grab (a perf knob, not a modality choice).\n\nThe response carries:\n\n- `tree_markdown` — every actionable element tagged `[N]`; the structured row\n  with the same `element_index` carries its opaque `element_token`. The tree can be very large (Finder is\n  ~1600 elements, ~190 KB); when it exceeds token limits the MCP\n  harness saves it to a file and returns the path. Use `Bash` +\n  `jq -r '.tree_markdown'` + `grep` to pull the section you need.\n- `effect` \u002F `escalation` \u002F `degraded` — the verify-then-escalate\n  signals (see the behavior matrix above): `degraded: true` means the\n  tree came back empty (non-AX surface), so you act by **`px`** off the\n  screenshot in the same response.\n- `screenshot_file_path` — present when the screenshot was written to\n  disk instead of inlined (you passed `screenshot_out_file`, or the\n  context-saving CLI path); otherwise the frame is inlined.\n- `screenshot_width` \u002F `_height` \u002F `_scale_factor` — dimensions of\n  the captured image. Present whenever a screenshot was taken (i.e.\n  unless you passed `include_screenshot:false`).\n\n**Getting the screenshot as a file (CLI and context-constrained agents):**\n\n```bash\n# write to file — stdout stays readable (AX\u002FUIA tree \u002F summary only, no base64)\ncua-driver get_window_state '{\"pid\":N,\"window_id\":W,\"screenshot_out_file\":\"\u002Ftmp\u002Fshot.png\"}'\n\n# CLI --screenshot-out-file flag is equivalent\ncua-driver get_window_state '{\"pid\":N,\"window_id\":W}' --screenshot-out-file \u002Ftmp\u002Fshot.png\n```\n\nPass `screenshot_out_file` when using `get_window_state` via CLI or\nfrom an agent whose context window can't absorb ~31 KB of inline\nbase64 (e.g. OpenCode with a local Ollama model). The MCP image\ncontent block is omitted from the response when this param is set —\nthe model receives only the tree and `screenshot_file_path`, then\nreads the image from disk.\n\n**The tree and the screenshot are complementary, not redundant — and\nthey come from the _same_ call.** Each half carries signal the other\ncan't, which is exactly why you cross-check them:\n\n- The **tree** tells you _what's clickable_ — roles, labels,\n  snapshot-bound element handles, advertised actions, parent-child\n  structure. This is the ground truth for an **element ax action**.\n- The **screenshot** tells you _which one_ — the tree often has many\n  buttons with similar or empty labels (\"Delete\", \"OK\", anonymous\n  UUID-labeled buttons, repeated static-text), and visual context\n  disambiguates. Captions, colors, layout relationships visible in\n  pixels often don't show up in the tree at all (especially in\n  Chromium \u002F Electron \u002F web content) — and the screenshot is where you\n  catch the tree _lying_ (an `h:1`\u002Foff-viewport row, a Catalyst null\n  value).\n\nDefault to dispatching by `element_token` (the **element ax action**) —\nit's the verifiable, backgroundable rung. Do an **element px action**\n(`x,y` off the same screenshot) when the tree can't disambiguate\n(repeated\u002Fempty labels), when it's empty (`degraded` — non-AX\nsurface), when an action came back `suspected_noop`, or when the tree\ndisagrees with the pixels. You never re-capture to switch — the\nscreenshot is already there; you just change _how you address_ the\ntarget.\n\nReach for pixel coordinates only when the target is a canvas \u002F\nvideo \u002F WebGL \u002F custom-drawn surface that isn't in the tree\n(see \"Pixel-coordinate clicks\" below).\n\nThe `actions=[...]` list on each element is **advisory**, not\nauthoritative. cua-driver does not pre-flight check against it —\n`click({pid, element_token})` always attempts the default action (or\nthe action you pass) and surfaces whatever the target returns. **Try\nthe click first** — pivot only on the returned error code.\n\n### Tool dispatch table\n\nEvery row assumes a fresh `get_window_state`. Prefer its opaque\n`element_token`. If a client uses the visible integer instead, it must send\nthe response's `snapshot_id` with `element_index`; bare indices fail closed in\n0.17. Pixel-only forms remain independent of snapshot handles.\n\n| Intent                           | Tool                                                                                                            | Notes                                                                                                                                                                                                                 |\n| -------------------------------- | --------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| List an app's windows            | `list_windows({pid})`                                                                                           | returns `window_id`, `title`, `bounds`, `z_index`, `is_on_screen`, `on_current_space`. Already included in `launch_app`'s response — only call this for long-lived pids                                               |\n| Set an exact window frame        | `set_window_frame({pid, window_id, x, y, width, height})`                                                       | uses the platform window manager and returns `confirmed` only after geometry readback; inspect `list_windows` again before continuing when the result is not confirmed                                                |\n| Invoke a native application menu | `invoke_menu({pid, window_id, path:[\"Window\",\"Arrange\",\"Left\"]})`                                              | resolves exact immediate-child labels from live native state at every hop; refuses missing, ambiguous, or disabled segments and never falls back to pixels; verify the command's semantic effect afterward          |\n| Snapshot a window                | `get_window_state({pid, window_id})`                                                                            | returns `tree_markdown` + `screenshot_*`; populates the `(pid, window_id)` element_index cache                                                                                                                        |\n| Verify a postcondition           | `verify_state({pid, window_id, expect, include_screenshot?})`                                                   | polls bounded structured predicates; returns `satisfied`, `unsatisfied`, or `unknown`. Optional final image is interpreted by the agent harness, never by the driver                                                 |\n| Left click                       | `click({pid, element_token})` or `click({pid, window_id, element_index, snapshot_id})`                          | default `action: \"press\"`. Pixel form: `click({pid, x, y})` (window_id optional) — `modifier: [\"cmd\"\\|\"ctrl\"]`                                                                                                        |\n| Double-click \u002F open              | `double_click({pid, element_token})`                                                                            | Default action when the element advertises one (Open on Finder items \u002F openable rows), else stamped pixel double-click at the element's center                                                                        |\n| Right click \u002F context menu       | `right_click({pid, element_token})` or `click({pid, element_token, action:\"show_menu\"})`                       | Browser page content should use the typed route where available; see `BROWSER.md`                                                                                                                                     |\n| Type at cursor                   | `type_text({pid, text, element_token})` (ax) or `type_text({pid, text, window_id, x, y})` (px)                  | ax focuses the element then writes via the platform's text-set primitive; **px** pixel-clicks `(x,y)` to focus the renderer, then types — the one-call fix for Chromium\u002FElectron inputs the AX path can't reach       |\n| Set whole non-text control value | `set_value({pid, element_token, value})`                                                                         | **AX-only by design** — dropdown\u002F`AXPopUpButton`, checkbox, slider, stepper; **also the keyboard-commit workaround on minimized windows.** For text use `type_text`; to pixel-manipulate a control use `click`\u002F`drag` |\n| Scroll                           | `scroll({pid, direction, amount, by, element_token})`                                                           | synthesizes per-pid PageUp\u002FPageDown\u002Farrows                                                                                                                                                                            |\n| Focus + send key                 | `press_key({pid, key, element_token, modifiers})` (ax) or `press_key({pid, key, x, y})` (px)                    | ax targets the element before posting the key; **px** pixel-clicks `(x,y)` to focus, then sends the key                                                                                                               |\n| Send key to pid                  | `press_key({pid, key, modifiers})`                                                                              | no focus change; key goes to pid's current focus                                                                                                                                                                      |\n| Modifier combo                   | `hotkey({pid, keys})` (no focus) or `hotkey({pid, x, y, keys})` (px)                                            | e.g. `[\"cmd\",\"c\"]` \u002F `[\"ctrl\",\"c\"]`; posted per-pid, not HID tap. **px** pixel-clicks `(x,y)` to focus a field first, e.g. `[\"cmd\",\"v\"]` to paste into it                                                             |\n\n`list_windows.z_index` uses one portable convention: higher integer\nvalues are closer to the front. Select a frontmost candidate with the\nmaximum non-null value. If all values are `null` (as they can be on\nnative Wayland), use an explicit fallback; never treat `null` as zero\nor infer stacking from array order. The `windows` records returned by\n`launch_app` use the same convention.\n\nIn effective desktop scope, the foreground\u002Fsystem equivalents omit\n`pid`\u002F`window_id` and pass `scope:\"desktop\"`: `click`, `scroll`, `drag`,\n`move_cursor`, `type_text`, `press_key`, and `hotkey`. Coordinates are\nscreen-absolute pixels from the latest `get_desktop_state` image.\n\n**Window-scope keyboard\u002Ftext primitives require `pid`.** They use the named\ntarget's per-pid event-post path. Only a strict\u002Feffective desktop session may\nomit `pid`, and it intentionally routes keyboard input to the current\nforeground application.\n\n**Why the snapshot-bound element target is the primary path:** works on hidden \u002F\noccluded \u002F off-desktop windows, avoids focus steal, and fails closed after a\ntree rebuild instead of silently retargeting a reused index. Labels tell you\nwhat you're clicking. Reach for pixel\ncoordinates only when the accessibility tree can't.\n\n## Cross-platform parameter contract\n\nThe capture, dispatch, and addressing params — `session`,\n`delivery_mode`, `capture_mode` (deprecated\u002Fignored — see the behavior\nmatrix; still in the schema only so old callers don't error), `scope`,\n`modifier`, `button`, `element_index`, `snapshot_id`, `element_token` — are a **shared\nschema contract**: identical _shape_ (`type`\u002F`enum`\u002F`items`) on macOS,\nWindows, and Linux.\nThey compose from canonical fragments in\n`cua-driver-core::tool_schema` (+ `capture_mode`), and a CI gate\n(`schema_consistency_test`) runs every tool's live `tools\u002Flist` through a\nstructural checker on each platform, so the three surfaces can't\nsilently drift. _Contributor note:_ when you add or edit one of these\nshared params on a tool, pull from the fragment — don't re-hand-write the\nJSON, or the gate fails. (Descriptions may legitimately vary per tool;\nthe gate compares shape, not prose.)\n\nTwo consequences for callers:\n\n- **`session` is accepted on every action and cursor tool, on all three\n  platforms.** It's cursor-wired where the platform glides a cursor and\n  schema-accepted everywhere else — so the same `session` you pass on\n  macOS is no longer _rejected_ by Windows\u002FLinux, which previously\n  refused unknown keys via `additionalProperties:false`.\n- **`delivery_mode` (`\"background\"` default \u002F `\"foreground\"`) is on the\n  whole input family** — `click`, `double_click`, `right_click`, `drag`,\n  `scroll`, `type_text`, `press_key`, `hotkey` — uniformly. The\n  `foreground` rung briefly fronts the target, acts, then restores the\n  prior frontmost: the explicit last resort when a background attempt\n  didn't land. **`foreground` is a reaction and a user-authorization gate,\n  never a prediction.** Always\n  fire the `background` default first and let the driver tell you it\n  can't (a `background_unavailable` error with\n  `escalation.recommended == \"foreground\"`, or a successful action result\n  with `escalation.target == \"foreground\"`) — or observe a confirmed no-op —\n  _before_ you consider escalation. Then use it only when the user already\n  authorized visible foreground control or after asking for approval.\n  Do **not** reason \"it's a GTK\u002FChromium\u002FElectron app, so background will\n  drop, so I'll front up-front\": the toolkit lists in the tool schemas\n  are the _driver's_ internal detectors, not a checklist for you to front\n  on a guess. (Concretely: GIMP's GTK toolbox accepts background pixel\n  clicks fine — a preemptive foreground click there just steals the\n  user's focus for nothing.) What each platform's _background_ rung can\n  actually carry differs (e.g. a Windows background click can't carry\n  `modifier` state — see `WINDOWS.md`); the schema is uniform, the\n  residual limits are per-OS.\n\n**Required-set contract.** `click` requires nothing (`required:[]`),\n`scroll` requires `[\"direction\"]`, `zoom` requires\n`[\"window_id\",\"x1\",\"y1\",\"x2\",\"y2\"]` — same on every platform. `pid` is\n**conditionally** required (needed unless a windowless desktop-scope\ncall) and validated in code with a clear error, NOT pinned in the schema\n— so omitting `pid` for a desktop-scope action is no longer\nschema-rejected.\n\nGenuinely platform-specific params stay OUT of the shared contract by\ndesign (launch-app identifiers, the Windows-only `debug_window_info`, the\nmacOS-only status-only `check_permissions.prompt`). The per-OS files list the\nresiduals that matter when you drive on that platform.\n\n## Pixel-coordinate clicks\n\nThe pixel path (`click({pid, x, y})`) is for surfaces the\naccessibility tree doesn't reach — canvases, video players, WebGL,\ncustom-drawn controls. Coords are **window-local screenshot pixels**\n(same space as the PNG `get_window_state` returns). Top-left origin,\ny-down. The driver handles screen-point conversion internally.\nPassing `window_id` alongside `x, y` is optional but recommended —\nit pins the coordinate conversion to the window whose screenshot\nproduced the pixel.\n\nPNGs returned by `get_window_state` are capped at **1568 px long-side\nby default** (`max_image_dimension` config), matching Anthropic's\nmultimodal-vision downsampling limit. The image the model reasons\nover and the image the click tool's coordinate system lives in are\nthe **same resolution** — just look at the PNG, pick a pixel, click\nat that pixel. No scaling math.\n\nThis is the default because the mismatch between \"rendered\nthumbnail\" and \"native PNG\" was a recurring coord-estimation\nfootgun. If you opt out (explicit `max_image_dimension=0` for\npixel-perfect verification flows), the old rule applies: don't\neyeball coords from whatever your client renders — it may be\n2-4× smaller than the PNG on disk, and a 2% error in thumbnail\nspace becomes ~80 px in the real image.\n\nFor precise targeting on small \u002F dense UIs:\n\n1. `get_window_state({pid, window_id})` → image capped at 1568\n   long-side plus `screenshot_width` \u002F `screenshot_height`. Write to\n   disk via `--screenshot-out-file \u003Cpath>`.\n2. Look at the PNG. Since it matches what you see, pick the target\n   pixel directly.\n3. When precision matters, draw a crosshair on the image (do\n   **not** crop — cropping loses the coordinate system) and verify\n   before clicking:\n\n```python\nfrom PIL import Image, ImageDraw\nimg = Image.open('\u002Ftmp\u002Fshot.png')\ndraw = ImageDraw.Draw(img)\nx, y = \u003Cyour_coordinate>\nr = 18\ndraw.ellipse([x-r, y-r, x+r, y+r], outline='red', width=4)\ndraw.line([x-30, y, x+30, y], fill='red', width=3)\ndraw.line([x, y-30, x, y+30], fill='red', width=3)\nimg.save('\u002Ftmp\u002Fshot_annotated.png')\n```\n\n4. Only dispatch the click after the user (or your own re-read of\n   the annotated image) confirms the crosshair is on target.\n\nAddressing variants:\n\n- `click({pid, x, y})` — single left-click.\n- `click({pid, x, y, count: 2})` — double-click.\n- `click({pid, x, y, modifier: [\"cmd\"\\|\"ctrl\"]})` — modifier click.\n  Accepts any subset of `cmd\u002Fshift\u002Foption\u002Falt\u002Fctrl`.\n- `right_click({pid, x, y})` — also takes `modifier`.\n\nThe pixel path animates the agent cursor overlay but never warps\nthe real cursor (the per-pid event paths the driver uses on macOS\nand Windows route around HID synthesis). If the pid has no on-screen\nwindow the call errors with `pid X has no on-screen window` — you\nneed a visible window to anchor the conversion. Dispatch details\n(SkyLight on macOS, layered UIA+PostMessage on Windows) are in the\nper-OS companion files.\n\n## Web-rendered apps (browsers, Electron, Tauri)\n\nFor Chromium-family browsers and Electron, use the exact, session-scoped\nbrowser capability workflow in **`BROWSER.md`**. It keeps native\n`(pid, window_id)` selection as the entry point, makes setup explicit through\n`browser_prepare`, and distinguishes trusted browser input from an explicitly\nrequested synthetic DOM event.\n\nUse the native `get_window_state` and AX\u002FPX action ladder for browser chrome,\npermission prompts, downloads, file pickers, Safari, Firefox, Tauri, and any\nembedded webview for which exact browser binding is unavailable. The legacy\n`page` tool remains a compatibility surface; do not use it as the starting\npoint for new browser workflows.\n\n## Verify after every action — mandatory\n\n**Always** verify after an action. Prefer\n`verify_state({pid, window_id, expect})` for structured state such as a\nwindow's existence\u002Fbounds or a semantic element's existence, value, enabled\nstate, or selected state. Use its bounded poll and stable-sample requirement\ninstead of hand-written sleeps. `unknown` means the driver could not establish\nthe predicate; it is not success. Once a session has effective desktop scope,\nuse a fresh `get_desktop_state(session)` result instead—window-scoped\n`verify_state` is denied by that capture policy.\n\nPass `include_screenshot:true` when visual evidence is useful. The same result\nthen contains a fresh final window image. The driver still evaluates only the\nstructured predicates; the multimodal agent harness reads the pixels and\ndecides whether to stop, retry, or advance the ladder. For a postcondition not\nexpressible by the tool, explicitly take a fresh `get_window_state` snapshot\nand have the harness judge its tree and image.\n\nSwitch to an **element px action** only on a real signal: the action\nresponse carried `effect:\"suspected_noop\"`, verification returned\n`unsatisfied`\u002F`unknown`, the snapshot came back `degraded` (empty tree →\nnon-AX surface), the tree looks unchanged\u002Funreadable or disagrees with the screenshot, or\n`escalation.target` points you there (`pixel`). That's the\nverify-then-escalate ladder in the behavior-matrix section. If the tree\nis unchanged AND the screenshot confirms nothing moved, the action\nlikely failed silently — **tell the user what you attempted and what\nyou observed**, don't paper over with \"done\" language (and consider\n`delivery_mode:\"foreground\"` when `escalation.target ==\n\"foreground\"`). Agents that skip this step report success on\nsilently-dropped actions — the single most common failure mode.\n\n## Recording trajectories\n\nSession-scoped action recording + replay, for demos, regressions,\nand training data. Only invoke when the user explicitly asks to\nrecord a session — the skill does not auto-enable this. CLI surface:\n`cua-driver recording start|stop|status`; raw tools:\n`start_recording` \u002F `stop_recording`. Video capture (main display →\n`recording.mp4`) is on by default; pass `record_video: false` to opt out.\n\nSee **`RECORDING.md`** for the full flow: enable\u002Fdisable, turn folder\ncontents, replay via `replay_trajectory`, and the element_index\ndoesn't-survive-across-sessions caveat.\n\n## Common error patterns (cross-platform)\n\n| Error text                                                                         | Meaning                                                                                                                                                                          | Fix                                                                                                                                                                                                                  |\n| ---------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |\n| `No cached AX state for pid X window_id W`                                         | You either skipped `get_window_state` this turn, or passed a different `window_id` to the click than the one the snapshot cached against                                         | Call `get_window_state({pid: X, window_id: W})` first — the same window_id you intend to click in                                                                                                                    |\n| `snapshot_id_required` \u002F `stale_element_token`                                    | A bare index was supplied, or a newer snapshot superseded this target                                                                                                            | Re-run `get_window_state`; use the new `element_token`, or send its `snapshot_id` with the matching integer                                                                                                          |\n| `window_id W belongs to pid P, not …`                                              | Passed a window_id that's owned by a different process                                                                                                                           | Use `list_windows({pid: X})` to enumerate this pid's own windows                                                                                                                                                     |\n| `ambiguous_window_target`                                                          | A PID-only window action matched multiple eligible top-level windows                                                                                                              | Use the returned candidates or `list_windows({pid: X})`, select the intended sibling, and retry with its explicit `window_id`                                                                                       |\n| `AX action … failed with code …` \u002F `UIA invoke failed`                             | Element doesn't support the default action                                                                                                                                       | Try `show_menu`, `confirm`, `cancel`, `pick`, or fall through to a pixel click on the element's center                                                                                                               |\n| `The user doesn't want to proceed with this tool use. The tool use was rejected …` | The harness uses this _exact_ string for BOTH a permission-prompt denial AND a manual interrupt (Esc \u002F stop) of a running tool — they are indistinguishable from the tool result | Treat as \"tool canceled, no result, await the user.\" Do NOT paraphrase (\"you stopped me\") — quote the literal message and name the canceled tool + its args, so the user can tell what was in flight vs. what landed |\n\nPlatform-specific errors (TCC dialogs on macOS, Session 0 \u002F UAC\nprompts on Windows, AT-SPI bus issues on Linux) live in their\nrespective companion files.\n\n## Things to avoid\n\n- **Never** reuse an element target across a re-snapshot of the same window.\n  A new snapshot invalidates older tokens immediately. Bare `element_index`\n  input is rejected; use `element_token` or `element_index` + `snapshot_id`.\n- **Don't conflate the two addressing modes.** The tree gives you\n  `element_index` handles; the screenshot (same call) gives you the\n  pixel frame. An **element ax action** addresses by index, an\n  **element px action** by `x,y`. Default to `element_index` and only\n  do a px action on a real signal (`suspected_noop` \u002F `degraded` \u002F\n  repeated labels \u002F tree-disagrees-with-pixels). Don't pass an\n  `element_index` you read off the screenshot, and don't pixel-click a\n  coordinate you computed from the tree's (possibly lying) frame\n  without checking it against the image.\n- **Prefer accessibility actions over pixels.** `click({pid, x, y})`\n  works for canvas \u002F WebView regions, but it lands blindly on raw\n  coordinates. Exhaust accessibility paths (menu bars, cmd-k palettes,\n  toolbar items, keyboard shortcuts) before dropping to coordinates.\n  (The AX path does **not** skip the agent-cursor overlay — it seeds and\n  pulses the session cursor and draws a focus rect on the targeted\n  element; it just doesn't play a long glide on the very first action.\n  See \"Agent cursor overlay\" for the demo-recording caveat.)\n- **Never** drive destructive actions (delete files, close unsaved\n  documents, send messages, submit forms) without explicit user\n  intent for that specific destructive step.\n- **Never** launch apps autonomously; confirm with the user first\n  unless their original request clearly implies the launch.\n\n## Example end-to-end task\n\n**User:** \"Open the Downloads folder in the system file manager.\"\n\n1. `launch_app({bundle_id: \"com.apple.finder\", urls: [\"~\u002FDownloads\"]})`\n   on macOS, or `launch_app({name: \"explorer\", args: [\"%USERPROFILE%\\\\Downloads\"]})`\n   on Windows. Returns `{pid, windows: [{window_id, title, ...}]}`.\n   Idempotent launch; the driver opens a hidden window via the\n   platform's launch primitive — zero activation, no focus steal.\n2. `get_window_state({pid, window_id})` → verify the expected window\n   title is present with a populated tree (sidebar, list view, files).\n3. Done.\n\nPlatform-specific examples and edge cases (Finder menu navigation,\nExplorer ribbon, GNOME Files) live in the per-OS companion files.\n",{"name":5,"version":53,"metadata":54,"description":7},"0.24.0",{"openclaw":55},{"envVars":56,"homepage":75,"requires":76},[57,60,63,66,69,72],{"name":58,"required":49,"description":59},"CUA_DRIVER_EMBEDDED","Set to 1 when a macOS host app launches the driver in embedded mode.",{"name":61,"required":49,"description":62},"CUA_DRIVER_HOST_BUNDLE_ID","Bundle identifier of the macOS host app in embedded mode.",{"name":64,"required":49,"description":65},"CUA_DRIVER_PATH","Optional path to a cua-driver binary used by an embedding host.",{"name":67,"required":49,"description":68},"CUA_DRIVER_RS_ENABLE_WAYLAND","Set to 1 to enable the native Wayland backend.",{"name":70,"required":49,"description":71},"CUA_DRIVER_RS_MCP_HTTP_PORT","Optional port for the local MCP HTTP endpoint.",{"name":73,"required":49,"description":74},"CUA_DRIVER_RS_MCP_HTTP_TOKEN","Required host-generated bearer token when the local MCP HTTP endpoint is enabled.","https:\u002F\u002Fcua.ai\u002Fdocs\u002Fcua-driver",{"bins":77},[5],"main","5acbba39180bbea80204d1a58db40ccbd813a13a","config\u002Fpi\u002Fagent\u002Fskills\u002Fcua-driver",0,"https:\u002F\u002Fgithub.com\u002Fsanurb\u002F.dotfiles","https:\u002F\u002Fgithub.com\u002Fsanurb","Building things","2026-11-08 21:22:35",{"source":8,"skillName":5,"commands":87,"prompt":97,"zipAvailable":98},[88,91,94],{"tool":89,"command":90},"npx","npx skills add sanurb\u002F.dotfiles --skill cua-driver",{"tool":92,"command":93},"bunx","bunx skills add sanurb\u002F.dotfiles --skill cua-driver",{"tool":95,"command":96},"pnpm","pnpm dlx skills add sanurb\u002F.dotfiles --skill cua-driver","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fsanurb\u002F.dotfiles\u002Fblob\u002Fmain\u002Fconfig\u002Fpi\u002Fagent\u002Fskills\u002Fcua-driver\u002FSKILL.md",true,[100,106,110,115,119,123,127,131,135],{"path":101,"resourceType":102,"resourceTypeDesc":103,"sizeBytes":104,"sourceHtmlUrl":105},"BROWSER.md","REFERENCE","参考文档",24203,"https:\u002F\u002Fgithub.com\u002Fsanurb\u002F.dotfiles\u002Fblob\u002Fmain\u002Fconfig\u002Fpi\u002Fagent\u002Fskills\u002Fcua-driver\u002FBROWSER.md",{"path":107,"resourceType":102,"resourceTypeDesc":103,"sizeBytes":108,"sourceHtmlUrl":109},"EMBEDDING.md",28678,"https:\u002F\u002Fgithub.com\u002Fsanurb\u002F.dotfiles\u002Fblob\u002Fmain\u002Fconfig\u002Fpi\u002Fagent\u002Fskills\u002Fcua-driver\u002FEMBEDDING.md",{"path":111,"resourceType":112,"resourceTypeDesc":113,"sizeBytes":114,"sourceHtmlUrl":16},"LICENSE.md","LICENSE","许可证",1069,{"path":116,"resourceType":102,"resourceTypeDesc":103,"sizeBytes":117,"sourceHtmlUrl":118},"LINUX.md",21346,"https:\u002F\u002Fgithub.com\u002Fsanurb\u002F.dotfiles\u002Fblob\u002Fmain\u002Fconfig\u002Fpi\u002Fagent\u002Fskills\u002Fcua-driver\u002FLINUX.md",{"path":120,"resourceType":102,"resourceTypeDesc":103,"sizeBytes":121,"sourceHtmlUrl":122},"LOCAL.md",2266,"https:\u002F\u002Fgithub.com\u002Fsanurb\u002F.dotfiles\u002Fblob\u002Fmain\u002Fconfig\u002Fpi\u002Fagent\u002Fskills\u002Fcua-driver\u002FLOCAL.md",{"path":124,"resourceType":102,"resourceTypeDesc":103,"sizeBytes":125,"sourceHtmlUrl":126},"MACOS.md",30791,"https:\u002F\u002Fgithub.com\u002Fsanurb\u002F.dotfiles\u002Fblob\u002Fmain\u002Fconfig\u002Fpi\u002Fagent\u002Fskills\u002Fcua-driver\u002FMACOS.md",{"path":128,"resourceType":102,"resourceTypeDesc":103,"sizeBytes":129,"sourceHtmlUrl":130},"README.md",4027,"https:\u002F\u002Fgithub.com\u002Fsanurb\u002F.dotfiles\u002Fblob\u002Fmain\u002Fconfig\u002Fpi\u002Fagent\u002Fskills\u002Fcua-driver\u002FREADME.md",{"path":132,"resourceType":102,"resourceTypeDesc":103,"sizeBytes":133,"sourceHtmlUrl":134},"RECORDING.md",8074,"https:\u002F\u002Fgithub.com\u002Fsanurb\u002F.dotfiles\u002Fblob\u002Fmain\u002Fconfig\u002Fpi\u002Fagent\u002Fskills\u002Fcua-driver\u002FRECORDING.md",{"path":136,"resourceType":102,"resourceTypeDesc":103,"sizeBytes":137,"sourceHtmlUrl":138},"WINDOWS.md",49830,"https:\u002F\u002Fgithub.com\u002Fsanurb\u002F.dotfiles\u002Fblob\u002Fmain\u002Fconfig\u002Fpi\u002Fagent\u002Fskills\u002Fcua-driver\u002FWINDOWS.md",[140,165,193,219,237,262],{"skillNo":141,"name":142,"nameZh":6,"description":143,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":144,"categories":145,"keywords":154,"publishedAt":48},"01M71SS19VAM2YZ2SA7HD0PT2P","writing-great-skills","Reference for writing and editing skills well — the vocabulary and principles that make a skill predictable.","https:\u002F\u002Fgithub.com\u002Fsanurb\u002F.dotfiles\u002Fblob\u002Fmain\u002Fconfig\u002Fpi\u002Fagent\u002Fskills\u002Fwriting-great-skills\u002FSKILL.md",[146,149],{"categoryNo":147,"categoryName":148,"taxonomyType":22,"taxonomyTypeDesc":23},"01JBSEEDCAT000000000000018","写作与文案",{"categoryNo":150,"categoryName":151,"taxonomyType":152,"taxonomyTypeDesc":153},"01JBSEEDCAT000000000000036","内容创作者","PROFESSION","适用职业",[155,156,157,158,159,160,161,162,163,164],"writing great skills","writing","principles","for","editing","a skill","true","reference","make","vocabulary",{"skillNo":166,"name":167,"nameZh":6,"description":168,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":169,"categories":170,"keywords":174,"publishedAt":48},"01M71SRXKJN6RYXPN9GFRVYSR3","write-discoverable-code","Rules for writing code that coding agents (and humans) can find and understand through\nplain-text search. Apply whenever writing or renaming code: functions, types, constants,\nfiles, error messages, doc comments.\n\nGrounded in measurement: agents navigate by plain-text search, not by AST or\nlanguage server, so every identifier is a search query and every search miss\ncosts wasted reads.","https:\u002F\u002Fgithub.com\u002Fsanurb\u002F.dotfiles\u002Fblob\u002Fmain\u002Fconfig\u002Fpi\u002Fagent\u002Fskills\u002Fwrite-discoverable-code\u002FSKILL.md",[171],{"categoryNo":172,"categoryName":173,"taxonomyType":152,"taxonomyTypeDesc":153},"01JBSEEDCAT000000000000030","开发工程师",[175,176,177,178,179,180,181,182,158,183,184,185,186,187,156,188,189,190,191,192],"write discoverable code","code","discoverable code","write","measurement","error","types","error messages","search","messages","constants","plain-text","files","Rules","query","apply","comments","can",{"skillNo":194,"name":195,"nameZh":6,"description":196,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":197,"categories":198,"keywords":200,"publishedAt":48},"01M71SRT760335ES5WSEAAQDN2","wrangler","Cloudflare Workers CLI for deploying, developing, and managing Workers, KV, R2, D1, Vectorize, Hyperdrive, Workers AI, Containers, Queues, Workflows, Pipelines, and Secrets Store. Load before running wrangler commands to ensure correct syntax and best practices. Biases towards retrieval from Cloudflare docs over pre-trained knowledge.","https:\u002F\u002Fgithub.com\u002Fsanurb\u002F.dotfiles\u002Fblob\u002Fmain\u002Fconfig\u002Fpi\u002Fagent\u002Fskills\u002Fwrangler\u002FSKILL.md",[199],{"categoryNo":20,"categoryName":21,"taxonomyType":22,"taxonomyTypeDesc":23},[195,201,202,203,204,205,206,207,208,209,210,211,212,158,213,214,215,216,217,218],"cloudflare","d1","knowledge","workflows","cli","secrets","commands","syntax","cloudflare workers","r2","queues","running","containers","correct","load","workers","best practices","store",{"skillNo":220,"name":221,"nameZh":6,"description":222,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":223,"categories":224,"keywords":225,"publishedAt":48},"01M71SRPHE452V400HD2EZC4XG","workers-best-practices","Reviews and authors Cloudflare Workers code against production best practices. Load when writing new Workers, reviewing Worker code, configuring wrangler.jsonc, or checking for common Workers anti-patterns (streaming, floating promises, global state, secrets, bindings, observability). Biases towards retrieval from Cloudflare docs over pre-trained knowledge.","https:\u002F\u002Fgithub.com\u002Fsanurb\u002F.dotfiles\u002Fblob\u002Fmain\u002Fconfig\u002Fpi\u002Fagent\u002Fskills\u002Fworkers-best-practices\u002FSKILL.md",[],[226,216,217,227,228,229,158,201,230,203,231,195,156,215,232,176,233,234,235,236],"workers best practices","observability","reviews","promises","worker","new","when","common","streaming","state","anti-patterns",{"skillNo":238,"name":239,"nameZh":6,"description":240,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":241,"categories":242,"keywords":244,"publishedAt":48},"01M71SRJ36HWMTQ9ZMQ2RB008S","web-perf","Analyzes web performance using Chrome DevTools MCP. Measures Core Web Vitals (LCP, INP, CLS) and supplementary metrics (FCP, TBT, Speed Index), identifies render-blocking resources, network dependency chains, layout shifts, caching issues, and accessibility gaps. Use when asked to audit, profile, debug, or optimize page load performance, Lighthouse scores, or site speed. Biases towards retrieval from current documentation over pre-trained knowledge.","https:\u002F\u002Fgithub.com\u002Fsanurb\u002F.dotfiles\u002Fblob\u002Fmain\u002Fconfig\u002Fpi\u002Fagent\u002Fskills\u002Fweb-perf\u002FSKILL.md",[243],{"categoryNo":25,"categoryName":26,"taxonomyType":22,"taxonomyTypeDesc":23},[245,246,247,248,249,250,251,252,253,254,203,39,232,255,256,257,258,259,260,261],"web perf","perf","web","resources","vitals","chrome devtools","performance","render","issues","devtools","caching","documentation","optimize","layout","audit","dependency","network",{"skillNo":263,"name":264,"nameZh":6,"description":265,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":266,"categories":267,"keywords":268,"publishedAt":48},"01M71SREWXFKFN6DM80YCCQZ46","wayfinder","Plan a huge chunk of work — more than one agent session can hold — as a shared map of decision tickets on your issue tracker, and resolve them one at a time until the way to the destination is clear.","https:\u002F\u002Fgithub.com\u002Fsanurb\u002F.dotfiles\u002Fblob\u002Fmain\u002Fconfig\u002Fpi\u002Fagent\u002Fskills\u002Fwayfinder\u002FSKILL.md",[],[264,269,270,271,272,273,274,161,275,276,277,278,192,279,280,281,282,283,284,285],"shared","as","chunk","time","session","decision","more","map","one","one at a time","agent session","work","plan","tracker","tickets","at","destination"]