agent-desktop-operator
Operate desktop applications across Windows, macOS, and Linux using accessibility tree snapshot maps, qualified ref allocation, and 2D delivery-retry recovery. Do not use for DOM-only web scraping or pixel-matching OCR.
- License 许可
- MIT
- 最近更新
- 2026-11-15
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- agent-desktop-operator
- description
- Operate desktop applications across Windows, macOS, and Linux using accessibility tree snapshot maps, qualified ref allocation, and 2D delivery-retry recovery. Do not use for DOM-only web scraping or pixel-matching OCR.
02 资源文件 3
- 参考文档CARD.md3.4 KB
- 参考文档config.default.yaml396 B
- 脚本scripts/desktop_operator_engine.py13.6 KB
03 SKILL.md 正文
Agent Desktop Operator: OS Accessibility Computer Use Engine
agent-desktop-operator equips AI coding agents with direct desktop GUI operation across Windows, macOS, and Linux through native OS accessibility trees (UI Automation on Windows, AXUIElement on macOS, and AT-SPI on Linux) without screenshots or fragile pixel matching.
See CARD.md for the companion summary card, vocabulary definitions, and invariants checklist. Consult config.default.yaml for baseline operational budgets and timeout thresholds.
The 5-Stage Observe-Drill-Act-Verify Loop
[1. Preflight Status] ──► [2. Skeleton Mapping] ──► [3. Subtree Drill-Down]
│
▼
[5. Verify & 2D Recovery] ◄── [4. Semantic Action Dispatch] ◄┘
1. Preflight Status & Surface Discovery
Before interacting with desktop windows, verify runtime permissions and surface capabilities:
- Query Platform Status:
- Run
agent-desktop status(ordesktop_statusvia IoC service). - Check that
permissions_grantedis true and notesupported_surfaces(window,menu,dialog,sheet,alert).
- Run
- Handle Elevation & Sandboxing:
- On Windows, UI Automation interacts with standard and elevated windows under appropriate privilege levels.
- On macOS, confirm Accessibility grant under System Settings > Privacy & Security > Accessibility.
Completion criterion: Runtime reports
ok: true, OS accessibility API active, and surface types enumerated.
2. Skeleton Mapping for Dense Applications
Large applications can contain thousands of accessible nodes. Never read the entire application tree at full depth in a single turn:
- Generate Shallow Skeleton:
- Run
agent-desktop snapshot --skeleton --app "AppName" -i --compact. --skeletonclamps branch depth while recordingchildren_counton truncated branches.-ifilters to interactive actionable elements only.--compactstrips redundant geometry for token efficiency.
- Run
- Respect Web-Wrapper Bypass:
- In Chromium and Electron applications (window class
Chrome_WidgetWin_1), anonymous transparent container wrappers consume zero logical depth budget, preventing web views from exhausting observation quotas.
- In Chromium and Electron applications (window class
- Never Skeleton Focused Surfaces:
- When inspecting popovers, dropdowns, or system sheets, pass
--surface menuor--surface sheetdirectly without--skeleton.
- When inspecting popovers, dropdowns, or system sheets, pass
Completion criterion: Shallow skeleton snapshot generated with root-level navigation refs and candidate branch ref tokens.
3. Subtree Drill-Down
Once candidate containers are identified from the skeleton, drill down to reveal leaf actionable controls:
- Execute Root Drill:
- Run
agent-desktop snapshot --root '@s8f3k2p9:e3' -i --compact. - Drilling expands the selected subtree and dynamically allocates fresh refs within the same
snapshot_id. - Refs outside the drilled subtree remain valid; previous refs inside the expanded branch are superseded.
- Run
- PowerShell Quote Syntax:
- In PowerShell, always quote the ref:
'@s8f3k2p9:e3'. A bare@tokentriggers PowerShell argument splatting.
- In PowerShell, always quote the ref:
Completion criterion: Subtree expanded into leaf actionable refs without invalidating the parent snapshot context.
4. Semantic Action Dispatch
Act on qualified element references using headless semantic actions before resorting to physical pointer movement:
- Verify Qualified Ref Token (Rule 12):
- Validate that the target matches
@<snapshot_id>:e<N>. Reject bare@e8tokens to prevent stale mutation.
- Validate that the target matches
- Execute Headless Semantic Actions:
- Preferred commands:
click,set-value,type,select,toggle,check,uncheck,expand,collapse. - Headless actions dispatch directly via the OS accessibility API without pointer repositioning or focus stealing.
- Preferred commands:
- Headed Fallback for Custom Canvas Elements:
- When an application control lacks semantic action patterns, pass
--headed mouse-clickto simulate physical cursor positioning and hardware events.
- When an application control lacks semantic action patterns, pass
Completion criterion: Action dispatched with structured activation steps and parsed 2D disposition metadata.
5. Verify & 2D Disposition Recovery
Inspect action envelopes to confirm delivery and handle stale states deterministically:
- Evaluate 2D Outcome Matrix:
DeliveryDisposition:NotDelivered,DeliveryUncertain,DeliveredUnverified, orDeliveredVerified.RetryDisposition:Safe,Unsafe, orUnknown.- Transaction Rollback Rule: If delivery is
DeliveryUncertainor retry isUnsafe, trigger transaction rollback (Rule 8) and take a fresh observation.
- Handle Standard Error Refusals:
STALE_REF: The UI changed; re-drill the region or take a new snapshot. Never retry the stale ref.AMBIGUOUS_TARGET: An identical twin appeared or uniqueness was lost; retake snapshot with higher timeout or target an adjacent unique anchor.TIMEOUT(resolution_identity_unknown): Live read delayed; safe to retry.
- Dynamic State Waiting:
- After asynchronous actions, call
agent-desktop wait --element '@s:eN' --timeout-ms 5000orwait --window "Title"instead of inserting arbitrary time sleeps.
- After asynchronous actions, call
Completion criterion: Action verified via subsequent observation or safely rolled back upon uncertain delivery.
The Three Foundational Pillars
1. The Visual Brief Pillar
Every major desktop introspection session renders an interactive HTML visual brief in %TEMP% displaying the hierarchical accessibility tree, surface boundaries, and active ref allocations with Tailwind CSS and Mermaid diagrams.
2. The Mandatory Checkpoint Pillar
The agent must never execute destructive or non-idempotent desktop mutations without verifying qualified ref tokens (@<snapshot_id>:e<N>) and checking is_retry_safe disposition gates before dispatch.
3. Evidence Tri-State & Subprocess Isolation
Never collapse Unknown into Absent or boolean False. A failed property read or timed-out subtree must withhold uniqueness claims (CaptureProof) rather than assuming an element is unique, executing behind isolated subprocess transports with Rule 14 pipe drainage.
Anti-Patterns
- Bare Ref Splatting — Passing
@e12instead of qualified'@s8f3k2p9:e12'in PowerShell, triggering argument splatting and command failure. - Fixed Sleep Polling — Inserting arbitrary
time.sleep()after actions instead of using condition-basedagent-desktop wait. - Skeletonizing Focused Surfaces — Applying
--skeletonto menus or modal dialogs, which are already focused and compact. - Double-Clicking Uncertain Actions — Repeating an action whose delivery was
DeliveryUncertainwithout checkingis_retry_safe. - Pixel-Matching Fragility — Attempting coordinate OCR or visual screenshot diffing when native semantic accessibility trees are available.
- Leaking Subprocess Transports — Omitting
finallyblock pipe drainage on async subprocess CLI runners (Rule 14 & Rule 53).
04 同仓库更多 Skills
ui-behavior-primitive-architect
Architect, audit, and verify zero-dependency headless UI behavior primitives, WebMCP agent bridges, WAI-ARIA/SSR/RTL tri-modal checks, and framework expert routing. Do not use for static CSS theme styling or backend database migrations.
trustgraph-ontology-governor
Govern knowledge graphs, SHACL-AF triple policies, OntologyRAG subclass validation, PROV-O dual named graphs, and Laya entropy decision scoring. Do not use for relational SQL queries or generic web scraping.
shim-harness-governor
Govern universal CLI agent loops, 4-tier tool permissions, ancestor-symlink workspace sandboxes, deterministic-first command routing, collision-safe MCP namespacing, and layered skill loading. Do not use for static UI styling or external SQL analytics.
graphiti-zep-memory-governor
Govern bi-temporal knowledge graphs with Graphiti future-edge clamping, MinHash/LSH entity dedup, RRF/MMR reranking, and Zep typed ontologies. Do not use for static SQL joins or generic web crawling.
agora-mcp-workbench-architect
Architect, audit, and orchestrate Agora Workbench MCP servers, loopback sidecars, unified state graphs, gateway/dispatcher connectors, and policy-enforced data lakes. Do not use for static frontend CSS styling or non-MCP desktop GUI automation.
skill-eval-assurance-runner
Evaluate and assure agent skills using small-N Wilson confidence intervals, unbiased pass^k reliability, deterministic trajectory graders, preflight boundary honesty, and sealed digest verification. Do not use for subjective prose review or unconstrained production deployment.