AGENT SKILL

open-computer-use

Platform-neutral guidance for using Open Computer Use, the open-source Computer Use MCP server and CLI for macOS, Linux, and Windows. Use when an agent needs to install, verify, troubleshoot, configure, or operate Open Computer Use through its native CLI, stdio MCP server, or direct Computer Use tool calls.

License 许可
MIT
最近更新
2026-07-19
Stars
★ 1
Forks
1

01 前言 Front Matter

name
open-computer-use
description
Platform-neutral guidance for using Open Computer Use, the open-source Computer Use MCP server and CLI for macOS, Linux, and Windows. Use when an agent needs to install, verify, troubleshoot, configure, or operate Open Computer Use through its native CLI, stdio MCP server, or direct Computer Use tool calls.

02 资源文件 3

03 SKILL.md 正文

Open Computer Use

Overview

Open Computer Use exposes Computer Use as a local CLI and stdio MCP server. It is not Codex.app-specific; adapt the commands and MCP config to the agent runtime you are operating in.

The macOS runtime requires macOS 14.0 or later. Windows and Linux use their own platform runtimes and are not subject to this macOS minimum.

It supports the same core tool surface across macOS, Linux, and Windows: list_apps, get_app_state, click, perform_secondary_action, scroll, drag, type_text, press_key, and set_value.

Core Workflow

  1. On macOS, run sw_vers -productVersion before invoking the CLI and require macOS 14.0 or later. On older versions, explain that the runtime cannot launch; do not recommend doctor or permission changes as a fix for binary incompatibility.
  2. Check the CLI is installed with open-computer-use -h or ocu -h. If installation or setup is missing, read references/installation.md.
  3. On supported macOS versions, run open-computer-use doctor before the first real GUI task. If permissions are missing, ask the user to approve Accessibility and Screen Recording in the onboarding UI.
  4. Inspect available apps before acting: open-computer-use call list_apps.
  5. Capture current UI state with open-computer-use call get_app_state --args '{"app":"TextEdit"}'. The default state is usually enough for UI operation.
  6. When the task needs longer semantic text, such as chat history, email bodies, document text, or long form content, call get_app_state with text_limit: 1000 or text_limit: "max".
  7. When visible long pages or lists appear incomplete even after scrolling, call get_app_state with a larger max_tree_nodes or max_tree_depth.
  8. Prefer element-targeted actions using element_index from the latest get_app_state result.
  9. For multi-step CLI work, use open-computer-use call --calls '<json-array>' so one process can reuse the latest element index mapping.
  10. For agent runtimes that support local MCP servers, configure open-computer-use mcp or ocu mcp and call the exposed Computer Use tools directly. Read references/usage.md.
  11. If communication, permission, or desktop-session access fails, read references/troubleshooting.md.

Operating Rules

  • Treat the target desktop as the user's real session. Do not inspect password managers, unrelated private content, or sensitive apps unless the user explicitly asked for that task.
  • Ask before sending, deleting, purchasing, approving, uploading, or making other externally visible changes.
  • Do not assume Codex.app plugin helpers are available. Use the installed open-computer-use / ocu CLI or an explicit MCP config.
  • Always run get_app_state before using element_index; do not guess indexes across sessions or after large UI changes.
  • Prefer semantic actions and set_value for editable controls. Use coordinate click, scroll, and drag only when the element tree does not expose a safer target.
  • On macOS, do not enable OPEN_COMPUTER_USE_ALLOW_GLOBAL_POINTER_FALLBACKS=1 unless the user explicitly wants diagnostic behavior that may move the real pointer.
  • On Windows and Linux, confirm the command is running inside the logged-in desktop session before assuming GUI automation is available.

Common CLI Actions

open-computer-use -h
ocu -h
open-computer-use doctor
open-computer-use call list_apps
ocu call list_apps
open-computer-use call get_app_state --args '{"app":"TextEdit"}'
open-computer-use call get_app_state --args '{"app":"TextEdit","text_limit":1000}'
open-computer-use call get_app_state --args '{"app":"TextEdit","text_limit":"max"}'
open-computer-use call get_app_state --args '{"app":"Google Chrome","max_tree_nodes":3000,"max_tree_depth":96}'
open-computer-use call click --args '{"app":"TextEdit","element_index":"0"}'
open-computer-use call type_text --args '{"app":"TextEdit","text":"Hello from Open Computer Use"}'

For a short sequence that reuses state in one process:

open-computer-use call --calls '[
  {"tool":"get_app_state","args":{"app":"TextEdit"}},
  {"tool":"press_key","args":{"app":"TextEdit","key":"Return"}}
]'

MCP Usage

For runtimes that can launch local MCP servers over stdio, use:

[mcp_servers.open_computer_use]
command = "open-computer-use"
args = ["mcp"]

Read references/usage.md for JSON config examples, direct tool-call patterns, and platform notes.

Safari / grok.com (this machine)

When driving authenticated Safari on grok.com (chat modes, prompts, harvest), load the learned playbook — do not re-discover address-bar or mode-select failure modes:

~/.grok/skills/safari-browser/references/grok-com-computer-use.md

Key OCU facts from that playbook:

  • NEVER BATCH. Screenshot → analyze → one Act → screenshot. Screenshot every 3–5s while operating.
  • OODA mandatory. Decisions from screenshots only.
  • type_text appends even when text is selected — clear with Backspace before typing URLs.
  • Prefer real cursor (Quartz) + System Events screen geometry for grok Model select and menus; screenshot pixel coords are not screen points.
  • After each round: harvest via Copy at the end of the response (action bar visible in screenshot) → clipboard.

References

  • references/installation.md: one-time CLI install, agent MCP install commands, and macOS permissions.
  • references/usage.md: MCP config, direct CLI calls, sequencing, and platform behavior.
  • references/troubleshooting.md: permission, desktop-session, app discovery, and action failures.
  • Safari grok playbook: ~/.grok/skills/safari-browser/references/grok-com-computer-use.md

04 同仓库更多 Skills

zfifteen
zfifteen/skills-catalog

Zotero

Use Zotero Desktop from Codex to enable/probe the local API, search a local Zotero library, list items/collections/tags, export BibTeX, insert citation keys into LaTeX or Markdown drafts, read indexed full text when requested, and import BibTeX/RIS records into Zotero through the connector server. Use when the user mentions Zotero, citations, references.bib, BibTeX export, local Zotero API, localhost:23119, or adding citations from a Zotero library.

★ 1
zfifteen
zfifteen/skills-catalog

zoom-apps-sdk

Use when using Apps SDK.

★ 1
zfifteen
zfifteen/skills-catalog

youtube

Obtain accurate, timestamped YouTube video transcripts (preferring manual captions over auto-generated) as the primary deliverable, then derive summaries, key claims, or notes from that transcript. Use when the user provides a YouTube URL (or video ID) and wants the transcript, a faithful summary, key claims with attribution, timestamped notes, or "what does this video actually say". The skill is transcript-first: never substitute metadata, title, description, or search snippets for the spoken content. Triggers: "youtube transcript", "get the transcript of this video", "summarize this youtube video", "what did they say in https://youtube.com/...", "key claims from this talk".

★ 1
zfifteen
zfifteen/skills-catalog

yeet

Publish local changes to GitHub by confirming scope, committing intentionally, pushing the branch, and opening a draft PR through the GitHub app from this plugin, with `gh` used only as a fallback where connector coverage is insufficient.

★ 1
zfifteen
zfifteen/skills-catalog

x-post-creator

Create X-ready post threads and co-located story packages from the active conversation or referenced project material. Use when the user says turn this into an X-post, /x-post, make this into an X thread, or wants plain-language social posts with supporting images saved to docs/x-posts/[story-name]/.

★ 1
zfifteen
zfifteen/skills-catalog

x-api

Search, read, and write X posts through the local X API MCP server (the same backend used by the Codex x-api skill). Use for searching X/x.com, reading threads and replies, creating lists, staging/publishing posts for @alltheputs, finding high-visibility reply targets for research outreach, ingesting public context, or any request that mentions /x-api or "search X".

★ 1