AGENT SKILL

visual-rpa

Visual RPA desktop automation skill. Use when user asks to operate desktop apps, click icons, open applications, type text in input fields, click buttons, scroll pages, send messages via WeChat or other apps. Uses screen capture and Qwen vision model for pure visual positioning without DOM or accessibility APIs.

License 许可
MIT
最近更新
2026-06-07
Stars
★ 0
Forks
1

01 前言 Front Matter

name
visual-rpa
description
Visual RPA desktop automation skill. Use when user asks to operate desktop apps, click icons, open applications, type text in input fields, click buttons, scroll pages, send messages via WeChat or other apps. Uses screen capture and Qwen vision model for pure visual positioning without DOM or accessibility APIs.

03 SKILL.md 正文

Visual RPA Desktop Automation

Auto-execute all steps without waiting for user confirmation between steps.

Desktop automation via screen capture + Qwen vision model (Qwen-VL). No DOM or accessibility API needed.

How it works

  1. Capture screen -> thumbnail rough positioning
  2. Full-resolution crop -> precise coordinate refinement
  3. Execute mouse/keyboard action -> screenshot verification
  4. Compound instructions automatically decomposed into atomic steps

Usage

Use exec tool to run commands. Script path: $env:TAXBOT_ROOT/skills/visual-rpa/scripts/visual_rpa.py

Requires DASHSCOPE_API_KEY environment variable to be set.

Single task

python "$env:TAXBOT_ROOT/skills/visual-rpa/scripts/visual_rpa.py" --mode task --task "click to open WeChat"

Compound task (auto-decomposed)

python "$env:TAXBOT_ROOT/skills/visual-rpa/scripts/visual_rpa.py" --mode task --task "open WeChat, open File Transfer chat, type hello in input box, click send"

Multi-step task (manually specified)

python "$env:TAXBOT_ROOT/skills/visual-rpa/scripts/visual_rpa.py" --mode task --task "click Chrome browser" "type baidu.com in address bar and press enter" "type weather in search box" "click search button"

Skip verification (faster)

python "$env:TAXBOT_ROOT/skills/visual-rpa/scripts/visual_rpa.py" --mode task --no-verify --task "click to open Calculator"

Parameters

Parameter Description
--mode task Batch task mode (required)
--mode interactive Interactive mode (default)
--task "step1" "step2" Task instructions, supports multiple
--no-verify Skip post-action verification
--model MODEL Vision model name (default: qwen-vl-max-latest)
--api-key KEY API Key (defaults to DASHSCOPE_API_KEY env var)

Supported actions

Action Example instructions
Click "click start menu", "click Chrome icon"
Double click "double click Recycle Bin on desktop"
Right click "right click on desktop blank area"
Type text "type weather in search box", "type hello in input box"
Hotkey "press Ctrl+C"
Scroll "scroll down the page"
Wait "wait for page to load"

Instruction tips

  • Be specific: "click WeChat icon on taskbar" is better than "open WeChat"
  • Instructions can be in Chinese or English, the model understands both
  • Complex operations can be written as compound instructions, system auto-decomposes
  • For text input: say "type XXX in YYY", system auto-detects as input action

Output format

  [OK] Step 0: click to open WeChat
       click @ (375,1591)
  [OK] Step 1: click File Transfer Assistant in WeChat
       click @ (154,97)
  [FAIL] Step 2: type hello in input box
       type @ (300,1364)
  2/3 succeeded
  • OK = action succeeded and verified
  • FAIL = action failed or verification failed, auto-retries up to 3 times

Common scenarios

Send WeChat message

python "$env:TAXBOT_ROOT/skills/visual-rpa/scripts/visual_rpa.py" --mode task --task "open WeChat, open File Transfer Assistant chat, type hello in input box, click send"

Open app and navigate

python "$env:TAXBOT_ROOT/skills/visual-rpa/scripts/visual_rpa.py" --mode task --task "click Chrome browser" "type https://www.baidu.com in address bar and press enter"

Desktop operations

python "$env:TAXBOT_ROOT/skills/visual-rpa/scripts/visual_rpa.py" --mode task --task "right click on desktop blank area" "click New Folder"

Notes

  • Each step takes 3-8 seconds (screenshot + API calls + verification)
  • Chinese text input uses clipboard paste, will overwrite current clipboard
  • Only operates on primary screen
  • Logs and screenshots saved in ./rpa_logs/ directory for debugging

04 同仓库更多 Skills

javimosch
javimosch/open-claw-skills

crowd-prompting

A marketplace where AI agents improve prompts, system instructions, tool descriptions, and other text-based content with domain expertise from real-world operations — and earn tokens for valuable contributions.

★ 0
javimosch
javimosch/open-claw-skills

ceaser

Interact with the Ceaser privacy protocol on Base L2 using the ceaser-mcp MCP tools. This skill uses the ceaser-mcp npm package for ALL operations -- shield, unshield, note management, and protocol queries. All ceaser tool calls use CLI subcommands (npx -y ceaser-mcp <subcommand>). Ceaser is a privacy-preserving ETH wrapper using Noir/UltraHonk zero-knowledge proofs on Base L2 (chain ID 8453).

★ 0
javimosch
javimosch/open-claw-skills

ceaser-send

Fully automated private ETH transfer via Ceaser Protocol on Base L2 using the ceaser-mcp MCP tools. This skill uses the ceaser-mcp npm package for ALL operations -- all ceaser tool calls use CLI subcommands (npx -y ceaser-mcp <subcommand>). WARNING -- creates observable on-chain links between user wallet and hot wallet (see Privacy Warning). Generates an ephemeral hot wallet with BIP-39 mnemonic recovery, user funds it, agent signs and broadcasts the Shield TX automatically. Exactly one manual step required (funding the hot wallet with ETH). Uses Noir/UltraHonk zero-knowledge proofs.

★ 0
javimosch
javimosch/open-claw-skills

dilemm-ai

Compete in DilemmAI, the prisoner's dilemma AI arena at dilemm.ai. Use when an OpenClaw agent wants to create an account, design and submit strategy prompts for their bot, enter matchmaking, analyze historical matches and leaderboards, and iterate on bot strategy. Requires headless browser automation (no API available yet).

★ 0
javimosch
javimosch/open-claw-skills

moldium

Post and manage content on the Moldium blog platform. Triggered by "post to Moldium", "write a blog post", "publish an article", etc.

★ 0
javimosch
javimosch/open-claw-skills

arxiv-paper-reviews

Interact with arXiv Crawler API to fetch papers, read reviews, submit comments, search papers, and import papers. Use when working with arXiv papers, fetching paper lists by date/category/interest, viewing paper details with comments, submitting paper reviews, searching papers by title, or importing papers from arXiv URLs via API at http://weakaccept.top:8000/.

★ 0