AGENT SKILL

webskrap

Use when writing, debugging, documenting, or reviewing Python scraping, web search, and visual browser automation with WebSkrap. Not for ordinary HTTP clients or search APIs.

License 许可
Apache-2.0
所属仓库
kacigaya/webskrap
最近更新
2026-11-15
Stars
★ 1
Forks
0

01 前言 Front Matter

name
webskrap
description
Use when writing, debugging, documenting, or reviewing Python scraping, web search, and visual browser automation with WebSkrap. Not for ordinary HTTP clients or search APIs.

02 资源文件 1

03 SKILL.md 正文

WebSkrap

WebSkrap is an async Python scraping package that drives Chromium through its own Chrome DevTools Protocol (CDP) client, with no Playwright. The Python API, webskrap CLI, and webskrap-mcp server share core behavior.

WebSkrap loads known URLs. To find URLs it loads DuckDuckGo or Bing's results page in the same browser and parses the hits. It calls no search API and it does not offer Google.

Workflow

  1. Identify the affected public surface: Python, CLI, MCP, or more than one.
  2. Read its implementation and nearest tests. Trace calls into shared code before editing.
  3. Change the shared implementation when behavior belongs to more than one surface. Do not patch each wrapper independently.
  4. Update affected public contracts: models, serialization, errors, command help, MCP schemas, guides, and tests.
  5. Run focused tests during development, then the full validation gate before handoff when practical.

Route by task

Task Implementation Tests and contracts
Fetching, sessions, links, or screenshots src/webskrap/client.py, src/webskrap/models.py tests/test_client_unit.py, tests/test_models.py
Search engines, redirect unwrapping, or results parsing src/webskrap/search.py, src/webskrap/models.py tests/test_search.py and its fixtures under tests/fixtures/search/
Parsing src/webskrap/parsing.py tests/test_parsing.py
Errors, hints, or exit statuses src/webskrap/errors.py, src/webskrap/cli_output.py tests/test_errors.py, CLI and MCP callers
Main CLI commands src/webskrap/cli.py tests/test_cli.py, tests/test_diagnostics.py
Persistent browser state src/webskrap/browser_session.py, src/webskrap/paths.py tests/test_browser_session_unit.py, tests/test_paths.py
Interactive browser commands src/webskrap/browser_cli.py, src/webskrap/mcp_server.py tests/test_browser_cli.py, tests/test_browser_integration.py, tests/test_mcp.py
MCP tools, resources, or annotations src/webskrap/mcp_server.py tests/test_mcp.py, src/webskrap/guide.md
Profiles or stealth defaults src/webskrap/profiles.py, src/webskrap/models.py, src/webskrap/client.py tests/test_profiles.py, opt-in bot-detection tests when needed
Cookie rejection src/webskrap/consent.py tests/test_consent.py; tests/test_consent_live.py only for selector rot
Documentation and examples README.md, src/webskrap/guide.md tests/test_docs.py, current CLI help, public Python exports

Search for every constructor, serializer, command, and schema that uses a changed model or field. Fetch and search behavior cross Python, CLI, and MCP. Persistent browser behavior crosses the browser CLI and MCP. Engine markup lives only in src/webskrap/search.py. When an engine changes its page, refresh that engine's fixture and fix that one extractor.

Choose the public surface

Need Python CLI MCP
Fetch one known URL client.fetch() webskrap fetch stealth_fetch
Find URLs for a query client.search() webskrap search search
Preserve cookies or storage client.session() webskrap browser open browser_open
See a rendered page or interact with canvas browser_session.view() on a session page webskrap browser view and mouse browser_view and browser_mouse
Click, fill, wait, or run a flow session.new_page() and human_click() webskrap browser browser_interact and related tools
Diagnose installation Inspect raised error webskrap doctor doctor

Use a one-shot fetch for one page. Use a persistent browser only when state or interaction must survive between operations. Do not repeatedly fetch a page to drive a multi-step flow.

Read only the reference needed for the task:

  • Python API: sessions, pages, profile modes, consent, links, and fingerprint controls.
  • CLI: one-shot fetches, persistent browser commands, and Linux ARM64 behavior.
  • MCP: tool selection, confined paths, annotations, and keeping tool results small.

Visual browser interaction

Use browser_view (MCP) or webskrap browser view --format json (CLI) when layout, images, canvas, or controls missing from the accessibility tree matter. MCP returns an image block directly. CLI saves a PNG; open its returned path with the agent's image-viewing tool before choosing coordinates. A path or base64 text alone does not give a model vision. The client and model must support image input.

The view covers the viewport at one PNG pixel per CSS pixel. Coordinates start at its top-left; use the original width and height if the image viewer resizes it. Do not add scroll_x or scroll_y to mouse coordinates. Full-page screenshots are for inspection or export and do not map directly to the mouse.

Prefer refs/selectors when they identify a control. Use browser_mouse / CLI mouse for visual targets, then browser_insert_text / CLI insert-text for literal text in the focused control and browser_press / CLI press for keys. Wait for the expected change and take a fresh view after scrolling, navigation, or layout changes. Coordinate clicks hit whatever occupies that point. Run dependent actions sequentially within a session.

This is the browser's rendered page, available headless without Xvfb. It does not show browser chrome, OS dialogs, or a desktop stream. One-shot fetch sessions are separate; open a persistent browser to drive a visual flow. Read the CLI or MCP reference for examples.

Guardrails

  • Use public pages, local test servers, or targets the user is authorized to access.
  • Do not add CAPTCHA solving, login-wall bypass, credential bypass, or access-control circumvention.
  • Do not commit cookies, storage state, proxy credentials, or persistent browser data such as .webskrap/.
  • Preserve Chromium's sandbox by default. Any opt-out must remain explicit and outside page-controlled input.
  • Keep MCP file paths confined to their configured roots. Reject absolute paths, traversal, and symlink escapes.
  • Prefer public webskrap exports. Use private helpers only in tests.
  • Keep changes typed. Add focused tests for parsing, state transitions, output contracts, and tool-safety behavior.

Public contract checklist

When public behavior changes, check all applicable items:

  • Python models and exports
  • CLI arguments, help, JSON output, and exit status
  • MCP input schema, result shape, annotations, and server instructions
  • README.md and src/webskrap/guide.md
  • unit tests for each affected surface

Use the implementation as the source of truth. In particular, read src/webskrap/models.py for result fields, src/webskrap/errors.py for error codes and exits, and src/webskrap/diagnostics.py for environment reporting. Do not copy these catalogs into the skill.

Validation

Run the nearest tests while editing. Before handoff, run the repository gate when practical:

uv run --extra dev pytest -q -m "not browser and not live"
uv run --extra dev ruff check .
uv run --extra dev ruff format --check .
uvx --from pyright==1.1.411 pyright --pythonpath .venv/bin/python src/webskrap
uv run --extra dev bandit -c pyproject.toml -q -r src/
uv run --extra dev --with pip-audit==2.9.0 pip-audit
uv build

Run browser tests when browser behavior changes and the required browsers are installed.

Use WEBSKRAP_LIVE=1 pytest -q -m live only when explicitly checking public third-party behavior. Live tests are non-deterministic and do not replace the default suite.

04 同仓库更多 Skills