webskrap
Use when writing, debugging, documenting, or reviewing Python scraping, web search, and visual browser automation with WebSkrap. Not for ordinary HTTP clients or search APIs.
- License 许可
- Apache-2.0
- 最近更新
- 2026-11-15
- Stars
- ★ 1
- Forks
- 0
01 前言 Front Matter
- name
- webskrap
- description
- Use when writing, debugging, documenting, or reviewing Python scraping, web search, and visual browser automation with WebSkrap. Not for ordinary HTTP clients or search APIs.
02 资源文件 1
- 其他references19 B
03 SKILL.md 正文
WebSkrap
WebSkrap is an async Python scraping package that drives Chromium through its
own Chrome DevTools Protocol (CDP) client, with no Playwright. The Python API,
webskrap CLI, and webskrap-mcp server share core behavior.
WebSkrap loads known URLs. To find URLs it loads DuckDuckGo or Bing's results page in the same browser and parses the hits. It calls no search API and it does not offer Google.
Workflow
- Identify the affected public surface: Python, CLI, MCP, or more than one.
- Read its implementation and nearest tests. Trace calls into shared code before editing.
- Change the shared implementation when behavior belongs to more than one surface. Do not patch each wrapper independently.
- Update affected public contracts: models, serialization, errors, command help, MCP schemas, guides, and tests.
- Run focused tests during development, then the full validation gate before handoff when practical.
Route by task
| Task | Implementation | Tests and contracts |
|---|---|---|
| Fetching, sessions, links, or screenshots | src/webskrap/client.py, src/webskrap/models.py |
tests/test_client_unit.py, tests/test_models.py |
| Search engines, redirect unwrapping, or results parsing | src/webskrap/search.py, src/webskrap/models.py |
tests/test_search.py and its fixtures under tests/fixtures/search/ |
| Parsing | src/webskrap/parsing.py |
tests/test_parsing.py |
| Errors, hints, or exit statuses | src/webskrap/errors.py, src/webskrap/cli_output.py |
tests/test_errors.py, CLI and MCP callers |
| Main CLI commands | src/webskrap/cli.py |
tests/test_cli.py, tests/test_diagnostics.py |
| Persistent browser state | src/webskrap/browser_session.py, src/webskrap/paths.py |
tests/test_browser_session_unit.py, tests/test_paths.py |
| Interactive browser commands | src/webskrap/browser_cli.py, src/webskrap/mcp_server.py |
tests/test_browser_cli.py, tests/test_browser_integration.py, tests/test_mcp.py |
| MCP tools, resources, or annotations | src/webskrap/mcp_server.py |
tests/test_mcp.py, src/webskrap/guide.md |
| Profiles or stealth defaults | src/webskrap/profiles.py, src/webskrap/models.py, src/webskrap/client.py |
tests/test_profiles.py, opt-in bot-detection tests when needed |
| Cookie rejection | src/webskrap/consent.py |
tests/test_consent.py; tests/test_consent_live.py only for selector rot |
| Documentation and examples | README.md, src/webskrap/guide.md |
tests/test_docs.py, current CLI help, public Python exports |
Search for every constructor, serializer, command, and schema that uses a
changed model or field. Fetch and search behavior cross Python, CLI, and MCP.
Persistent browser behavior crosses the browser CLI and MCP. Engine markup
lives only in src/webskrap/search.py. When an engine changes its page,
refresh that engine's fixture and fix that one extractor.
Choose the public surface
| Need | Python | CLI | MCP |
|---|---|---|---|
| Fetch one known URL | client.fetch() |
webskrap fetch |
stealth_fetch |
| Find URLs for a query | client.search() |
webskrap search |
search |
| Preserve cookies or storage | client.session() |
webskrap browser open |
browser_open |
| See a rendered page or interact with canvas | browser_session.view() on a session page |
webskrap browser view and mouse |
browser_view and browser_mouse |
| Click, fill, wait, or run a flow | session.new_page() and human_click() |
webskrap browser |
browser_interact and related tools |
| Diagnose installation | Inspect raised error | webskrap doctor |
doctor |
Use a one-shot fetch for one page. Use a persistent browser only when state or interaction must survive between operations. Do not repeatedly fetch a page to drive a multi-step flow.
Read only the reference needed for the task:
- Python API: sessions, pages, profile modes, consent, links, and fingerprint controls.
- CLI: one-shot fetches, persistent browser commands, and Linux ARM64 behavior.
- MCP: tool selection, confined paths, annotations, and keeping tool results small.
Visual browser interaction
Use browser_view (MCP) or webskrap browser view --format json (CLI) when
layout, images, canvas, or controls missing from the accessibility tree matter.
MCP returns an image block directly. CLI saves a PNG; open its returned path
with the agent's image-viewing tool before choosing coordinates. A path or
base64 text alone does not give a model vision. The client and model must
support image input.
The view covers the viewport at one PNG pixel per CSS pixel. Coordinates start
at its top-left; use the original width and height if the image viewer
resizes it. Do not add scroll_x or scroll_y to mouse coordinates. Full-page
screenshots are for inspection or export and do not map directly to the mouse.
Prefer refs/selectors when they identify a control. Use browser_mouse / CLI
mouse for visual targets, then browser_insert_text / CLI insert-text for
literal text in the focused control and browser_press / CLI press for keys.
Wait for the expected change and take a fresh view after scrolling, navigation,
or layout changes. Coordinate clicks hit whatever occupies that point.
Run dependent actions sequentially within a session.
This is the browser's rendered page, available headless without Xvfb. It does not show browser chrome, OS dialogs, or a desktop stream. One-shot fetch sessions are separate; open a persistent browser to drive a visual flow. Read the CLI or MCP reference for examples.
Guardrails
- Use public pages, local test servers, or targets the user is authorized to access.
- Do not add CAPTCHA solving, login-wall bypass, credential bypass, or access-control circumvention.
- Do not commit cookies, storage state, proxy credentials, or persistent browser
data such as
.webskrap/. - Preserve Chromium's sandbox by default. Any opt-out must remain explicit and outside page-controlled input.
- Keep MCP file paths confined to their configured roots. Reject absolute paths, traversal, and symlink escapes.
- Prefer public
webskrapexports. Use private helpers only in tests. - Keep changes typed. Add focused tests for parsing, state transitions, output contracts, and tool-safety behavior.
Public contract checklist
When public behavior changes, check all applicable items:
- Python models and exports
- CLI arguments, help, JSON output, and exit status
- MCP input schema, result shape, annotations, and server instructions
README.mdandsrc/webskrap/guide.md- unit tests for each affected surface
Use the implementation as the source of truth. In particular, read
src/webskrap/models.py for result fields, src/webskrap/errors.py for error
codes and exits, and src/webskrap/diagnostics.py for environment reporting.
Do not copy these catalogs into the skill.
Validation
Run the nearest tests while editing. Before handoff, run the repository gate when practical:
uv run --extra dev pytest -q -m "not browser and not live"
uv run --extra dev ruff check .
uv run --extra dev ruff format --check .
uvx --from pyright==1.1.411 pyright --pythonpath .venv/bin/python src/webskrap
uv run --extra dev bandit -c pyproject.toml -q -r src/
uv run --extra dev --with pip-audit==2.9.0 pip-audit
uv build
Run browser tests when browser behavior changes and the required browsers are installed.
Use WEBSKRAP_LIVE=1 pytest -q -m live only when explicitly checking public
third-party behavior. Live tests are non-deterministic and do not replace the
default suite.