scrapling
Web scraping using Scrapling — a Python framework with anti-bot bypass (Cloudflare Turnstile, fingerprint spoofing), adaptive element tracking, stealth headless browser, and full CSS/XPath extraction. Use when web_fetch fails (Cloudflare, JS-rendered pages), or when extracting structured data from websites (prices, articles, lists). Supports HTTP, stealth, and full browser modes. Source: github.com/D4Vinci/Scrapling (PyPI: scrapling). Only use on sites you have permission to scrape.
- License 许可
- MIT
- 最近更新
- 2026-05-17
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- scrapling
- license
- MIT
- metadata
- { "pypi": "https://pypi.org/project/scrapling/", "source": "https://github.com/D4Vinci/Scrapling" }
- description
- Web scraping using Scrapling — a Python framework with anti-bot bypass (Cloudflare Turnstile, fingerprint spoofing), adaptive element tracking, stealth headless browser, and full CSS/XPath extraction. Use when web_fetch fails (Cloudflare, JS-rendered pages), or when extracting structured data from websites (prices, articles, lists). Supports HTTP, stealth, and full browser modes. Source: github.com/D4Vinci/Scrapling (PyPI: scrapling). Only use on sites you have permission to scrape.
02 资源文件 3
- 参考文档_meta.json481 B
- 参考文档references/patterns.md2.9 KB
- 脚本scripts/scrape.py2.7 KB
03 SKILL.md 正文
Scrapling Skill
Source: https://github.com/D4Vinci/Scrapling (open source, MIT-like license)
PyPI: scrapling — install before first use (see below)
⚠️ Only scrape sites you have permission to access. Respect
robots.txtand Terms of Service. Do not use stealth modes to bypass paywalls or access restricted content without authorization.
Installation (one-time, confirm with user before running)
pip install scrapling[all]
patchright install chromium # required for stealth/dynamic modes
scrapling[all]installspatchright(a stealth fork of Playwright, bundled as a PyPI package — not a typo),curl_cffi, MCP server deps, and IPython shell.patchright install chromiumdownloads Chromium (~100 MB) via patchright's own installer (same mechanism asplaywright install chromium).- Confirm with user before running — installs ~200 MB of dependencies and browser binaries.
Script
scripts/scrape.py — CLI wrapper for all three fetcher modes.
# Basic fetch (text output)
python3 ~/skills/scrapling/scripts/scrape.py <url> -q
# CSS selector extraction
python3 ~/skills/scrapling/scripts/scrape.py <url> --selector ".class" -q
# Stealth mode (Cloudflare bypass) — only on sites you're authorized to access
python3 ~/skills/scrapling/scripts/scrape.py <url> --mode stealth -q
# JSON output
python3 ~/skills/scrapling/scripts/scrape.py <url> --selector "h2" --json -q
Fetcher Modes
- http (default) — Fast HTTP with browser TLS fingerprint spoofing. Most sites.
- stealth — Headless Chrome with anti-detect. For Cloudflare/anti-bot.
- dynamic — Full Playwright browser. For heavy JS SPAs.
When to Use Each Mode
web_fetchreturns 403/429/Cloudflare challenge → use--mode stealth- Page content requires JS execution → use
--mode dynamic - Regular site, just need text/data → use
--mode http(default)
Python Inline Usage
For custom logic beyond the CLI, write inline Python. See references/patterns.md for:
- Adaptive scraping (
auto_save/adaptive— saves element fingerprints locally) - Session/cookie handling
- Async usage
- XPath, find_similar, attribute extraction
Notes
- MCP server (
scrapling mcp): starts a local network service for AI-native scraping. Only start if explicitly needed and trusted — it exposes a local HTTP server. auto_save=True: persists element fingerprints to disk for adaptive re-scraping. Creates local state in working directory.- Stealth/dynamic modes use Chromium headless — no
xvfb-runneeded. - For large-scale crawls, use the Spider API (see Scrapling docs).
04 同仓库更多 Skills
zvukogram
Text-to-Speech via Zvukogram API with SSML support. Use when you need to generate speech from text, create podcasts, voice notifications, or work with audio. Supports speed control, stress marks, English word transcription, and audio fragment merging.
zohomail
Zoho Mail API integration with managed OAuth. Send, receive, and manage emails, folders, and labels. Use this skill when users want to send emails, read messages, manage folders, or work with email labels in Zoho Mail. For other third party apps, use the api-gateway skill (https://clawhub.ai/byungkyu/api-gateway). Requires network access and valid Maton API key.
zoho-recruit
Zoho Recruit API integration with managed OAuth. Manage candidates, job openings, interviews, and recruitment workflows. Use this skill when users want to read, create, update, or search recruitment data like candidates, job openings, interviews, and applications in Zoho Recruit. For other third party apps, use the api-gateway skill (https://clawhub.ai/byungkyu/api-gateway). Requires network access and valid Maton API key.
zoho-people
Zoho People API integration with managed OAuth. Manage employees, departments, designations, attendance, and leave. Use this skill when users want to read, create, update, or query HR data like employees, departments, designations, and forms in Zoho People. For other third party apps, use the api-gateway skill (https://clawhub.ai/byungkyu/api-gateway). Requires network access and valid Maton API key.
zoho-mail
Zoho Mail API integration with managed OAuth. Send, receive, and manage emails, folders, and labels. Use this skill when users want to send emails, read messages, manage folders, or work with email labels in Zoho Mail. For other third party apps, use the api-gateway skill (https://clawhub.ai/byungkyu/api-gateway). Requires network access and valid Maton API key.
zoho-inventory
Zoho Inventory API integration with managed OAuth. Manage items, sales orders, invoices, purchase orders, bills, contacts, and shipments. Use this skill when users want to read, create, update, or delete inventory items, sales orders, invoices, purchase orders, bills, or other inventory records in Zoho Inventory. For other third party apps, use the api-gateway skill (https://clawhub.ai/byungkyu/api-gateway). Requires network access and valid Maton API key.