[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6F0J4F8AMFDSKZTG7ZK14MC":3,"skill-same:01M6F0J4F8AMFDSKZTG7ZK14MC":135},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":23,"publishedAt":43,"linkOnly":44,"markdownContent":45,"markdownBody":46,"frontMatter":47,"refName":48,"commitSha":49,"skillDirectory":50,"forksCount":51,"repositoryHtmlUrl":52,"repositoryHomepageUrl":53,"ownerHtmlUrl":54,"ownerBlogUrl":6,"ownerDescription":55,"updatedAt":56,"install":57,"resources":70},"01M6F0J4F8AMFDSKZTG7ZK14MC","crawl4ai",null,"This skill should be used when users need to scrape websites, extract structured data, handle JavaScript-heavy pages, crawl multiple URLs, or build automated web data pipelines. Includes optimized extraction patterns with schema generation for efficient, LLM-free extraction.","ComeOnOliver\u002Fskillshub","ComeOnOliver","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F43215937?v=4","01M6E5JNFECJAAVFE0W7SMWWD2","Haoran Shu","01M6E5JNNDQS7AJZDYFQ8TPZCK",67,"MIT","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Faiskillstore\u002Fmarketplace\u002Fsmallnest\u002Fcrawl4ai\u002FSKILL.md",[18],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",[5,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42],"build","with","users","free","pages","generation","structured","crawl","patterns","structured data","pipelines","schema","extract","scrape","data","extraction","data pipelines","web","javascript","2026-11-10 10:12:30",false,"---\nname: crawl4ai\ndescription: This skill should be used when users need to scrape websites, extract structured data, handle JavaScript-heavy pages, crawl multiple URLs, or build automated web data pipelines. Includes optimized extraction patterns with schema generation for efficient, LLM-free extraction.\n---\n\n# Crawl4AI\n\n## Overview\n\nCrawl4AI provides comprehensive web crawling and data extraction capabilities. This skill supports both **CLI** (recommended for quick tasks) and **Python SDK** (for programmatic control).\n\n**Choose your interface:**\n- **CLI** (`crwl`) - Quick, scriptable commands: [CLI Guide](references\u002Fcli-guide.md)\n- **Python SDK** - Full programmatic control: [SDK Guide](references\u002Fsdk-guide.md)\n\n---\n\n## Quick Start\n\n### Installation\n\n```bash\npip install crawl4ai\ncrawl4ai-setup\n\n# Verify installation\ncrawl4ai-doctor\n```\n\n### CLI (Recommended)\n\n```bash\n# Basic crawling - returns markdown\ncrwl https:\u002F\u002Fexample.com\n\n# Get markdown output\ncrwl https:\u002F\u002Fexample.com -o markdown\n\n# JSON output with cache bypass\ncrwl https:\u002F\u002Fexample.com -o json -v --bypass-cache\n\n# See more examples\ncrwl --example\n```\n\n### Python SDK\n\n```python\nimport asyncio\nfrom crawl4ai import AsyncWebCrawler\n\nasync def main():\n    async with AsyncWebCrawler() as crawler:\n        result = await crawler.arun(\"https:\u002F\u002Fexample.com\")\n        print(result.markdown[:500])\n\nasyncio.run(main())\n```\n\nFor SDK configuration details: [SDK Guide - Configuration](references\u002Fsdk-guide.md#configuration) (lines 61-150)\n\n---\n\n## Core Concepts\n\n### Configuration Layers\n\nBoth CLI and SDK use the same underlying configuration:\n\n| Concept | CLI | SDK |\n|---------|-----|-----|\n| Browser settings | `-B browser.yml` or `-b \"param=value\"` | `BrowserConfig(...)` |\n| Crawl settings | `-C crawler.yml` or `-c \"param=value\"` | `CrawlerRunConfig(...)` |\n| Extraction | `-e extract.yml -s schema.json` | `extraction_strategy=...` |\n| Content filter | `-f filter.yml` | `markdown_generator=...` |\n\n### Key Parameters\n\n**Browser Configuration:**\n- `headless`: Run with\u002Fwithout GUI\n- `viewport_width\u002Fheight`: Browser dimensions\n- `user_agent`: Custom user agent\n- `proxy_config`: Proxy settings\n\n**Crawler Configuration:**\n- `page_timeout`: Max page load time (ms)\n- `wait_for`: CSS selector or JS condition to wait for\n- `cache_mode`: bypass, enabled, disabled\n- `js_code`: JavaScript to execute\n- `css_selector`: Focus on specific element\n\nFor complete parameters: [CLI Config](references\u002Fcli-guide.md#configuration) | [SDK Config](references\u002Fsdk-guide.md#configuration)\n\n### Output Content\n\nEvery crawl returns:\n- **markdown** - Clean, formatted markdown\n- **html** - Raw HTML\n- **links** - Internal and external links discovered\n- **media** - Images, videos, audio found\n- **extracted_content** - Structured data (if extraction configured)\n\n---\n\n## Markdown Generation (Primary Use Case)\n\nCrawl4AI excels at generating clean, well-formatted markdown:\n\n### CLI\n\n```bash\n# Basic markdown\ncrwl https:\u002F\u002Fdocs.example.com -o markdown\n\n# Filtered markdown (removes noise)\ncrwl https:\u002F\u002Fdocs.example.com -o markdown-fit\n\n# With content filter\ncrwl https:\u002F\u002Fdocs.example.com -f filter_bm25.yml -o markdown-fit\n```\n\n**Filter configuration:**\n```yaml\n# filter_bm25.yml (relevance-based)\ntype: \"bm25\"\nquery: \"machine learning tutorials\"\nthreshold: 1.0\n```\n\n### Python SDK\n\n```python\nfrom crawl4ai.content_filter_strategy import BM25ContentFilter\nfrom crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator\n\nbm25_filter = BM25ContentFilter(user_query=\"machine learning\", bm25_threshold=1.0)\nmd_generator = DefaultMarkdownGenerator(content_filter=bm25_filter)\n\nconfig = CrawlerRunConfig(markdown_generator=md_generator)\nresult = await crawler.arun(url, config=config)\n\nprint(result.markdown.fit_markdown)  # Filtered\nprint(result.markdown.raw_markdown)  # Original\n```\n\nFor content filters: [Content Processing](references\u002Fcomplete-sdk-reference.md#content-processing) (lines 2481-3101)\n\n---\n\n## Data Extraction\n\n### 1. Schema-Based CSS Extraction (Most Efficient)\n\n**No LLM required** - fast, deterministic, cost-free.\n\n**CLI:**\n```bash\n# Generate schema once (uses LLM)\npython scripts\u002Fextraction_pipeline.py --generate-schema https:\u002F\u002Fshop.com \"extract products\"\n\n# Use schema for extraction (no LLM)\ncrwl https:\u002F\u002Fshop.com -e extract_css.yml -s product_schema.json -o json\n```\n\n**Schema format:**\n```json\n{\n  \"name\": \"products\",\n  \"baseSelector\": \".product-card\",\n  \"fields\": [\n    {\"name\": \"title\", \"selector\": \"h2\", \"type\": \"text\"},\n    {\"name\": \"price\", \"selector\": \".price\", \"type\": \"text\"},\n    {\"name\": \"link\", \"selector\": \"a\", \"type\": \"attribute\", \"attribute\": \"href\"}\n  ]\n}\n```\n\n### 2. LLM-Based Extraction\n\nFor complex or irregular content:\n\n**CLI:**\n```yaml\n# extract_llm.yml\ntype: \"llm\"\nprovider: \"openai\u002Fgpt-4o-mini\"\ninstruction: \"Extract product names and prices\"\napi_token: \"your-token\"\n```\n\n```bash\ncrwl https:\u002F\u002Fshop.com -e extract_llm.yml -o json\n```\n\nFor extraction details: [Extraction Strategies](references\u002Fcomplete-sdk-reference.md#extraction-strategies) (lines 4522-5429)\n\n---\n\n## Advanced Patterns\n\n### Dynamic Content (JavaScript-Heavy Sites)\n\n**CLI:**\n```bash\ncrwl https:\u002F\u002Fexample.com -c \"wait_for=css:.ajax-content,scan_full_page=true,page_timeout=60000\"\n```\n\n**Crawler config:**\n```yaml\n# crawler.yml\nwait_for: \"css:.ajax-content\"\nscan_full_page: true\npage_timeout: 60000\ndelay_before_return_html: 2.0\n```\n\n### Multi-URL Processing\n\n**CLI (sequential):**\n```bash\nfor url in url1 url2 url3; do crwl \"$url\" -o markdown; done\n```\n\n**Python SDK (concurrent):**\n```python\nurls = [\"https:\u002F\u002Fsite1.com\", \"https:\u002F\u002Fsite2.com\", \"https:\u002F\u002Fsite3.com\"]\nresults = await crawler.arun_many(urls, config=config)\n```\n\nFor batch processing: [arun_many() Reference](references\u002Fcomplete-sdk-reference.md#arunmany-reference) (lines 1057-1224)\n\n### Session & Authentication\n\n**CLI:**\n```yaml\n# login_crawler.yml\nsession_id: \"user_session\"\njs_code: |\n  document.querySelector('#username').value = 'user';\n  document.querySelector('#password').value = 'pass';\n  document.querySelector('#submit').click();\nwait_for: \"css:.dashboard\"\n```\n\n```bash\n# Login\ncrwl https:\u002F\u002Fsite.com\u002Flogin -C login_crawler.yml\n\n# Access protected content (session reused)\ncrwl https:\u002F\u002Fsite.com\u002Fprotected -c \"session_id=user_session\"\n```\n\nFor session management: [Advanced Features](references\u002Fcomplete-sdk-reference.md#advanced-features) (lines 5429-5940)\n\n### Anti-Detection & Proxies\n\n**CLI:**\n```yaml\n# browser.yml\nheadless: true\nproxy_config:\n  server: \"http:\u002F\u002Fproxy:8080\"\n  username: \"user\"\n  password: \"pass\"\nuser_agent_mode: \"random\"\n```\n\n```bash\ncrwl https:\u002F\u002Fexample.com -B browser.yml\n```\n\n---\n\n## Common Use Cases\n\n### Google Search Scraping\n\n```bash\n# Search Google and get results as JSON\npython scripts\u002Fgoogle_search.py \"your search query\" 20\n\n# Example\npython scripts\u002Fgoogle_search.py \"2026年Go语言展望\" 20\n```\n\nThe script extracts:\n- Search result titles\n- URLs (cleaned, removes Google redirects)\n- Descriptions\u002Fsnippets\n- Site names\n\nOutput is saved to `google_search_results.json` and printed to stdout.\n\n### Documentation to Markdown\n\n```bash\ncrwl https:\u002F\u002Fdocs.example.com -o markdown > docs.md\n```\n\n### E-commerce Product Monitoring\n\n```bash\n# Generate schema once\npython scripts\u002Fextraction_pipeline.py --generate-schema https:\u002F\u002Fshop.com \"extract products\"\n\n# Monitor (no LLM costs)\ncrwl https:\u002F\u002Fshop.com -e extract_css.yml -s schema.json -o json\n```\n\n### News Aggregation\n\n```bash\n# Multiple sources with filtering\nfor url in news1.com news2.com news3.com; do\n  crwl \"https:\u002F\u002F$url\" -f filter_bm25.yml -o markdown-fit\ndone\n```\n\n### Interactive Q&A\n\n```bash\n# First view content\ncrwl https:\u002F\u002Fexample.com -o markdown\n\n# Then ask questions\ncrwl https:\u002F\u002Fexample.com -q \"What are the main conclusions?\"\ncrwl https:\u002F\u002Fexample.com -q \"Summarize the key points\"\n```\n\n---\n\n## Resources\n\n### Provided Scripts\n\n- **scripts\u002Fgoogle_search.py** - Google search scraper with JSON output\n- **scripts\u002Fextraction_pipeline.py** - Schema generation and extraction\n- **scripts\u002Fbasic_crawler.py** - Simple markdown extraction\n- **scripts\u002Fbatch_crawler.py** - Multi-URL processing\n\n### Reference Documentation\n\n| Document | Purpose |\n|----------|---------|\n| [CLI Guide](references\u002Fcli-guide.md) | Command-line interface reference |\n| [SDK Guide](references\u002Fsdk-guide.md) | Python SDK quick reference |\n| [Complete SDK Reference](references\u002Fcomplete-sdk-reference.md) | Full API documentation (5900+ lines) |\n\n---\n\n## Best Practices\n\n1. **Start with CLI** for quick tasks, SDK for automation\n2. **Use schema-based extraction** - 10-100x more efficient than LLM\n3. **Enable caching during development** - `--bypass-cache` only when needed\n4. **Set appropriate timeouts** - 30s normal, 60s+ for JS-heavy sites\n5. **Use content filters** for cleaner, focused markdown\n6. **Respect rate limits** - Add delays between requests\n\n---\n\n## Troubleshooting\n\n### JavaScript Not Loading\n\n```bash\ncrwl https:\u002F\u002Fexample.com -c \"wait_for=css:.dynamic-content,page_timeout=60000\"\n```\n\n### Bot Detection Issues\n\n```bash\ncrwl https:\u002F\u002Fexample.com -B browser.yml\n```\n\n```yaml\n# browser.yml\nheadless: false\nviewport_width: 1920\nviewport_height: 1080\nuser_agent: \"Mozilla\u002F5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\u002F537.36\"\n```\n\n### Content Not Extracted\n\n```bash\n# Debug: see full output\ncrwl https:\u002F\u002Fexample.com -o all -v\n\n# Try different wait strategy\ncrwl https:\u002F\u002Fexample.com -c \"wait_for=js:document.querySelector('.content')!==null\"\n```\n\n### Session Issues\n\n```bash\n# Verify session\ncrwl https:\u002F\u002Fsite.com -c \"session_id=test\" -o all | grep -i session\n```\n\n---\n\nFor comprehensive API documentation, see [Complete SDK Reference](references\u002Fcomplete-sdk-reference.md).\n","# Crawl4AI\n\n## Overview\n\nCrawl4AI provides comprehensive web crawling and data extraction capabilities. This skill supports both **CLI** (recommended for quick tasks) and **Python SDK** (for programmatic control).\n\n**Choose your interface:**\n- **CLI** (`crwl`) - Quick, scriptable commands: [CLI Guide](references\u002Fcli-guide.md)\n- **Python SDK** - Full programmatic control: [SDK Guide](references\u002Fsdk-guide.md)\n\n---\n\n## Quick Start\n\n### Installation\n\n```bash\npip install crawl4ai\ncrawl4ai-setup\n\n# Verify installation\ncrawl4ai-doctor\n```\n\n### CLI (Recommended)\n\n```bash\n# Basic crawling - returns markdown\ncrwl https:\u002F\u002Fexample.com\n\n# Get markdown output\ncrwl https:\u002F\u002Fexample.com -o markdown\n\n# JSON output with cache bypass\ncrwl https:\u002F\u002Fexample.com -o json -v --bypass-cache\n\n# See more examples\ncrwl --example\n```\n\n### Python SDK\n\n```python\nimport asyncio\nfrom crawl4ai import AsyncWebCrawler\n\nasync def main():\n    async with AsyncWebCrawler() as crawler:\n        result = await crawler.arun(\"https:\u002F\u002Fexample.com\")\n        print(result.markdown[:500])\n\nasyncio.run(main())\n```\n\nFor SDK configuration details: [SDK Guide - Configuration](references\u002Fsdk-guide.md#configuration) (lines 61-150)\n\n---\n\n## Core Concepts\n\n### Configuration Layers\n\nBoth CLI and SDK use the same underlying configuration:\n\n| Concept | CLI | SDK |\n|---------|-----|-----|\n| Browser settings | `-B browser.yml` or `-b \"param=value\"` | `BrowserConfig(...)` |\n| Crawl settings | `-C crawler.yml` or `-c \"param=value\"` | `CrawlerRunConfig(...)` |\n| Extraction | `-e extract.yml -s schema.json` | `extraction_strategy=...` |\n| Content filter | `-f filter.yml` | `markdown_generator=...` |\n\n### Key Parameters\n\n**Browser Configuration:**\n- `headless`: Run with\u002Fwithout GUI\n- `viewport_width\u002Fheight`: Browser dimensions\n- `user_agent`: Custom user agent\n- `proxy_config`: Proxy settings\n\n**Crawler Configuration:**\n- `page_timeout`: Max page load time (ms)\n- `wait_for`: CSS selector or JS condition to wait for\n- `cache_mode`: bypass, enabled, disabled\n- `js_code`: JavaScript to execute\n- `css_selector`: Focus on specific element\n\nFor complete parameters: [CLI Config](references\u002Fcli-guide.md#configuration) | [SDK Config](references\u002Fsdk-guide.md#configuration)\n\n### Output Content\n\nEvery crawl returns:\n- **markdown** - Clean, formatted markdown\n- **html** - Raw HTML\n- **links** - Internal and external links discovered\n- **media** - Images, videos, audio found\n- **extracted_content** - Structured data (if extraction configured)\n\n---\n\n## Markdown Generation (Primary Use Case)\n\nCrawl4AI excels at generating clean, well-formatted markdown:\n\n### CLI\n\n```bash\n# Basic markdown\ncrwl https:\u002F\u002Fdocs.example.com -o markdown\n\n# Filtered markdown (removes noise)\ncrwl https:\u002F\u002Fdocs.example.com -o markdown-fit\n\n# With content filter\ncrwl https:\u002F\u002Fdocs.example.com -f filter_bm25.yml -o markdown-fit\n```\n\n**Filter configuration:**\n```yaml\n# filter_bm25.yml (relevance-based)\ntype: \"bm25\"\nquery: \"machine learning tutorials\"\nthreshold: 1.0\n```\n\n### Python SDK\n\n```python\nfrom crawl4ai.content_filter_strategy import BM25ContentFilter\nfrom crawl4ai.markdown_generation_strategy import DefaultMarkdownGenerator\n\nbm25_filter = BM25ContentFilter(user_query=\"machine learning\", bm25_threshold=1.0)\nmd_generator = DefaultMarkdownGenerator(content_filter=bm25_filter)\n\nconfig = CrawlerRunConfig(markdown_generator=md_generator)\nresult = await crawler.arun(url, config=config)\n\nprint(result.markdown.fit_markdown)  # Filtered\nprint(result.markdown.raw_markdown)  # Original\n```\n\nFor content filters: [Content Processing](references\u002Fcomplete-sdk-reference.md#content-processing) (lines 2481-3101)\n\n---\n\n## Data Extraction\n\n### 1. Schema-Based CSS Extraction (Most Efficient)\n\n**No LLM required** - fast, deterministic, cost-free.\n\n**CLI:**\n```bash\n# Generate schema once (uses LLM)\npython scripts\u002Fextraction_pipeline.py --generate-schema https:\u002F\u002Fshop.com \"extract products\"\n\n# Use schema for extraction (no LLM)\ncrwl https:\u002F\u002Fshop.com -e extract_css.yml -s product_schema.json -o json\n```\n\n**Schema format:**\n```json\n{\n  \"name\": \"products\",\n  \"baseSelector\": \".product-card\",\n  \"fields\": [\n    {\"name\": \"title\", \"selector\": \"h2\", \"type\": \"text\"},\n    {\"name\": \"price\", \"selector\": \".price\", \"type\": \"text\"},\n    {\"name\": \"link\", \"selector\": \"a\", \"type\": \"attribute\", \"attribute\": \"href\"}\n  ]\n}\n```\n\n### 2. LLM-Based Extraction\n\nFor complex or irregular content:\n\n**CLI:**\n```yaml\n# extract_llm.yml\ntype: \"llm\"\nprovider: \"openai\u002Fgpt-4o-mini\"\ninstruction: \"Extract product names and prices\"\napi_token: \"your-token\"\n```\n\n```bash\ncrwl https:\u002F\u002Fshop.com -e extract_llm.yml -o json\n```\n\nFor extraction details: [Extraction Strategies](references\u002Fcomplete-sdk-reference.md#extraction-strategies) (lines 4522-5429)\n\n---\n\n## Advanced Patterns\n\n### Dynamic Content (JavaScript-Heavy Sites)\n\n**CLI:**\n```bash\ncrwl https:\u002F\u002Fexample.com -c \"wait_for=css:.ajax-content,scan_full_page=true,page_timeout=60000\"\n```\n\n**Crawler config:**\n```yaml\n# crawler.yml\nwait_for: \"css:.ajax-content\"\nscan_full_page: true\npage_timeout: 60000\ndelay_before_return_html: 2.0\n```\n\n### Multi-URL Processing\n\n**CLI (sequential):**\n```bash\nfor url in url1 url2 url3; do crwl \"$url\" -o markdown; done\n```\n\n**Python SDK (concurrent):**\n```python\nurls = [\"https:\u002F\u002Fsite1.com\", \"https:\u002F\u002Fsite2.com\", \"https:\u002F\u002Fsite3.com\"]\nresults = await crawler.arun_many(urls, config=config)\n```\n\nFor batch processing: [arun_many() Reference](references\u002Fcomplete-sdk-reference.md#arunmany-reference) (lines 1057-1224)\n\n### Session & Authentication\n\n**CLI:**\n```yaml\n# login_crawler.yml\nsession_id: \"user_session\"\njs_code: |\n  document.querySelector('#username').value = 'user';\n  document.querySelector('#password').value = 'pass';\n  document.querySelector('#submit').click();\nwait_for: \"css:.dashboard\"\n```\n\n```bash\n# Login\ncrwl https:\u002F\u002Fsite.com\u002Flogin -C login_crawler.yml\n\n# Access protected content (session reused)\ncrwl https:\u002F\u002Fsite.com\u002Fprotected -c \"session_id=user_session\"\n```\n\nFor session management: [Advanced Features](references\u002Fcomplete-sdk-reference.md#advanced-features) (lines 5429-5940)\n\n### Anti-Detection & Proxies\n\n**CLI:**\n```yaml\n# browser.yml\nheadless: true\nproxy_config:\n  server: \"http:\u002F\u002Fproxy:8080\"\n  username: \"user\"\n  password: \"pass\"\nuser_agent_mode: \"random\"\n```\n\n```bash\ncrwl https:\u002F\u002Fexample.com -B browser.yml\n```\n\n---\n\n## Common Use Cases\n\n### Google Search Scraping\n\n```bash\n# Search Google and get results as JSON\npython scripts\u002Fgoogle_search.py \"your search query\" 20\n\n# Example\npython scripts\u002Fgoogle_search.py \"2026年Go语言展望\" 20\n```\n\nThe script extracts:\n- Search result titles\n- URLs (cleaned, removes Google redirects)\n- Descriptions\u002Fsnippets\n- Site names\n\nOutput is saved to `google_search_results.json` and printed to stdout.\n\n### Documentation to Markdown\n\n```bash\ncrwl https:\u002F\u002Fdocs.example.com -o markdown > docs.md\n```\n\n### E-commerce Product Monitoring\n\n```bash\n# Generate schema once\npython scripts\u002Fextraction_pipeline.py --generate-schema https:\u002F\u002Fshop.com \"extract products\"\n\n# Monitor (no LLM costs)\ncrwl https:\u002F\u002Fshop.com -e extract_css.yml -s schema.json -o json\n```\n\n### News Aggregation\n\n```bash\n# Multiple sources with filtering\nfor url in news1.com news2.com news3.com; do\n  crwl \"https:\u002F\u002F$url\" -f filter_bm25.yml -o markdown-fit\ndone\n```\n\n### Interactive Q&A\n\n```bash\n# First view content\ncrwl https:\u002F\u002Fexample.com -o markdown\n\n# Then ask questions\ncrwl https:\u002F\u002Fexample.com -q \"What are the main conclusions?\"\ncrwl https:\u002F\u002Fexample.com -q \"Summarize the key points\"\n```\n\n---\n\n## Resources\n\n### Provided Scripts\n\n- **scripts\u002Fgoogle_search.py** - Google search scraper with JSON output\n- **scripts\u002Fextraction_pipeline.py** - Schema generation and extraction\n- **scripts\u002Fbasic_crawler.py** - Simple markdown extraction\n- **scripts\u002Fbatch_crawler.py** - Multi-URL processing\n\n### Reference Documentation\n\n| Document | Purpose |\n|----------|---------|\n| [CLI Guide](references\u002Fcli-guide.md) | Command-line interface reference |\n| [SDK Guide](references\u002Fsdk-guide.md) | Python SDK quick reference |\n| [Complete SDK Reference](references\u002Fcomplete-sdk-reference.md) | Full API documentation (5900+ lines) |\n\n---\n\n## Best Practices\n\n1. **Start with CLI** for quick tasks, SDK for automation\n2. **Use schema-based extraction** - 10-100x more efficient than LLM\n3. **Enable caching during development** - `--bypass-cache` only when needed\n4. **Set appropriate timeouts** - 30s normal, 60s+ for JS-heavy sites\n5. **Use content filters** for cleaner, focused markdown\n6. **Respect rate limits** - Add delays between requests\n\n---\n\n## Troubleshooting\n\n### JavaScript Not Loading\n\n```bash\ncrwl https:\u002F\u002Fexample.com -c \"wait_for=css:.dynamic-content,page_timeout=60000\"\n```\n\n### Bot Detection Issues\n\n```bash\ncrwl https:\u002F\u002Fexample.com -B browser.yml\n```\n\n```yaml\n# browser.yml\nheadless: false\nviewport_width: 1920\nviewport_height: 1080\nuser_agent: \"Mozilla\u002F5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\u002F537.36\"\n```\n\n### Content Not Extracted\n\n```bash\n# Debug: see full output\ncrwl https:\u002F\u002Fexample.com -o all -v\n\n# Try different wait strategy\ncrwl https:\u002F\u002Fexample.com -c \"wait_for=js:document.querySelector('.content')!==null\"\n```\n\n### Session Issues\n\n```bash\n# Verify session\ncrwl https:\u002F\u002Fsite.com -c \"session_id=test\" -o all | grep -i session\n```\n\n---\n\nFor comprehensive API documentation, see [Complete SDK Reference](references\u002Fcomplete-sdk-reference.md).\n",{"name":5,"description":7},"main","def8531e65114c0fca8fb8551c1871ee0eed705c","skills\u002Faiskillstore\u002Fmarketplace\u002Fsmallnest\u002Fcrawl4ai",24,"https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub","https:\u002F\u002Fskillshub.wtf","https:\u002F\u002Fgithub.com\u002FComeOnOliver","AI & Blockchain Engineer\r\n","2026-06-24 22:07:19",{"source":8,"skillName":5,"commands":58,"prompt":68,"zipAvailable":69},[59,62,65],{"tool":60,"command":61},"npx","npx skills add ComeOnOliver\u002Fskillshub --skill crawl4ai",{"tool":63,"command":64},"bunx","bunx skills add ComeOnOliver\u002Fskillshub --skill crawl4ai",{"tool":66,"command":67},"pnpm","pnpm dlx skills add ComeOnOliver\u002Fskillshub --skill crawl4ai","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Faiskillstore\u002Fmarketplace\u002Fsmallnest\u002Fcrawl4ai\u002FSKILL.md",true,[71,77,81,85,89,95,99,103,107,111,115,119,123,127,131],{"path":72,"resourceType":73,"resourceTypeDesc":74,"sizeBytes":75,"sourceHtmlUrl":76},"README.md","REFERENCE","参考文档",7351,"https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Faiskillstore\u002Fmarketplace\u002Fsmallnest\u002Fcrawl4ai\u002FREADME.md",{"path":78,"resourceType":73,"resourceTypeDesc":74,"sizeBytes":79,"sourceHtmlUrl":80},"references\u002Fcli-guide.md",8130,"https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Faiskillstore\u002Fmarketplace\u002Fsmallnest\u002Fcrawl4ai\u002Freferences\u002Fcli-guide.md",{"path":82,"resourceType":73,"resourceTypeDesc":74,"sizeBytes":83,"sourceHtmlUrl":84},"references\u002Fcomplete-sdk-reference.md",224947,"https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Faiskillstore\u002Fmarketplace\u002Fsmallnest\u002Fcrawl4ai\u002Freferences\u002Fcomplete-sdk-reference.md",{"path":86,"resourceType":73,"resourceTypeDesc":74,"sizeBytes":87,"sourceHtmlUrl":88},"references\u002Fsdk-guide.md",10033,"https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Faiskillstore\u002Fmarketplace\u002Fsmallnest\u002Fcrawl4ai\u002Freferences\u002Fsdk-guide.md",{"path":90,"resourceType":91,"resourceTypeDesc":92,"sizeBytes":93,"sourceHtmlUrl":94},"scripts\u002Fbasic_crawler.py","SCRIPT","脚本",2660,"https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Faiskillstore\u002Fmarketplace\u002Fsmallnest\u002Fcrawl4ai\u002Fscripts\u002Fbasic_crawler.py",{"path":96,"resourceType":91,"resourceTypeDesc":92,"sizeBytes":97,"sourceHtmlUrl":98},"scripts\u002Fbatch_crawler.py",7674,"https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Faiskillstore\u002Fmarketplace\u002Fsmallnest\u002Fcrawl4ai\u002Fscripts\u002Fbatch_crawler.py",{"path":100,"resourceType":91,"resourceTypeDesc":92,"sizeBytes":101,"sourceHtmlUrl":102},"scripts\u002Fextraction_pipeline.py",13312,"https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Faiskillstore\u002Fmarketplace\u002Fsmallnest\u002Fcrawl4ai\u002Fscripts\u002Fextraction_pipeline.py",{"path":104,"resourceType":91,"resourceTypeDesc":92,"sizeBytes":105,"sourceHtmlUrl":106},"scripts\u002Fgoogle_search.py",11770,"https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Faiskillstore\u002Fmarketplace\u002Fsmallnest\u002Fcrawl4ai\u002Fscripts\u002Fgoogle_search.py",{"path":108,"resourceType":73,"resourceTypeDesc":74,"sizeBytes":109,"sourceHtmlUrl":110},"skill-report.json",10816,"https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Faiskillstore\u002Fmarketplace\u002Fsmallnest\u002Fcrawl4ai\u002Fskill-report.json",{"path":112,"resourceType":73,"resourceTypeDesc":74,"sizeBytes":113,"sourceHtmlUrl":114},"tests\u002FREADME.md",1400,"https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Faiskillstore\u002Fmarketplace\u002Fsmallnest\u002Fcrawl4ai\u002Ftests\u002FREADME.md",{"path":116,"resourceType":91,"resourceTypeDesc":92,"sizeBytes":117,"sourceHtmlUrl":118},"tests\u002Frun_all_tests.py",1415,"https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Faiskillstore\u002Fmarketplace\u002Fsmallnest\u002Fcrawl4ai\u002Ftests\u002Frun_all_tests.py",{"path":120,"resourceType":91,"resourceTypeDesc":92,"sizeBytes":121,"sourceHtmlUrl":122},"tests\u002Ftest_advanced_patterns.py",2233,"https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Faiskillstore\u002Fmarketplace\u002Fsmallnest\u002Fcrawl4ai\u002Ftests\u002Ftest_advanced_patterns.py",{"path":124,"resourceType":91,"resourceTypeDesc":92,"sizeBytes":125,"sourceHtmlUrl":126},"tests\u002Ftest_basic_crawling.py",1594,"https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Faiskillstore\u002Fmarketplace\u002Fsmallnest\u002Fcrawl4ai\u002Ftests\u002Ftest_basic_crawling.py",{"path":128,"resourceType":91,"resourceTypeDesc":92,"sizeBytes":129,"sourceHtmlUrl":130},"tests\u002Ftest_data_extraction.py",2189,"https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Faiskillstore\u002Fmarketplace\u002Fsmallnest\u002Fcrawl4ai\u002Ftests\u002Ftest_data_extraction.py",{"path":132,"resourceType":91,"resourceTypeDesc":92,"sizeBytes":133,"sourceHtmlUrl":134},"tests\u002Ftest_markdown_generation.py",3123,"https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Faiskillstore\u002Fmarketplace\u002Fsmallnest\u002Fcrawl4ai\u002Ftests\u002Ftest_markdown_generation.py",[136,146,156,165,171,178],{"skillNo":137,"name":138,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":139,"categories":140,"keywords":142,"publishedAt":43},"01M6FKJN13GMG4KY57J3QD8GY5","model-hierarchy-skill","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzscole\u002Fmodel-hierarchy-skill\u002Fmodel-hierarchy-skill\u002FSKILL.md",[141],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},[143,144,145],"model","model hierarchy","model hierarchy skill",{"skillNo":147,"name":148,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":149,"categories":150,"keywords":152,"publishedAt":43},"01M6FKJJJ6GCAMR4PPPRD12M5X","zoom-mcp","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fzoom-mcp\u002FSKILL.md",[151],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},[153,154,155],"MCP","zoom","zoom mcp",{"skillNo":157,"name":158,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":159,"categories":160,"keywords":161,"publishedAt":43},"01M6FKJE3KYT3KPEZMEEHYKA6K","zoom-apps-sdk","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fzoom-apps-sdk\u002FSKILL.md",[],[162,163,164,154],"sdk","zoom apps sdk","apps",{"skillNo":166,"name":167,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":168,"categories":169,"keywords":170,"publishedAt":43},"01M6FKJ9EDH9Q5VWDF8HAJRSP0","windows","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fwindows\u002FSKILL.md",[],[167],{"skillNo":172,"name":173,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":174,"categories":175,"keywords":177,"publishedAt":43},"01M6FKJ666BEEFC3AE9GJ72Q93","whiteboard","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fwhiteboard\u002FSKILL.md",[176],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},[173],{"skillNo":179,"name":180,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":181,"categories":182,"keywords":183,"publishedAt":43},"01M6FKJ21WFCDGTJQ2Q91T9DVW","websockets","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fwebsockets\u002FSKILL.md",[],[180]]