[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6JMG4PB8XREG91Q293MEFT4":3,"skill-same:01M6JMG4PB8XREG91Q293MEFT4":66},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":44,"commitSha":45,"skillDirectory":46,"forksCount":47,"repositoryHtmlUrl":48,"repositoryHomepageUrl":6,"ownerHtmlUrl":49,"ownerBlogUrl":6,"ownerDescription":50,"updatedAt":51,"install":52,"resources":65},"01M6JMG4PB8XREG91Q293MEFT4","cf-crawl",null,"Crawl entire websites using Cloudflare Browser Rendering \u002Fcrawl API. Initiates async crawl jobs, polls for completion, and saves results as markdown files. Useful for ingesting documentation sites, knowledge bases, or any web content into your project context. Requires CLOUDFLARE_ACCOUNT_ID and CLOUDFLARE_API_TOKEN environment variables.","transaurus\u002Fstaging-davila7-claude-code-templates","transaurus","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F210882017?v=4","01M6JH1R6ZCRXGJCTDP5ZKZBE6","Page Turner Agent","01M6JH1RBBR2P6Y0XWDGM4B0G7",0,"MIT","https:\u002F\u002Fgithub.com\u002Ftransaurus\u002Fstaging-davila7-claude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Futilities\u002Fcf-crawl\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"cf crawl","crawl","for","documentation","cloudflare browser rendering","Variables","web content","rendering","environment variables","as","token","project","files","environment","context","knowledge","markdown","api","cloudflare","browser","2026-11-10 10:12:30",false,"---\nname: cf-crawl\ndescription: \"Crawl entire websites using Cloudflare Browser Rendering \u002Fcrawl API. Initiates async crawl jobs, polls for completion, and saves results as markdown files. Useful for ingesting documentation sites, knowledge bases, or any web content into your project context. Requires CLOUDFLARE_ACCOUNT_ID and CLOUDFLARE_API_TOKEN environment variables.\"\n---\n\n# Cloudflare Website Crawler\n\nYou are a web crawling assistant that uses Cloudflare's Browser Rendering \u002Fcrawl REST API to crawl websites and save their content as markdown files for local use.\n\n## Prerequisites\n\nThe user must have:\n1. A Cloudflare account with Browser Rendering enabled\n2. `CLOUDFLARE_ACCOUNT_ID` and `CLOUDFLARE_API_TOKEN` available (see below)\n\n## Workflow\n\nWhen the user asks to crawl a website, follow this exact workflow:\n\n### Step 1: Load Credentials\n\nLook for `CLOUDFLARE_ACCOUNT_ID` and `CLOUDFLARE_API_TOKEN` in this order:\n\n1. **Current environment variables** - Check if already exported in the shell\n2. **Project `.env` file** - Read `.env` in the current working directory and extract the values\n3. **Project `.env.local` file** - Read `.env.local` in the current working directory\n4. **Home directory `.env`** - Read `~\u002F.env` as a last resort\n\nTo load from a `.env` file, parse it line by line looking for `CLOUDFLARE_ACCOUNT_ID=` and `CLOUDFLARE_API_TOKEN=` entries. Use this bash approach:\n\n```bash\n# Load from .env if vars are not already set\nif [ -z \"$CLOUDFLARE_ACCOUNT_ID\" ] || [ -z \"$CLOUDFLARE_API_TOKEN\" ]; then\n  for envfile in .env .env.local \"$HOME\u002F.env\"; do\n    if [ -f \"$envfile\" ]; then\n      eval \"$(grep -E '^CLOUDFLARE_(ACCOUNT_ID|API_TOKEN)=' \"$envfile\" | sed 's\u002F^\u002Fexport \u002F')\"\n    fi\n  done\nfi\n```\n\nIf credentials are still missing after checking all sources, tell the user to add them to their project `.env` file:\n```\nCLOUDFLARE_ACCOUNT_ID=your-account-id\nCLOUDFLARE_API_TOKEN=your-api-token\n```\n\nThe API token needs \"Browser Rendering - Edit\" permission. Create one at [Cloudflare Dashboard > API Tokens](https:\u002F\u002Fdash.cloudflare.com\u002Fprofile\u002Fapi-tokens).\n\n### Step 2: Validate Credentials\n\nVerify both variables are set and non-empty before proceeding.\n\n### Step 3: Initiate Crawl\n\nSend a POST request to start the crawl job. Choose parameters based on user needs:\n\n```bash\ncurl -s -X POST \"https:\u002F\u002Fapi.cloudflare.com\u002Fclient\u002Fv4\u002Faccounts\u002F${CLOUDFLARE_ACCOUNT_ID}\u002Fbrowser-rendering\u002Fcrawl\" \\\n  -H \"Authorization: Bearer ${CLOUDFLARE_API_TOKEN}\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"url\": \"\u003CTARGET_URL>\",\n    \"limit\": \u003CNUMBER_OF_PAGES>,\n    \"formats\": [\"markdown\"],\n    \"options\": {\n      \"excludePatterns\": [\"**\u002Fchangelog\u002F**\", \"**\u002Fapi-reference\u002F**\"]\n    }\n  }'\n```\n\nFor incremental crawls, add the `modifiedSince` parameter (Unix timestamp in seconds):\n\n```bash\ncurl -s -X POST \"https:\u002F\u002Fapi.cloudflare.com\u002Fclient\u002Fv4\u002Faccounts\u002F${CLOUDFLARE_ACCOUNT_ID}\u002Fbrowser-rendering\u002Fcrawl\" \\\n  -H \"Authorization: Bearer ${CLOUDFLARE_API_TOKEN}\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"url\": \"\u003CTARGET_URL>\",\n    \"limit\": \u003CNUMBER_OF_PAGES>,\n    \"formats\": [\"markdown\"],\n    \"modifiedSince\": \u003CUNIX_TIMESTAMP>\n  }'\n```\n\nWhen `--since` is provided, convert to Unix timestamp: `date -d \"2026-03-10\" +%s` (Linux) or `date -j -f \"%Y-%m-%d\" \"2026-03-10\" +%s` (macOS).\n\nThe response returns a job ID:\n```json\n{\"success\": true, \"result\": \"job-uuid-here\"}\n```\n\n### Step 4: Poll for Completion\n\nPoll the job status every 5 seconds until it completes:\n\n```bash\ncurl -s -X GET \"https:\u002F\u002Fapi.cloudflare.com\u002Fclient\u002Fv4\u002Faccounts\u002F${CLOUDFLARE_ACCOUNT_ID}\u002Fbrowser-rendering\u002Fcrawl\u002F\u003CJOB_ID>?limit=1\" \\\n  -H \"Authorization: Bearer ${CLOUDFLARE_API_TOKEN}\" | python3 -c \"import sys,json; d=json.load(sys.stdin); print(f'Status: {d[\\\"result\\\"][\\\"status\\\"]} | Finished: {d[\\\"result\\\"][\\\"finished\\\"]}\u002F{d[\\\"result\\\"][\\\"total\\\"]}')\"\n```\n\nPossible job statuses:\n- `running` - Still in progress, keep polling\n- `completed` - All pages processed\n- `cancelled_due_to_timeout` - Exceeded 7-day limit\n- `cancelled_due_to_limits` - Hit account limits\n- `errored` - Something went wrong\n\n### Step 5: Retrieve Results\n\nWhen using `modifiedSince`, check for skipped pages to see what was unchanged:\n\n```bash\n# See which pages were skipped (not modified since the given timestamp)\ncurl -s -X GET \"https:\u002F\u002Fapi.cloudflare.com\u002Fclient\u002Fv4\u002Faccounts\u002F${CLOUDFLARE_ACCOUNT_ID}\u002Fbrowser-rendering\u002Fcrawl\u002F\u003CJOB_ID>?status=skipped&limit=50\" \\\n  -H \"Authorization: Bearer ${CLOUDFLARE_API_TOKEN}\"\n```\n\nFetch all completed records using pagination (cursor-based):\n\n```bash\ncurl -s -X GET \"https:\u002F\u002Fapi.cloudflare.com\u002Fclient\u002Fv4\u002Faccounts\u002F${CLOUDFLARE_ACCOUNT_ID}\u002Fbrowser-rendering\u002Fcrawl\u002F\u003CJOB_ID>?status=completed&limit=50\" \\\n  -H \"Authorization: Bearer ${CLOUDFLARE_API_TOKEN}\"\n```\n\nIf there are more records, use the `cursor` value from the response:\n```bash\ncurl -s -X GET \"https:\u002F\u002Fapi.cloudflare.com\u002Fclient\u002Fv4\u002Faccounts\u002F${CLOUDFLARE_ACCOUNT_ID}\u002Fbrowser-rendering\u002Fcrawl\u002F\u003CJOB_ID>?status=completed&limit=50&cursor=\u003CCURSOR>\" \\\n  -H \"Authorization: Bearer ${CLOUDFLARE_API_TOKEN}\"\n```\n\n### Step 6: Save Results\n\nSave each page's markdown content to a local directory. Use a script like:\n\n```bash\n# Create output directory\nmkdir -p .crawl-output\n\n# Fetch and save all pages\npython3 -c \"\nimport json, os, re, sys, urllib.request\n\naccount_id = os.environ['CLOUDFLARE_ACCOUNT_ID']\napi_token = os.environ['CLOUDFLARE_API_TOKEN']\njob_id = '\u003CJOB_ID>'\nbase = f'https:\u002F\u002Fapi.cloudflare.com\u002Fclient\u002Fv4\u002Faccounts\u002F{account_id}\u002Fbrowser-rendering\u002Fcrawl\u002F{job_id}'\noutdir = '.crawl-output'\nos.makedirs(outdir, exist_ok=True)\n\ncursor = None\ntotal_saved = 0\n\nwhile True:\n    url = f'{base}?status=completed&limit=50'\n    if cursor:\n        url += f'&cursor={cursor}'\n\n    req = urllib.request.Request(url, headers={\n        'Authorization': f'Bearer {api_token}'\n    })\n    with urllib.request.urlopen(req) as resp:\n        data = json.load(resp)\n\n    records = data.get('result', {}).get('records', [])\n    if not records:\n        break\n\n    for rec in records:\n        page_url = rec.get('url', '')\n        md = rec.get('markdown', '')\n        if not md:\n            continue\n        # Convert URL to filename\n        name = re.sub(r'https?:\u002F\u002F', '', page_url)\n        name = re.sub(r'[^a-zA-Z0-9]', '_', name).strip('_')[:120]\n        filepath = os.path.join(outdir, f'{name}.md')\n        with open(filepath, 'w') as f:\n            f.write(f'\u003C!-- Source: {page_url} -->\\n\\n')\n            f.write(md)\n        total_saved += 1\n\n    cursor = data.get('result', {}).get('cursor')\n    if cursor is None:\n        break\n\nprint(f'Saved {total_saved} pages to {outdir}\u002F')\n\"\n```\n\n## Parameter Reference\n\n### Core Parameters\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `url` | string | (required) | Starting URL to crawl |\n| `limit` | number | 10 | Max pages to crawl (up to 100,000) |\n| `depth` | number | 100,000 | Max link depth from starting URL |\n| `formats` | array | [\"html\"] | Output formats: `html`, `markdown`, `json` |\n| `render` | boolean | true | `true` = headless browser, `false` = fast HTML fetch |\n| `source` | string | \"all\" | Page discovery: `all`, `sitemaps`, `links` |\n| `maxAge` | number | 86400 | Cache validity in seconds (max 604800) |\n| `modifiedSince` | number | - | Unix timestamp; only crawl pages modified after this time |\n\n### Options Object\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `includePatterns` | array | [] | Wildcard patterns to include (`*` and `**`) |\n| `excludePatterns` | array | [] | Wildcard patterns to exclude (higher priority) |\n| `includeSubdomains` | boolean | false | Follow links to subdomains |\n| `includeExternalLinks` | boolean | false | Follow external links |\n\n### Advanced Parameters\n\n| Parameter | Type | Description |\n|-----------|------|-------------|\n| `jsonOptions` | object | AI-powered structured extraction (prompt, response_format) |\n| `authenticate` | object | HTTP basic auth (username, password) |\n| `setExtraHTTPHeaders` | object | Custom headers for requests |\n| `rejectResourceTypes` | array | Skip: image, media, font, stylesheet |\n| `userAgent` | string | Custom user agent string |\n| `cookies` | array | Custom cookies for requests |\n\n## Usage Examples\n\n### Crawl documentation site (most common)\n```\n\u002Fcf-crawl https:\u002F\u002Fdocs.example.com --limit 50\n```\nCrawls up to 50 pages, saves as markdown.\n\n### Crawl with filters\n```\n\u002Fcf-crawl https:\u002F\u002Fdocs.example.com --limit 100 --include \"\u002Fguides\u002F**,\u002Fapi\u002F**\" --exclude \"\u002Fchangelog\u002F**\"\n```\n\n### Incremental crawl (diff detection)\n```\n\u002Fcf-crawl https:\u002F\u002Fdocs.example.com --limit 50 --since 2026-03-10\n```\nOnly crawls pages modified since the given date. Skipped pages appear with `status=skipped` in results. This is ideal for daily doc-syncing: do one full crawl, then incremental updates to see only what changed.\n\n### Fast crawl without JavaScript rendering\n```\n\u002Fcf-crawl https:\u002F\u002Fdocs.example.com --no-render --limit 200\n```\nUses static HTML fetch - faster and cheaper but won't capture JS-rendered content.\n\n### Crawl and merge into single file\n```\n\u002Fcf-crawl https:\u002F\u002Fdocs.example.com --limit 50 --merge\n```\nMerges all pages into a single markdown file for easy context loading.\n\n## Argument Parsing\n\nWhen invoked as `\u002Fcf-crawl`, parse the arguments as follows:\n\n- First positional argument: the URL to crawl\n- `--limit N` or `-l N`: max pages (default: 20)\n- `--depth N` or `-d N`: max depth (default: 100000)\n- `--include \"pattern1,pattern2\"`: include URL patterns\n- `--exclude \"pattern1,pattern2\"`: exclude URL patterns\n- `--no-render`: disable JavaScript rendering (faster)\n- `--merge`: combine all output into a single file\n- `--output DIR` or `-o DIR`: output directory (default: `.crawl-output`)\n- `--source sitemaps|links|all`: page discovery method (default: all)\n- `--since DATE`: only crawl pages modified since DATE (ISO date like `2026-03-10` or Unix timestamp). Converts to Unix timestamp for the `modifiedSince` API parameter\n\nIf no URL is provided, ask the user for the target URL.\n\n## Important Notes\n\n- The \u002Fcrawl endpoint respects robots.txt directives including crawl-delay\n- Blocked URLs appear with `\"status\": \"disallowed\"` in results\n- Free plan: 10 minutes of browser time per day\n- Job results are available for 14 days after completion\n- Max job runtime: 7 days\n- Response page size limit: 10 MB per page\n- Use `render: false` for static sites to save browser time\n- Pattern wildcards: `*` matches any character except `\u002F`, `**` matches including `\u002F`\n","# Cloudflare Website Crawler\n\nYou are a web crawling assistant that uses Cloudflare's Browser Rendering \u002Fcrawl REST API to crawl websites and save their content as markdown files for local use.\n\n## Prerequisites\n\nThe user must have:\n1. A Cloudflare account with Browser Rendering enabled\n2. `CLOUDFLARE_ACCOUNT_ID` and `CLOUDFLARE_API_TOKEN` available (see below)\n\n## Workflow\n\nWhen the user asks to crawl a website, follow this exact workflow:\n\n### Step 1: Load Credentials\n\nLook for `CLOUDFLARE_ACCOUNT_ID` and `CLOUDFLARE_API_TOKEN` in this order:\n\n1. **Current environment variables** - Check if already exported in the shell\n2. **Project `.env` file** - Read `.env` in the current working directory and extract the values\n3. **Project `.env.local` file** - Read `.env.local` in the current working directory\n4. **Home directory `.env`** - Read `~\u002F.env` as a last resort\n\nTo load from a `.env` file, parse it line by line looking for `CLOUDFLARE_ACCOUNT_ID=` and `CLOUDFLARE_API_TOKEN=` entries. Use this bash approach:\n\n```bash\n# Load from .env if vars are not already set\nif [ -z \"$CLOUDFLARE_ACCOUNT_ID\" ] || [ -z \"$CLOUDFLARE_API_TOKEN\" ]; then\n  for envfile in .env .env.local \"$HOME\u002F.env\"; do\n    if [ -f \"$envfile\" ]; then\n      eval \"$(grep -E '^CLOUDFLARE_(ACCOUNT_ID|API_TOKEN)=' \"$envfile\" | sed 's\u002F^\u002Fexport \u002F')\"\n    fi\n  done\nfi\n```\n\nIf credentials are still missing after checking all sources, tell the user to add them to their project `.env` file:\n```\nCLOUDFLARE_ACCOUNT_ID=your-account-id\nCLOUDFLARE_API_TOKEN=your-api-token\n```\n\nThe API token needs \"Browser Rendering - Edit\" permission. Create one at [Cloudflare Dashboard > API Tokens](https:\u002F\u002Fdash.cloudflare.com\u002Fprofile\u002Fapi-tokens).\n\n### Step 2: Validate Credentials\n\nVerify both variables are set and non-empty before proceeding.\n\n### Step 3: Initiate Crawl\n\nSend a POST request to start the crawl job. Choose parameters based on user needs:\n\n```bash\ncurl -s -X POST \"https:\u002F\u002Fapi.cloudflare.com\u002Fclient\u002Fv4\u002Faccounts\u002F${CLOUDFLARE_ACCOUNT_ID}\u002Fbrowser-rendering\u002Fcrawl\" \\\n  -H \"Authorization: Bearer ${CLOUDFLARE_API_TOKEN}\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"url\": \"\u003CTARGET_URL>\",\n    \"limit\": \u003CNUMBER_OF_PAGES>,\n    \"formats\": [\"markdown\"],\n    \"options\": {\n      \"excludePatterns\": [\"**\u002Fchangelog\u002F**\", \"**\u002Fapi-reference\u002F**\"]\n    }\n  }'\n```\n\nFor incremental crawls, add the `modifiedSince` parameter (Unix timestamp in seconds):\n\n```bash\ncurl -s -X POST \"https:\u002F\u002Fapi.cloudflare.com\u002Fclient\u002Fv4\u002Faccounts\u002F${CLOUDFLARE_ACCOUNT_ID}\u002Fbrowser-rendering\u002Fcrawl\" \\\n  -H \"Authorization: Bearer ${CLOUDFLARE_API_TOKEN}\" \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"url\": \"\u003CTARGET_URL>\",\n    \"limit\": \u003CNUMBER_OF_PAGES>,\n    \"formats\": [\"markdown\"],\n    \"modifiedSince\": \u003CUNIX_TIMESTAMP>\n  }'\n```\n\nWhen `--since` is provided, convert to Unix timestamp: `date -d \"2026-03-10\" +%s` (Linux) or `date -j -f \"%Y-%m-%d\" \"2026-03-10\" +%s` (macOS).\n\nThe response returns a job ID:\n```json\n{\"success\": true, \"result\": \"job-uuid-here\"}\n```\n\n### Step 4: Poll for Completion\n\nPoll the job status every 5 seconds until it completes:\n\n```bash\ncurl -s -X GET \"https:\u002F\u002Fapi.cloudflare.com\u002Fclient\u002Fv4\u002Faccounts\u002F${CLOUDFLARE_ACCOUNT_ID}\u002Fbrowser-rendering\u002Fcrawl\u002F\u003CJOB_ID>?limit=1\" \\\n  -H \"Authorization: Bearer ${CLOUDFLARE_API_TOKEN}\" | python3 -c \"import sys,json; d=json.load(sys.stdin); print(f'Status: {d[\\\"result\\\"][\\\"status\\\"]} | Finished: {d[\\\"result\\\"][\\\"finished\\\"]}\u002F{d[\\\"result\\\"][\\\"total\\\"]}')\"\n```\n\nPossible job statuses:\n- `running` - Still in progress, keep polling\n- `completed` - All pages processed\n- `cancelled_due_to_timeout` - Exceeded 7-day limit\n- `cancelled_due_to_limits` - Hit account limits\n- `errored` - Something went wrong\n\n### Step 5: Retrieve Results\n\nWhen using `modifiedSince`, check for skipped pages to see what was unchanged:\n\n```bash\n# See which pages were skipped (not modified since the given timestamp)\ncurl -s -X GET \"https:\u002F\u002Fapi.cloudflare.com\u002Fclient\u002Fv4\u002Faccounts\u002F${CLOUDFLARE_ACCOUNT_ID}\u002Fbrowser-rendering\u002Fcrawl\u002F\u003CJOB_ID>?status=skipped&limit=50\" \\\n  -H \"Authorization: Bearer ${CLOUDFLARE_API_TOKEN}\"\n```\n\nFetch all completed records using pagination (cursor-based):\n\n```bash\ncurl -s -X GET \"https:\u002F\u002Fapi.cloudflare.com\u002Fclient\u002Fv4\u002Faccounts\u002F${CLOUDFLARE_ACCOUNT_ID}\u002Fbrowser-rendering\u002Fcrawl\u002F\u003CJOB_ID>?status=completed&limit=50\" \\\n  -H \"Authorization: Bearer ${CLOUDFLARE_API_TOKEN}\"\n```\n\nIf there are more records, use the `cursor` value from the response:\n```bash\ncurl -s -X GET \"https:\u002F\u002Fapi.cloudflare.com\u002Fclient\u002Fv4\u002Faccounts\u002F${CLOUDFLARE_ACCOUNT_ID}\u002Fbrowser-rendering\u002Fcrawl\u002F\u003CJOB_ID>?status=completed&limit=50&cursor=\u003CCURSOR>\" \\\n  -H \"Authorization: Bearer ${CLOUDFLARE_API_TOKEN}\"\n```\n\n### Step 6: Save Results\n\nSave each page's markdown content to a local directory. Use a script like:\n\n```bash\n# Create output directory\nmkdir -p .crawl-output\n\n# Fetch and save all pages\npython3 -c \"\nimport json, os, re, sys, urllib.request\n\naccount_id = os.environ['CLOUDFLARE_ACCOUNT_ID']\napi_token = os.environ['CLOUDFLARE_API_TOKEN']\njob_id = '\u003CJOB_ID>'\nbase = f'https:\u002F\u002Fapi.cloudflare.com\u002Fclient\u002Fv4\u002Faccounts\u002F{account_id}\u002Fbrowser-rendering\u002Fcrawl\u002F{job_id}'\noutdir = '.crawl-output'\nos.makedirs(outdir, exist_ok=True)\n\ncursor = None\ntotal_saved = 0\n\nwhile True:\n    url = f'{base}?status=completed&limit=50'\n    if cursor:\n        url += f'&cursor={cursor}'\n\n    req = urllib.request.Request(url, headers={\n        'Authorization': f'Bearer {api_token}'\n    })\n    with urllib.request.urlopen(req) as resp:\n        data = json.load(resp)\n\n    records = data.get('result', {}).get('records', [])\n    if not records:\n        break\n\n    for rec in records:\n        page_url = rec.get('url', '')\n        md = rec.get('markdown', '')\n        if not md:\n            continue\n        # Convert URL to filename\n        name = re.sub(r'https?:\u002F\u002F', '', page_url)\n        name = re.sub(r'[^a-zA-Z0-9]', '_', name).strip('_')[:120]\n        filepath = os.path.join(outdir, f'{name}.md')\n        with open(filepath, 'w') as f:\n            f.write(f'\u003C!-- Source: {page_url} -->\\n\\n')\n            f.write(md)\n        total_saved += 1\n\n    cursor = data.get('result', {}).get('cursor')\n    if cursor is None:\n        break\n\nprint(f'Saved {total_saved} pages to {outdir}\u002F')\n\"\n```\n\n## Parameter Reference\n\n### Core Parameters\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `url` | string | (required) | Starting URL to crawl |\n| `limit` | number | 10 | Max pages to crawl (up to 100,000) |\n| `depth` | number | 100,000 | Max link depth from starting URL |\n| `formats` | array | [\"html\"] | Output formats: `html`, `markdown`, `json` |\n| `render` | boolean | true | `true` = headless browser, `false` = fast HTML fetch |\n| `source` | string | \"all\" | Page discovery: `all`, `sitemaps`, `links` |\n| `maxAge` | number | 86400 | Cache validity in seconds (max 604800) |\n| `modifiedSince` | number | - | Unix timestamp; only crawl pages modified after this time |\n\n### Options Object\n\n| Parameter | Type | Default | Description |\n|-----------|------|---------|-------------|\n| `includePatterns` | array | [] | Wildcard patterns to include (`*` and `**`) |\n| `excludePatterns` | array | [] | Wildcard patterns to exclude (higher priority) |\n| `includeSubdomains` | boolean | false | Follow links to subdomains |\n| `includeExternalLinks` | boolean | false | Follow external links |\n\n### Advanced Parameters\n\n| Parameter | Type | Description |\n|-----------|------|-------------|\n| `jsonOptions` | object | AI-powered structured extraction (prompt, response_format) |\n| `authenticate` | object | HTTP basic auth (username, password) |\n| `setExtraHTTPHeaders` | object | Custom headers for requests |\n| `rejectResourceTypes` | array | Skip: image, media, font, stylesheet |\n| `userAgent` | string | Custom user agent string |\n| `cookies` | array | Custom cookies for requests |\n\n## Usage Examples\n\n### Crawl documentation site (most common)\n```\n\u002Fcf-crawl https:\u002F\u002Fdocs.example.com --limit 50\n```\nCrawls up to 50 pages, saves as markdown.\n\n### Crawl with filters\n```\n\u002Fcf-crawl https:\u002F\u002Fdocs.example.com --limit 100 --include \"\u002Fguides\u002F**,\u002Fapi\u002F**\" --exclude \"\u002Fchangelog\u002F**\"\n```\n\n### Incremental crawl (diff detection)\n```\n\u002Fcf-crawl https:\u002F\u002Fdocs.example.com --limit 50 --since 2026-03-10\n```\nOnly crawls pages modified since the given date. Skipped pages appear with `status=skipped` in results. This is ideal for daily doc-syncing: do one full crawl, then incremental updates to see only what changed.\n\n### Fast crawl without JavaScript rendering\n```\n\u002Fcf-crawl https:\u002F\u002Fdocs.example.com --no-render --limit 200\n```\nUses static HTML fetch - faster and cheaper but won't capture JS-rendered content.\n\n### Crawl and merge into single file\n```\n\u002Fcf-crawl https:\u002F\u002Fdocs.example.com --limit 50 --merge\n```\nMerges all pages into a single markdown file for easy context loading.\n\n## Argument Parsing\n\nWhen invoked as `\u002Fcf-crawl`, parse the arguments as follows:\n\n- First positional argument: the URL to crawl\n- `--limit N` or `-l N`: max pages (default: 20)\n- `--depth N` or `-d N`: max depth (default: 100000)\n- `--include \"pattern1,pattern2\"`: include URL patterns\n- `--exclude \"pattern1,pattern2\"`: exclude URL patterns\n- `--no-render`: disable JavaScript rendering (faster)\n- `--merge`: combine all output into a single file\n- `--output DIR` or `-o DIR`: output directory (default: `.crawl-output`)\n- `--source sitemaps|links|all`: page discovery method (default: all)\n- `--since DATE`: only crawl pages modified since DATE (ISO date like `2026-03-10` or Unix timestamp). Converts to Unix timestamp for the `modifiedSince` API parameter\n\nIf no URL is provided, ask the user for the target URL.\n\n## Important Notes\n\n- The \u002Fcrawl endpoint respects robots.txt directives including crawl-delay\n- Blocked URLs appear with `\"status\": \"disallowed\"` in results\n- Free plan: 10 minutes of browser time per day\n- Job results are available for 14 days after completion\n- Max job runtime: 7 days\n- Response page size limit: 10 MB per page\n- Use `render: false` for static sites to save browser time\n- Pattern wildcards: `*` matches any character except `\u002F`, `**` matches including `\u002F`\n",{"name":5,"description":7},"main","34fa6acf794b9a6310d298cbb8fb8c9420a5edcf","cli-tool\u002Fcomponents\u002Fskills\u002Futilities\u002Fcf-crawl",2,"https:\u002F\u002Fgithub.com\u002Ftransaurus\u002Fstaging-davila7-claude-code-templates","https:\u002F\u002Fgithub.com\u002Ftransaurus","Automated works of Page Turner","2026-03-28 10:12:59",{"source":8,"skillName":5,"commands":53,"prompt":63,"zipAvailable":64},[54,57,60],{"tool":55,"command":56},"npx","npx skills add transaurus\u002Fstaging-davila7-claude-code-templates --skill cf-crawl",{"tool":58,"command":59},"bunx","bunx skills add transaurus\u002Fstaging-davila7-claude-code-templates --skill cf-crawl",{"tool":61,"command":62},"pnpm","pnpm dlx skills add transaurus\u002Fstaging-davila7-claude-code-templates --skill cf-crawl","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Ftransaurus\u002Fstaging-davila7-claude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Futilities\u002Fcf-crawl\u002FSKILL.md",true,[],[67,98,119,143,163,186],{"skillNo":68,"name":69,"nameZh":6,"description":70,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":71,"categories":72,"keywords":78,"publishedAt":39},"01M6JMRJR3BE0HGWFWDZSB853S","zapier-make-patterns","No-code automation democratizes workflow building. Zapier and Make (formerly Integromat) let non-developers automate business processes without writing code. But no-code doesn't mean no-complexity - these platforms have their own patterns, pitfalls, and breaking points.  This skill covers when to use which platform, how to build reliable automations, and when to graduate to code-based solutions. Key insight: Zapier optimizes for simplicity and integrations (7000+ apps), Make optimizes for power","https:\u002F\u002Fgithub.com\u002Ftransaurus\u002Fstaging-davila7-claude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Fzapier-make-patterns\u002FSKILL.md",[73],{"categoryNo":74,"categoryName":75,"taxonomyType":76,"taxonomyTypeDesc":77},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",[79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,21,97],"zapier make patterns","patterns","make","zapier","writing","workflow","automation","code","integrations","business","use","power","complexity","platform","apps","how to","No-Code","which","spawner",{"skillNo":99,"name":100,"nameZh":6,"description":101,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":102,"categories":103,"keywords":104,"publishedAt":39},"01M6JMR8YK6SWPCTG1MCQTMN9S","yeet","Use only when the user explicitly asks to stage, commit, push, and open a GitHub pull request in one flow using the GitHub CLI (`gh`).","https:\u002F\u002Fgithub.com\u002Ftransaurus\u002Fstaging-davila7-claude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Fyeet\u002FSKILL.md",[],[100,105,106,107,108,109,110,111,112,113,114,115,89,116,117,118],"cli","github","commit","pull request","openai","pull","push","flow","open","gh","user","only","stage","github cli",{"skillNo":120,"name":121,"nameZh":6,"description":122,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":123,"categories":124,"keywords":126,"publishedAt":39},"01M6JMQNFE64ZASQK90HARRXW8","workflow-automation","Workflow automation is the infrastructure that makes AI agents reliable. Without durable execution, a network hiccup during a 10-step payment flow means lost money and angry customers. With it, workflows resume exactly where they left off.  This skill covers the platforms (n8n, Temporal, Inngest) and patterns (sequential, parallel, orchestrator-worker) that turn brittle scripts into production-grade automation.  Key insight: The platforms make different tradeoffs. n8n optimizes for accessibility","https:\u002F\u002Fgithub.com\u002Ftransaurus\u002Fstaging-davila7-claude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Fworkflow-automation\u002FSKILL.md",[125],{"categoryNo":74,"categoryName":75,"taxonomyType":76,"taxonomyTypeDesc":77},[84,85,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,21,97],"Workflow Automation","ai agents","accessibility","workflows","apache","execution","inngest","parallel","orchestrator","resume","n8n","payment","worker","scripts","temporal","with",{"skillNo":144,"name":145,"nameZh":6,"description":146,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":147,"categories":148,"keywords":149,"publishedAt":39},"01M6JMQDYJ0XETERPHGDR3EVGN","trigger-dev","Trigger.dev expert for background jobs, AI workflows, and reliable async execution with excellent developer experience and TypeScript-first design. Use when: trigger.dev, trigger dev, background task, ai background job, long running task.","https:\u002F\u002Fgithub.com\u002Ftransaurus\u002Fstaging-davila7-claude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Ftrigger-dev\u002FSKILL.md",[],[150,151,152,153,154,130,155,131,132,156,89,157,158,159,160,161,162,142,21,97],"dev","trigger dev","trigger","design","typescript","async execution","first","async","developer experience","expert","task","background jobs","long",{"skillNo":164,"name":165,"nameZh":6,"description":166,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":167,"categories":168,"keywords":172,"publishedAt":39},"01M6JMQ5J6KK3GSX9FJCWWK6G6","planning-with-files","Transforms workflow to use Manus-style persistent markdown files for planning, progress tracking, and knowledge storage. Use when starting complex tasks, multi-step projects, research tasks, or when the user mentions planning, organizing work, tracking progress, or wants structured output.","https:\u002F\u002Fgithub.com\u002Ftransaurus\u002Fstaging-davila7-claude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Fplanning-with-files\u002FSKILL.md",[169],{"categoryNo":170,"categoryName":171,"taxonomyType":76,"taxonomyTypeDesc":77},"01JBSEEDCAT000000000000021","资料检索与研究",[173,174,31,142,175,84,176,177,35,34,178,179,89,180,181,182,183,184,185,21],"planning with files","planning","with files","storage","tasks","projects","work","manus","structured output","progress","transforms","multi","output",{"skillNo":187,"name":188,"nameZh":6,"description":189,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":190,"categories":191,"keywords":196,"publishedAt":39},"01M6JMPWAT60RXXNEC4V9TRBRS","n8n-workflow-patterns","Proven workflow architectural patterns from real n8n workflows. Use when building new workflows, designing workflow structure, choosing workflow patterns, planning workflow architecture, or asking about webhook processing, HTTP API integration, database operations, AI agent workflows, or scheduled tasks.","https:\u002F\u002Fgithub.com\u002Ftransaurus\u002Fstaging-davila7-claude-code-templates\u002Fblob\u002Fmain\u002Fcli-tool\u002Fcomponents\u002Fskills\u002Fworkflow-automation\u002Fn8n\u002Fn8n-workflow-patterns\u002FSKILL.md",[192,195],{"categoryNo":193,"categoryName":194,"taxonomyType":76,"taxonomyTypeDesc":77},"01JBSEEDCAT000000000000010","后端与接口",{"categoryNo":74,"categoryName":75,"taxonomyType":76,"taxonomyTypeDesc":77},[84,197,137,198,80,199,200,130,201,177,174,202,36,203,204,205,206,207,208,209],"n8n workflow patterns","workflow patterns","n8n workflow","ai agent","architecture","database","planning workflow","structure","database operations","webhook","processing","http api","scheduled"]