[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6EEQGDVVQ2JR2X5EWC61KXR":3,"skill-same:01M6EEQGDVVQ2JR2X5EWC61KXR":45},{"skillNo":4,"name":5,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":18,"linkOnly":19,"markdownContent":20,"markdownBody":20,"frontMatter":21,"refName":22,"commitSha":23,"skillDirectory":24,"forksCount":25,"repositoryHtmlUrl":26,"repositoryHomepageUrl":27,"ownerHtmlUrl":28,"ownerBlogUrl":6,"ownerDescription":29,"updatedAt":30,"install":31,"resources":44},"01M6EEQGDVVQ2JR2X5EWC61KXR","crawlee",null,"ComeOnOliver\u002Fskillshub","ComeOnOliver","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F43215937?v=4","01M6E5JNFECJAAVFE0W7SMWWD2","Haoran Shu","01M6E5JNNDQS7AJZDYFQ8TPZCK",67,"MIT","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002FTerminalSkills\u002Fskills\u002Fcrawlee\u002FSKILL.md",[],[5],"2026-11-10 10:12:30",false,"# Crawlee\n\n## Overview\n\nCrawlee is a web scraping and crawling library that handles the hard parts — request queuing, retries, proxy rotation, browser fingerprinting, and rate limiting. Use Cheerio for fast HTML-only scraping or Playwright\u002FPuppeteer for JavaScript-rendered pages. Built-in storage for datasets, request queues, and key-value stores. Scales from single pages to millions of URLs.\n\n## When to Use\n\n- Scraping data from websites (product prices, job listings, articles)\n- Crawling entire sites for content or link analysis\n- JavaScript-rendered pages (SPAs, React\u002FVue sites)\n- Scraping at scale with proxy rotation and anti-blocking\n- Structured data extraction with automatic retries\n\n## Instructions\n\n### Setup\n\n```bash\nnpm install crawlee playwright\nnpx playwright install chromium  # Only for browser crawling\n```\n\n### HTTP Crawling (Fast, No Browser)\n\n```typescript\n\u002F\u002F scraper.ts — Fast scraping with Cheerio (no browser needed)\nimport { CheerioCrawler, Dataset } from \"crawlee\";\n\nconst crawler = new CheerioCrawler({\n  maxConcurrency: 10,          \u002F\u002F Parallel requests\n  maxRequestRetries: 3,        \u002F\u002F Retry failed requests\n  requestHandlerTimeoutSecs: 30,\n\n  async requestHandler({ request, $, enqueueLinks, pushData }) {\n    \u002F\u002F $ is Cheerio — jQuery-like selector API\n    const title = $(\"h1\").text().trim();\n    const price = $(\"[data-testid='price']\").text().trim();\n    const description = $(\"meta[name='description']\").attr(\"content\");\n\n    \u002F\u002F Save structured data\n    await pushData({\n      url: request.url,\n      title,\n      price,\n      description,\n      scrapedAt: new Date().toISOString(),\n    });\n\n    \u002F\u002F Follow pagination links\n    await enqueueLinks({\n      selector: \"a.next-page\",\n      label: \"LISTING\",\n    });\n  },\n\n  \u002F\u002F Handle different page types\n  async failedRequestHandler({ request }) {\n    console.error(`Failed: ${request.url} after ${request.retryCount} retries`);\n  },\n});\n\n\u002F\u002F Start crawling\nawait crawler.run([\"https:\u002F\u002Fexample-shop.com\u002Fproducts\"]);\n\n\u002F\u002F Export data\nconst dataset = await Dataset.open();\nawait dataset.exportToCSV(\"products\");\n```\n\n### Browser Crawling (JavaScript-Rendered Pages)\n\n```typescript\n\u002F\u002F browser-scraper.ts — Scrape JS-rendered pages with Playwright\nimport { PlaywrightCrawler } from \"crawlee\";\n\nconst crawler = new PlaywrightCrawler({\n  maxConcurrency: 5,           \u002F\u002F Fewer concurrent — browsers are heavy\n  headless: true,\n  launchContext: {\n    launchOptions: {\n      args: [\"--disable-blink-features=AutomationControlled\"],\n    },\n  },\n\n  async requestHandler({ page, request, pushData, enqueueLinks }) {\n    \u002F\u002F Wait for dynamic content to load\n    await page.waitForSelector(\"[data-loaded='true']\", { timeout: 10000 });\n\n    \u002F\u002F Extract data using Playwright selectors\n    const items = await page.$$eval(\".product-card\", (cards) =>\n      cards.map((card) => ({\n        name: card.querySelector(\"h3\")?.textContent?.trim(),\n        price: card.querySelector(\".price\")?.textContent?.trim(),\n        rating: card.querySelector(\".stars\")?.getAttribute(\"data-rating\"),\n      }))\n    );\n\n    for (const item of items) {\n      await pushData({ ...item, sourceUrl: request.url });\n    }\n\n    \u002F\u002F Scroll to load more (infinite scroll)\n    await page.evaluate(() => window.scrollTo(0, document.body.scrollHeight));\n    await page.waitForTimeout(2000);\n\n    \u002F\u002F Click \"Load More\" if exists\n    const loadMore = page.locator(\"button:has-text('Load More')\");\n    if (await loadMore.isVisible()) {\n      await loadMore.click();\n      await page.waitForLoadState(\"networkidle\");\n    }\n  },\n});\n\nawait crawler.run([\"https:\u002F\u002Fspa-example.com\u002Fproducts\"]);\n```\n\n### Proxy Rotation\n\n```typescript\n\u002F\u002F proxy-scraper.ts — Rotate proxies to avoid blocking\nimport { CheerioCrawler, ProxyConfiguration } from \"crawlee\";\n\nconst proxyConfig = new ProxyConfiguration({\n  proxyUrls: [\n    \"http:\u002F\u002Fuser:pass@proxy1.example.com:8080\",\n    \"http:\u002F\u002Fuser:pass@proxy2.example.com:8080\",\n    \"http:\u002F\u002Fuser:pass@proxy3.example.com:8080\",\n  ],\n});\n\nconst crawler = new CheerioCrawler({\n  proxyConfiguration: proxyConfig,\n  \u002F\u002F Crawlee automatically rotates and retires failing proxies\n  async requestHandler({ request, $, pushData, proxyInfo }) {\n    console.log(`Using proxy: ${proxyInfo?.url}`);\n    \u002F\u002F ... scraping logic\n  },\n});\n```\n\n## Examples\n\n### Example 1: Scrape product data from an e-commerce site\n\n**User prompt:** \"Scrape all product names, prices, and ratings from example-shop.com and export to CSV.\"\n\nThe agent will create a CheerioCrawler with pagination handling, structured data extraction, and CSV export.\n\n### Example 2: Monitor competitor prices\n\n**User prompt:** \"Build a daily scraper that checks competitor prices and alerts when they change.\"\n\nThe agent will create a PlaywrightCrawler for JS-rendered pages, store prices in a dataset, compare with previous runs, and send alerts on changes.\n\n## Guidelines\n\n- **Cheerio for static HTML** — 10x faster than browser crawling\n- **Playwright for SPAs** — use only when JavaScript rendering is required\n- **`enqueueLinks` for crawling** — automatically follows and deduplicates links\n- **`pushData` for structured output** — builds a dataset that exports to CSV\u002FJSON\n- **Proxy rotation for scale** — Crawlee retires failing proxies automatically\n- **Respect robots.txt** — check `robotsTxtUrl` in crawler config\n- **Rate limit** — `maxRequestsPerMinute` to avoid overwhelming targets\n- **Request labels** — use labels to route different page types to different handlers\n- **Error handling** — `failedRequestHandler` catches and logs failed URLs\n- **Storage persists** — datasets and queues survive restarts by default\n",{},"main","def8531e65114c0fca8fb8551c1871ee0eed705c","skills\u002FTerminalSkills\u002Fskills\u002Fcrawlee",24,"https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub","https:\u002F\u002Fskillshub.wtf","https:\u002F\u002Fgithub.com\u002FComeOnOliver","AI & Blockchain Engineer\r\n","2026-06-24 22:07:19",{"source":7,"skillName":5,"commands":32,"prompt":42,"zipAvailable":43},[33,36,39],{"tool":34,"command":35},"npx","npx skills add ComeOnOliver\u002Fskillshub --skill crawlee",{"tool":37,"command":38},"bunx","bunx skills add ComeOnOliver\u002Fskillshub --skill crawlee",{"tool":40,"command":41},"pnpm","pnpm dlx skills add ComeOnOliver\u002Fskillshub --skill crawlee","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002FTerminalSkills\u002Fskills\u002Fcrawlee\u002FSKILL.md",true,[],[46,60,70,79,85,92],{"skillNo":47,"name":48,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":49,"categories":50,"keywords":56,"publishedAt":18},"01M6FKJN13GMG4KY57J3QD8GY5","model-hierarchy-skill","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzscole\u002Fmodel-hierarchy-skill\u002Fmodel-hierarchy-skill\u002FSKILL.md",[51],{"categoryNo":52,"categoryName":53,"taxonomyType":54,"taxonomyTypeDesc":55},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",[57,58,59],"model","model hierarchy","model hierarchy skill",{"skillNo":61,"name":62,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":63,"categories":64,"keywords":66,"publishedAt":18},"01M6FKJJJ6GCAMR4PPPRD12M5X","zoom-mcp","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fzoom-mcp\u002FSKILL.md",[65],{"categoryNo":52,"categoryName":53,"taxonomyType":54,"taxonomyTypeDesc":55},[67,68,69],"MCP","zoom","zoom mcp",{"skillNo":71,"name":72,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":73,"categories":74,"keywords":75,"publishedAt":18},"01M6FKJE3KYT3KPEZMEEHYKA6K","zoom-apps-sdk","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fzoom-apps-sdk\u002FSKILL.md",[],[76,77,78,68],"sdk","zoom apps sdk","apps",{"skillNo":80,"name":81,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":82,"categories":83,"keywords":84,"publishedAt":18},"01M6FKJ9EDH9Q5VWDF8HAJRSP0","windows","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fwindows\u002FSKILL.md",[],[81],{"skillNo":86,"name":87,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":88,"categories":89,"keywords":91,"publishedAt":18},"01M6FKJ666BEEFC3AE9GJ72Q93","whiteboard","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fwhiteboard\u002FSKILL.md",[90],{"categoryNo":52,"categoryName":53,"taxonomyType":54,"taxonomyTypeDesc":55},[87],{"skillNo":93,"name":94,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":95,"categories":96,"keywords":97,"publishedAt":18},"01M6FKJ21WFCDGTJQ2Q91T9DVW","websockets","https:\u002F\u002Fgithub.com\u002FComeOnOliver\u002Fskillshub\u002Fblob\u002Fmain\u002Fskills\u002Fzoom\u002Fskills\u002Fwebsockets\u002FSKILL.md",[],[94]]