[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6H7VVFPTSGD0D8YRBW7M8XZ":3,"skill-same:01M6H7VVFPTSGD0D8YRBW7M8XZ":65},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":32,"linkOnly":33,"markdownContent":34,"markdownBody":35,"frontMatter":36,"refName":43,"commitSha":44,"skillDirectory":45,"forksCount":46,"repositoryHtmlUrl":47,"repositoryHomepageUrl":6,"ownerHtmlUrl":48,"ownerBlogUrl":49,"ownerDescription":6,"updatedAt":50,"install":51,"resources":64},"01M6H7VVFPTSGD0D8YRBW7M8XZ","web-scraping-expert",null,"Use when implementing web scraping functionality with production-grade patterns and safeguards.","0xharryriddle\u002Fcodex-field-kit","0xharryriddle","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F100945007?v=4","01M6H686JV0QKZYP2RWG6B2B8T","harryriddle","01M6H686PVZ8JGKWEG7B3GM5PD",3,"MIT","https:\u002F\u002Fgithub.com\u002F0xharryriddle\u002Fcodex-field-kit\u002Fblob\u002Fmain\u002Fskills-hermes\u002Fcodex-agents\u002Fweb-scraping-expert\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31],"web","web scraping","scraping","expert","web scraping expert","codex","Production","use","patterns","runtime","with","kit","implementing","2026-11-10 10:12:30",false,"---\nname: web-scraping-expert\ndescription: Use when implementing web scraping functionality with production-grade patterns and safeguards.\nmetadata:\n  hermes:\n    tags: [codex-agent, languages-runtime]\n    source: codex-field-kit\u002Flanguages-runtime\n---\n\n# Web Scraping Expert\n\n# Python Web Scraping Expert Agent\n\n## Role & Expertise\n\nI am a specialized Python web scraping expert with comprehensive knowledge of:\n\n**Core Scraping Technologies:**\n- **BeautifulSoup 4**: HTML\u002FXML parsing and navigation\n- **Scrapy**: High-performance web crawling framework\n- **Selenium**: Browser automation and dynamic content\n- **Playwright**: Modern browser automation and testing\n- **Requests\u002Fhttpx**: HTTP client libraries for web requests\n- **aiohttp**: Asynchronous HTTP client\u002Fserver framework\n\n**Advanced Techniques:**\n- **Async\u002FAwait Scraping**: High-performance concurrent scraping\n- **Anti-Bot Evasion**: Headers, proxies, rate limiting, CAPTCHA handling\n- **Data Extraction**: Complex parsing patterns, regex, XPath, CSS selectors\n- **Session Management**: Cookies, authentication, form handling\n- **Performance Optimization**: Connection pooling, caching, batch processing\n- **Data Pipeline**: ETL processes, data cleaning, storage integration\n\n**Specialized Areas:**\n- **JavaScript-Heavy Sites**: SPA scraping, dynamic content loading\n- **API Scraping**: REST\u002FGraphQL API interaction and reverse engineering\n- **Large-Scale Scraping**: Distributed scraping, queue systems\n- **Legal & Ethical**: Robots.txt compliance, rate limiting, respectful scraping\n\n## Key Principles\n\n### 1. **Respectful Scraping**\n- Always check robots.txt and terms of service\n- Implement proper rate limiting and delays\n- Use appropriate User-Agent headers\n- Respect website resources and bandwidth\n\n### 2. **Robustness & Reliability**\n- Handle failures gracefully with retries\n- Implement comprehensive error handling\n- Use circuit breakers for unstable sites\n- Log all operations for debugging\n\n### 3. **Performance & Scalability**\n- Asynchronous operations for high throughput\n- Connection pooling and session reuse\n- Efficient parsing and data processing\n- Memory-conscious design patterns\n\n### 4. **Data Quality**\n- Comprehensive data validation\n- Duplicate detection and handling\n- Data normalization and cleaning\n- Schema enforcement and type checking\n\n## Implementation Examples\n\n### 1. **Modern Async Scraping Framework**\n\n**scraper\u002Fcore.py**:\n```python\nimport asyncio\nimport aiohttp\nimport aiofiles\nimport time\nfrom typing import List, Dict, Any, Optional, AsyncGenerator\nfrom dataclasses import dataclass, asdict\nfrom urllib.parse import urljoin, urlparse\nimport json\nimport logging\nfrom pathlib import Path\nimport random\nfrom bs4 import BeautifulSoup\nimport httpx\nfrom tenacity import retry, stop_after_attempt, wait_exponential\n\n# Configure logging\nlogging.basicConfig(level=logging.INFO)\nlogger = logging.getLogger(__name__)\n\n@dataclass\nclass ScrapingConfig:\n\n","# Web Scraping Expert\n\n# Python Web Scraping Expert Agent\n\n## Role & Expertise\n\nI am a specialized Python web scraping expert with comprehensive knowledge of:\n\n**Core Scraping Technologies:**\n- **BeautifulSoup 4**: HTML\u002FXML parsing and navigation\n- **Scrapy**: High-performance web crawling framework\n- **Selenium**: Browser automation and dynamic content\n- **Playwright**: Modern browser automation and testing\n- **Requests\u002Fhttpx**: HTTP client libraries for web requests\n- **aiohttp**: Asynchronous HTTP client\u002Fserver framework\n\n**Advanced Techniques:**\n- **Async\u002FAwait Scraping**: High-performance concurrent scraping\n- **Anti-Bot Evasion**: Headers, proxies, rate limiting, CAPTCHA handling\n- **Data Extraction**: Complex parsing patterns, regex, XPath, CSS selectors\n- **Session Management**: Cookies, authentication, form handling\n- **Performance Optimization**: Connection pooling, caching, batch processing\n- **Data Pipeline**: ETL processes, data cleaning, storage integration\n\n**Specialized Areas:**\n- **JavaScript-Heavy Sites**: SPA scraping, dynamic content loading\n- **API Scraping**: REST\u002FGraphQL API interaction and reverse engineering\n- **Large-Scale Scraping**: Distributed scraping, queue systems\n- **Legal & Ethical**: Robots.txt compliance, rate limiting, respectful scraping\n\n## Key Principles\n\n### 1. **Respectful Scraping**\n- Always check robots.txt and terms of service\n- Implement proper rate limiting and delays\n- Use appropriate User-Agent headers\n- Respect website resources and bandwidth\n\n### 2. **Robustness & Reliability**\n- Handle failures gracefully with retries\n- Implement comprehensive error handling\n- Use circuit breakers for unstable sites\n- Log all operations for debugging\n\n### 3. **Performance & Scalability**\n- Asynchronous operations for high throughput\n- Connection pooling and session reuse\n- Efficient parsing and data processing\n- Memory-conscious design patterns\n\n### 4. **Data Quality**\n- Comprehensive data validation\n- Duplicate detection and handling\n- Data normalization and cleaning\n- Schema enforcement and type checking\n\n## Implementation Examples\n\n### 1. **Modern Async Scraping Framework**\n\n**scraper\u002Fcore.py**:\n```python\nimport asyncio\nimport aiohttp\nimport aiofiles\nimport time\nfrom typing import List, Dict, Any, Optional, AsyncGenerator\nfrom dataclasses import dataclass, asdict\nfrom urllib.parse import urljoin, urlparse\nimport json\nimport logging\nfrom pathlib import Path\nimport random\nfrom bs4 import BeautifulSoup\nimport httpx\nfrom tenacity import retry, stop_after_attempt, wait_exponential\n\n# Configure logging\nlogging.basicConfig(level=logging.INFO)\nlogger = logging.getLogger(__name__)\n\n@dataclass\nclass ScrapingConfig:\n\n",{"name":5,"metadata":37,"description":7},{"hermes":38},{"tags":39,"source":42},[40,41],"codex-agent","languages-runtime","codex-field-kit\u002Flanguages-runtime","main","6c13f6f4ac5a3bc1e6b78cd2bb1704f122c4a73e","skills-hermes\u002Fcodex-agents\u002Fweb-scraping-expert",0,"https:\u002F\u002Fgithub.com\u002F0xharryriddle\u002Fcodex-field-kit","https:\u002F\u002Fgithub.com\u002F0xharryriddle","https:\u002F\u002Fharryriddle.vercel.app","2026-04-11 11:49:51",{"source":8,"skillName":5,"commands":52,"prompt":62,"zipAvailable":63},[53,56,59],{"tool":54,"command":55},"npx","npx skills add 0xharryriddle\u002Fcodex-field-kit --skill web-scraping-expert",{"tool":57,"command":58},"bunx","bunx skills add 0xharryriddle\u002Fcodex-field-kit --skill web-scraping-expert",{"tool":60,"command":61},"pnpm","pnpm dlx skills add 0xharryriddle\u002Fcodex-field-kit --skill web-scraping-expert","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002F0xharryriddle\u002Fcodex-field-kit\u002Fblob\u002Fmain\u002Fskills-hermes\u002Fcodex-agents\u002Fweb-scraping-expert\u002FSKILL.md",true,[],[66,98,117,127,152,179],{"skillNo":67,"name":68,"nameZh":6,"description":69,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":70,"categories":71,"keywords":77,"publishedAt":32},"01M6H7ZKE6MR33HC84QZ1TA6MV","vercel-react-best-practices","React and Next.js performance optimization guidelines from Vercel Engineering. This skill should be used when writing, reviewing, or refactoring React\u002FNext.js code to ensure optimal performance patterns. Triggers on tasks involving React components, Next.js pages, data fetching, bundle optimization, or performance improvements.","https:\u002F\u002Fgithub.com\u002F0xharryriddle\u002Fcodex-field-kit\u002Fblob\u002Fmain\u002Fskills\u002Fvercel-react-best-practices\u002FSKILL.md",[72],{"categoryNo":73,"categoryName":74,"taxonomyType":75,"taxonomyTypeDesc":76},"01JBSEEDCAT000000000000009","前端开发","SCENARIO","使用场景",[78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97],"react best practices","react","vercel react best practices","best practices","vercel","tasks","data fetching","triggers","pages","next","performance optimization","refactoring","Next.js","Optimization","code","guidelines","react components","components","performance","writing",{"skillNo":99,"name":100,"nameZh":6,"description":101,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":102,"categories":103,"keywords":104,"publishedAt":32},"01M6H7ZGC2WK8SEKYHWDNTMDA9","swarm-planner","[EXPLICIT INVOCATION ONLY] Creates dependency-aware implementation plans optimized for parallel multi-agent execution.","https:\u002F\u002Fgithub.com\u002F0xharryriddle\u002Fcodex-field-kit\u002Fblob\u002Fmain\u002Fskills\u002Fswarm-planner\u002FSKILL.md",[],[105,106,107,108,109,110,111,112,113,114,115,116],"swarm","swarm planner","planner","parallel","multi-agent","only","execution","plans","dependency","for","multi","implementation",{"skillNo":118,"name":119,"nameZh":6,"description":120,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":121,"categories":122,"keywords":123,"publishedAt":32},"01M6H7ZD93853RZCYQWH1D38TM","super-swarm-spark","Only to be triggered by explicit super-swarm-spark commands.","https:\u002F\u002Fgithub.com\u002F0xharryriddle\u002Fcodex-field-kit\u002Fblob\u002Fmain\u002Fskills\u002Fsuper-swarm-spark\u002FSKILL.md",[],[124,105,125,110,126],"super swarm spark","spark","commands",{"skillNo":128,"name":129,"nameZh":6,"description":130,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":131,"categories":132,"keywords":133,"publishedAt":32},"01M6H7Z987QBY49JCCPT953EAZ","role-creator","Create and install Codex custom agent roles in ~\u002F.codex\u002Fconfig.toml, generate role config files, enforce supported keys, and guide users through required role inputs (model, reasoning effort, developer_instructions).","https:\u002F\u002Fgithub.com\u002F0xharryriddle\u002Fcodex-field-kit\u002Fblob\u002Fmain\u002Fskills\u002Frole-creator\u002FSKILL.md",[],[134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,24,151],"role","creator","role creator","keys","users","roles","toml","enforce","install","guide","instructions","model","developer","reasoning","generate","create","config","files",{"skillNo":153,"name":154,"nameZh":6,"description":155,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":156,"categories":157,"keywords":161,"publishedAt":32},"01M6H7Z5TRPMDQ7YSZQXPN1G60","read-github","Read and search GitHub repository documentation via gitmcp.io MCP service.\n\n**WHEN TO USE:**\n- User provides a GitHub URL\n- User mentions a specific repo in owner\u002Frepo format\n- User asks \"what does this repo do?\", \"read the docs for X repo\", or similar\n- User wants to search code or docs within a repo","https:\u002F\u002Fgithub.com\u002F0xharryriddle\u002Fcodex-field-kit\u002Fblob\u002Fmain\u002Fskills\u002Fread-github\u002FSKILL.md",[158],{"categoryNo":159,"categoryName":160,"taxonomyType":75,"taxonomyTypeDesc":76},"01JBSEEDCAT000000000000023","Skill与智能体开发",[162,163,164,92,165,166,167,168,114,169,170,171,172,173,174,26,175,176,177,178],"github","read github","read","documentation","docs","MCP","search","when to use","URL","via","io","service","do","user","repo","format","repository",{"skillNo":180,"name":107,"nameZh":6,"description":181,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":182,"categories":183,"keywords":184,"publishedAt":32},"01M6H7Z215CTKYJ4Y1X6EQWN6M","Create comprehensive, phased implementation plans with sprints and atomic tasks. Use when user says: \"make a plan\", \"create a plan\", \"plan this out\", \"plan the implementation\", \"help me plan\", \"design a plan\", \"draft a plan\", \"write a plan\", \"outline the steps\", \"break this down into tasks\", \"what's the plan for\", or any similar planning request. Also triggers on explicit \"\u002Fplanner\" or \"\u002Fplan\" commands.","https:\u002F\u002Fgithub.com\u002F0xharryriddle\u002Fcodex-field-kit\u002Fblob\u002Fmain\u002Fskills\u002Fplanner\u002FSKILL.md",[],[107,185,114,186,29,187,85,112,126,188,189,26,190,191,192,193,194,149,83,195],"outline","atomic","sprints","draft","me","write","phased implementation","plan","help","planning","design"]