AGENT SKILL
smart_scraper
Intelligent web scraping with automatic retry, rate limiting, and anti-detection via Firecrawl patterns.
- License 许可
- MIT
- 最近更新
- 2026-10-09
- Stars
- ★ 2
- Forks
- 1
01 前言 Front Matter
- name
- smart_scraper
- tags
- [ "scraping", "data-extraction", "firecrawl", "web-scraping", "automation" ]
- version
- 1.0.0
- connector
- scripts/connectors/scraper_agent.py
- description
- Intelligent web scraping with automatic retry, rate limiting, and anti-detection via Firecrawl patterns.
03 SKILL.md 正文
Skill: Smart Scraper
Execution Steps
- Accept target URL(s) and data extraction schema (what fields to extract).
- Analyze target: static HTML vs JS-rendered vs API-backed.
- Select strategy: Simple HTTP for static, Firecrawl/Playwright for dynamic.
- Execute with guardrails: rate limiting, rotating user agents, retry with backoff.
- Extract and validate data against schema.
- Output: Clean CSV/JSON with extracted data.
Scraping Strategies
| Target Type | Tool | Method |
|---|---|---|
| Static HTML | HTTP + BeautifulSoup | Direct parse |
| JS-rendered SPA | Firecrawl / Playwright | Headless browser |
| Paginated lists | HTTP + pagination logic | Follow next links |
| API-backed | Direct API calls | REST/GraphQL |
Quality Validation
- [ ] robots.txt respected
- [ ] Rate limiting applied (max 1 req/second default)
- [ ] Extracted data matches schema (no missing fields)
- [ ] No PII inadvertently captured
04 同仓库更多 Skills
web_crawler_katana
High-speed web crawling using Katana patterns for deep site mapping, URL discovery, and content extraction.
★ 2
server_log_analyzer
Analyzes Nginx/Apache server logs to identify Googlebot crawl patterns, errors, and performance bottlenecks.
★ 2
schema_markup_generator
Auto-generates JSON-LD structured data for any page type (Article, Product, FAQ, LocalBusiness, etc.).
★ 2
internal_linking_optimizer
Analyzes and optimizes the internal link graph of a website for SEO authority distribution.
★ 2
google_trends_analyzer
Detects trending topics and seasonal patterns using Google Trends data for content planning and keyword strategy.
★ 2
brand_context_analyzer
Tracks brand mentions, sentiment, and online reputation across web sources.
★ 2