AGENT SKILL

smart_scraper

Intelligent web scraping with automatic retry, rate limiting, and anti-detection via Firecrawl patterns.

License 许可
MIT
最近更新
2026-10-09
Stars
★ 2
Forks
1

01 前言 Front Matter

name
smart_scraper
tags
[ "scraping", "data-extraction", "firecrawl", "web-scraping", "automation" ]
version
1.0.0
connector
scripts/connectors/scraper_agent.py
description
Intelligent web scraping with automatic retry, rate limiting, and anti-detection via Firecrawl patterns.

03 SKILL.md 正文

Skill: Smart Scraper

Execution Steps

  1. Accept target URL(s) and data extraction schema (what fields to extract).
  2. Analyze target: static HTML vs JS-rendered vs API-backed.
  3. Select strategy: Simple HTTP for static, Firecrawl/Playwright for dynamic.
  4. Execute with guardrails: rate limiting, rotating user agents, retry with backoff.
  5. Extract and validate data against schema.
  6. Output: Clean CSV/JSON with extracted data.

Scraping Strategies

Target Type Tool Method
Static HTML HTTP + BeautifulSoup Direct parse
JS-rendered SPA Firecrawl / Playwright Headless browser
Paginated lists HTTP + pagination logic Follow next links
API-backed Direct API calls REST/GraphQL

Quality Validation

  • [ ] robots.txt respected
  • [ ] Rate limiting applied (max 1 req/second default)
  • [ ] Extracted data matches schema (no missing fields)
  • [ ] No PII inadvertently captured

04 同仓库更多 Skills