firecrawl
Scrape and crawl websites for AI with Firecrawl — scrape single URLs to clean Markdown/HTML, crawl entire sites with depth/path filters, extract structured data with LLM schema, use map to discover all URLs, and batch scrape multiple pages in parallel.
- License 许可
- Apache-2.0
- 最近更新
- 2026-11-14
- Stars
- ★ 13
- Forks
- 2
01 前言 Front Matter
- name
- firecrawl
- see_also
- [ "crawl4ai", "browser-use" ]
- triggers
- [ "firecrawl", "fire crawl", "firecrawl scrape", "firecrawl crawl", "firecrawl extract", "web scraping ai", "scrape to markdown", "crawl website ai", "firecrawl map", "firecrawl batch", "structured data extraction web", "web scraping llm" ]
- description
- Scrape and crawl websites for AI with Firecrawl — scrape single URLs to clean Markdown/HTML, crawl entire sites with depth/path filters, extract structured data with LLM schema, use map to discover all URLs, and batch scrape multiple pages in parallel.
- do_not_use_for
- [ "General web crawling without AI parsing — use Playwright or Puppeteer instead", "Web automation (clicking, forms) — use browser-use instead", "Simple fetch/parse — use BeautifulSoup + requests instead" ]
03 SKILL.md 正文
Firecrawl — Web Scraping for AI
Source: mendableai/firecrawl (AGPL-3.0) — turn websites into LLM-ready data
Why Firecrawl
- Returns clean Markdown (not raw HTML) — ready for LLM context
- Handles JavaScript-rendered pages (headless browser under the hood)
- Built-in LLM extraction with Pydantic schema
- Crawl entire sites, map all URLs, batch scrape in parallel
- Managed API or self-hosted Docker
Install
pip install firecrawl-py
# or
npm install @mendable/firecrawl-js
Scrape a Single URL
from firecrawl import FirecrawlApp
app = FirecrawlApp(api_key="fc-your-api-key")
# Basic scrape — returns Markdown
result = app.scrape_url(
"https://docs.anthropic.com/en/docs/about-claude/models",
params={"formats": ["markdown"]},
)
print(result["markdown"]) # clean markdown content
print(result["metadata"]["title"])
print(result["metadata"]["description"])
# Get both markdown + raw HTML
result = app.scrape_url(
"https://example.com",
params={"formats": ["markdown", "html"]},
)
Crawl an Entire Website
from firecrawl import FirecrawlApp
app = FirecrawlApp(api_key="fc-your-api-key")
# Async crawl (recommended for large sites)
crawl_job = app.async_crawl_url(
"https://docs.example.com",
params={
"limit": 100, # max pages
"maxDepth": 3, # crawl depth
"includePaths": ["/docs/.*"], # regex: only docs pages
"excludePaths": ["/blog/.*"], # regex: skip blog
"formats": ["markdown"],
"onlyMainContent": True, # strip nav/footer/ads
},
)
job_id = crawl_job["id"]
# Poll until done (or use webhook)
import time
while True:
status = app.check_crawl_status(job_id)
if status["status"] == "completed":
break
print(f"Crawled: {status['completed']}/{status['total']}")
time.sleep(5)
# Get results
pages = status["data"] # list of scraped pages
for page in pages:
print(page["url"], len(page["markdown"]))
Structured Data Extraction (LLM-powered)
from firecrawl import FirecrawlApp
from pydantic import BaseModel
app = FirecrawlApp(api_key="fc-your-api-key")
class ProductInfo(BaseModel):
name: str
price: float
rating: float
reviews_count: int
in_stock: bool
features: list[str]
result = app.scrape_url(
"https://www.amazon.com/dp/B08N5WRWNW",
params={
"formats": ["extract"],
"extract": {
"schema": ProductInfo.model_json_schema(),
"systemPrompt": "Extract product information accurately.",
},
},
)
product = ProductInfo(**result["extract"])
print(product.name, product.price, product.rating)
Map — Discover All URLs
from firecrawl import FirecrawlApp
app = FirecrawlApp(api_key="fc-your-api-key")
# Get all URLs from a site without scraping content
map_result = app.map_url(
"https://docs.example.com",
params={
"search": "authentication", # filter URLs by keyword
"limit": 500,
},
)
urls = map_result["links"]
print(f"Found {len(urls)} URLs")
for url in urls[:10]:
print(url)
Batch Scrape (Multiple URLs in Parallel)
from firecrawl import FirecrawlApp
app = FirecrawlApp(api_key="fc-your-api-key")
urls = [
"https://docs.example.com/page-1",
"https://docs.example.com/page-2",
"https://docs.example.com/page-3",
]
# Async batch scrape
batch_job = app.async_batch_scrape_urls(
urls,
params={"formats": ["markdown"], "onlyMainContent": True},
)
# Poll for completion
status = app.check_batch_scrape_status(batch_job["id"])
results = status["data"]
Webhook (Push Results)
from firecrawl import FirecrawlApp
app = FirecrawlApp(api_key="fc-your-api-key")
# Crawl with webhook — no polling needed
crawl_job = app.async_crawl_url(
"https://docs.example.com",
params={
"limit": 200,
"webhook": {
"url": "https://your-server.com/webhook/firecrawl",
"headers": {"Authorization": "Bearer your-token"},
"events": ["completed", "page"], # page = per-page results
},
},
)
With LangChain
from langchain_community.document_loaders import FireCrawlLoader
loader = FireCrawlLoader(
api_key="fc-your-api-key",
url="https://docs.example.com",
mode="crawl", # "scrape" | "crawl"
params={"limit": 50, "formats": ["markdown"]},
)
docs = loader.load()
# Each doc is a LangChain Document with page_content + metadata
for doc in docs[:3]:
print(doc.metadata["url"])
print(doc.page_content[:200])
Self-Hosted (Docker)
git clone https://github.com/mendableai/firecrawl
cd firecrawl
# Copy env file and configure
cp apps/api/.env.example apps/api/.env
# Edit .env: set OPENAI_API_KEY (for LLM extraction)
# Start all services
docker compose up -d
# Use with local endpoint
app = FirecrawlApp(api_key="test", api_url="http://localhost:3002")
JavaScript (Node.js)
import FirecrawlApp from "@mendable/firecrawl-js";
import { z } from "zod";
const app = new FirecrawlApp({ apiKey: "fc-your-api-key" });
// Scrape
const scrapeResult = await app.scrapeUrl("https://example.com", {
formats: ["markdown"],
});
console.log(scrapeResult.markdown);
// Extract with Zod schema
const schema = z.object({
company_mission: z.string(),
founding_year: z.number(),
team_size: z.number(),
});
const extractResult = await app.scrapeUrl("https://example.com/about", {
formats: ["extract"],
extract: { schema },
});
console.log(extractResult.extract);
Anti-Fake-Pass Checks
- [ ]
result["markdown"]— notresult.contentorresult.text - [ ] Async crawl needs
check_crawl_status(job_id)polling —async_crawl_urlonly starts the job - [ ]
onlyMainContent: truestrips nav/footer — important for clean LLM context - [ ]
extractformat requires LLM API key configured in Firecrawl (cloud or self-hosted) - [ ]
map_urlreturns URLs only — no content; usebatch_scrapeto get content for those URLs - [ ] Rate limits on free tier: 500 pages/month, 10 pages/min — use async + webhook for large crawls
04 同仓库更多 Skills
openai--sentry--sentry
Use when the user asks to inspect Sentry issues or events, summarize recent production errors, or pull basic Sentry health data via the Sentry API; perform read-only queries with the bundled script and require `SENTRY_AUTH_TOKEN`.
ito-basket-compare
Compare Itô prediction-market baskets against a user's knowledge base, portfolio notes, financial context, watchlist, or research thesis. Use for read-only basket comparison and gap analysis without investment advice or live trading.
implementing-network-traffic-baselining
Build network traffic baselines from NetFlow/IPFIX data using Python pandas for statistical analysis, z-score anomaly detection, and hourly/daily traffic pattern profiling
detecting-evasion-techniques-in-endpoint-logs
Detects defense evasion techniques used by adversaries in endpoint logs including log tampering, timestomping, process injection, and security tool disabling. Use when investigating suspicious endpoint behavior, building detection rules for evasion tactics, or conducting threat hunting for stealthy adversary activity. Activates for requests involving evasion detection, defense evasion analysis, log tampering detection, or MITRE ATT&CK TA0005.
implementing-microsegmentation-with-guardicore
Implementing microsegmentation using Akamai Guardicore Segmentation to map application dependencies, create granular network policies, visualize east-west traffic flows, and enforce least-privilege communication between workloads across data centers and cloud.
ramdisk-tmpfs-patterns
暂无简介