firecrawl-performance-tuning
Optimize Firecrawl scraping performance with caching, batch scraping, and format selection. Use when experiencing slow scrapes, optimizing credit usage per page, or building high-throughput scraping pipelines. Trigger with phrases like "firecrawl performance", "optimize firecrawl", "firecrawl latency", "firecrawl caching", "firecrawl slow", "firecrawl batch".
- License 许可
- MIT
- 最近更新
- 2026-07-13
- Stars
- ★ 1
- Forks
- 0
01 前言 Front Matter
- name
- firecrawl-performance-tuning
- tags
- [ "saas", "firecrawl", "api", "performance" ]
- author
- Jeremy Longshore <jeremy@intentsolutions.io>
- license
- MIT
- version
- 1.0.0
- description
- Optimize Firecrawl scraping performance with caching, batch scraping, and format selection. Use when experiencing slow scrapes, optimizing credit usage per page, or building high-throughput scraping pipelines. Trigger with phrases like "firecrawl performance", "optimize firecrawl", "firecrawl latency", "firecrawl caching", "firecrawl slow", "firecrawl batch".
- allowed-tools
- Read, Write, Edit
- compatible-with
- claude-code, codex, openclaw
03 SKILL.md 正文
Firecrawl Performance Tuning
Overview
Optimize Firecrawl API performance by choosing efficient scraping modes, caching results, using batch endpoints, and minimizing unnecessary rendering. Key levers: format selection (markdown vs HTML vs screenshot), waitFor tuning, onlyMainContent, and batch vs individual scraping.
Latency Benchmarks
| Operation | Typical | With JS Wait | With Screenshot |
|---|---|---|---|
| scrapeUrl (markdown) | 2-5s | 5-10s | 8-15s |
| scrapeUrl (extract) | 3-8s | 8-15s | N/A |
| crawlUrl (10 pages) | 20-40s | 40-80s | N/A |
| mapUrl | 1-3s | N/A | N/A |
| batchScrapeUrls (10) | 10-20s | 20-40s | N/A |
Instructions
Step 1: Minimize Formats (Biggest Win)
import FirecrawlApp from "@mendable/firecrawl-js";
const firecrawl = new FirecrawlApp({
apiKey: process.env.FIRECRAWL_API_KEY!,
});
// SLOW: requesting everything
const slow = await firecrawl.scrapeUrl(url, {
formats: ["markdown", "html", "links", "screenshot"],
// screenshot + full HTML = 3-5x slower
});
// FAST: request only what you need
const fast = await firecrawl.scrapeUrl(url, {
formats: ["markdown"], // markdown only = fastest
onlyMainContent: true, // skip nav/footer/sidebar
});
Step 2: Tune waitFor for JS-Heavy Pages
// Default: no JS wait (fastest, works for static sites)
const staticResult = await firecrawl.scrapeUrl("https://docs.example.com", {
formats: ["markdown"],
// No waitFor needed — content is in initial HTML
});
// SPA/dynamic pages: add minimal wait
const spaResult = await firecrawl.scrapeUrl("https://app.example.com", {
formats: ["markdown"],
waitFor: 3000, // 3s — enough for most SPAs
onlyMainContent: true,
});
// Heavy interactive page: use actions instead of long wait
const heavyResult = await firecrawl.scrapeUrl("https://dashboard.example.com", {
formats: ["markdown"],
actions: [
{ type: "wait", selector: ".data-table" }, // wait for specific element
{ type: "scroll", direction: "down" }, // trigger lazy loading
],
});
Step 3: Cache Scraped Content
import { LRUCache } from "lru-cache";
import { createHash } from "crypto";
const scrapeCache = new LRUCache<string, any>({
max: 500, // max 500 cached pages
ttl: 3600000, // 1 hour TTL
});
async function cachedScrape(url: string) {
const key = createHash("md5").update(url).digest("hex");
const cached = scrapeCache.get(key);
if (cached) {
console.log(`Cache hit: ${url}`);
return cached;
}
const result = await firecrawl.scrapeUrl(url, {
formats: ["markdown"],
onlyMainContent: true,
});
if (result.success) {
scrapeCache.set(key, result);
}
return result;
}
// Typical savings: 50-80% credit reduction for repeated scrapes
Step 4: Use Batch Scrape for Multiple URLs
// SLOW: sequential individual scrapes
const urls = ["https://a.com", "https://b.com", "https://c.com"];
for (const url of urls) {
await firecrawl.scrapeUrl(url, { formats: ["markdown"] }); // 3 API calls
}
// FAST: single batch scrape call
const batchResult = await firecrawl.batchScrapeUrls(urls, {
formats: ["markdown"],
onlyMainContent: true,
});
// 1 API call, internally parallelized
Step 5: Map Before Crawl (Save Credits)
// EXPENSIVE: crawl everything, filter later
await firecrawl.crawlUrl("https://docs.example.com", { limit: 1000 });
// CHEAPER: map first (1 credit), then scrape only what you need
const map = await firecrawl.mapUrl("https://docs.example.com");
const apiDocs = (map.links || []).filter(url =>
url.includes("/api/") || url.includes("/reference/")
);
console.log(`Map: ${map.links?.length} total, ${apiDocs.length} relevant`);
// Batch scrape only relevant URLs
const result = await firecrawl.batchScrapeUrls(apiDocs.slice(0, 50), {
formats: ["markdown"],
});
Step 6: Measure Scrape Performance
async function timedScrape(url: string) {
const start = Date.now();
const result = await firecrawl.scrapeUrl(url, { formats: ["markdown"] });
const duration = Date.now() - start;
console.log({
url,
durationMs: duration,
contentLength: result.markdown?.length || 0,
success: result.success,
charsPerSecond: Math.round((result.markdown?.length || 0) / (duration / 1000)),
});
return result;
}
Error Handling
| Issue | Cause | Solution |
|---|---|---|
| Scrape > 10s | Screenshot or full HTML requested | Use formats: ["markdown"] only |
| Empty content | waitFor too short for SPA |
Increase or use actions with selector |
| High credit burn | Scraping same URLs repeatedly | Implement URL-based caching |
| Batch timeout | Too many URLs in one batch | Split into chunks of 50 |
| Cache stale data | TTL too long | Reduce TTL or add cache invalidation |
Examples
Performance Comparison Script
const url = "https://docs.firecrawl.dev";
// Compare different format configurations
for (const formats of [["markdown"], ["markdown", "html"], ["markdown", "html", "screenshot"]]) {
const start = Date.now();
await firecrawl.scrapeUrl(url, { formats: formats as any, onlyMainContent: true });
console.log(`${formats.join(",")}: ${Date.now() - start}ms`);
}
Resources
Next Steps
For cost optimization, see firecrawl-cost-tuning.
04 同仓库更多 Skills
zustand-store-creator
Create zustand store creator operations. Auto-activating skill for Frontend Development. Triggers on: zustand store creator, zustand store creator Part of the Frontend Development skill category. Use when working with zustand store creator functionality. Trigger with phrases like "zustand store creator", "zustand creator", "zustand".
zero-trust-config-helper
Configure with zero trust config helper operations. Auto-activating skill for Security Advanced. Triggers on: zero trust config helper, zero trust config helper Part of the Security Advanced skill category. Use when configuring systems or services. Trigger with phrases like "zero trust config helper", "zero helper", "zero".
zapier-integration-helper
Assist with zapier integration helper operations. Auto-activating skill for Business Automation. Triggers on: zapier integration helper, zapier integration helper Part of the Business Automation skill category. Use when working with APIs or building integrations. Trigger with phrases like "zapier integration helper", "zapier helper", "zapier".
yaml-config-validator
Validate yaml config validator operations. Auto-activating skill for DevOps Basics. Triggers on: yaml config validator, yaml config validator Part of the DevOps Basics skill category. Use when configuring systems or services. Trigger with phrases like "yaml config validator", "yaml validator", "yaml".
xvary-stock-research
Thesis-driven equity analysis from public SEC EDGAR and market data; /analyze, /score, /compare workflows with bundled Python tools (Claude Code, Cursor, Codex).
xss-vulnerability-scanner
Scan xss vulnerability scanner operations. Auto-activating skill for Security Fundamentals. Triggers on: xss vulnerability scanner, xss vulnerability scanner Part of the Security Fundamentals skill category. Use when working with xss vulnerability scanner functionality. Trigger with phrases like "xss vulnerability scanner", "xss scanner", "xss".