firecrawl-rate-limits
Implement Firecrawl rate limiting, backoff, and request queuing patterns. Use when handling 429 errors, implementing retry logic, or optimizing API request throughput for Firecrawl. Trigger with phrases like "firecrawl rate limit", "firecrawl throttling", "firecrawl 429", "firecrawl retry", "firecrawl backoff".
- License 许可
- MIT
- 最近更新
- 2026-07-13
- Stars
- ★ 1
- Forks
- 0
01 前言 Front Matter
- name
- firecrawl-rate-limits
- tags
- [ "saas", "firecrawl", "api" ]
- author
- Jeremy Longshore <jeremy@intentsolutions.io>
- license
- MIT
- version
- 1.0.0
- description
- Implement Firecrawl rate limiting, backoff, and request queuing patterns. Use when handling 429 errors, implementing retry logic, or optimizing API request throughput for Firecrawl. Trigger with phrases like "firecrawl rate limit", "firecrawl throttling", "firecrawl 429", "firecrawl retry", "firecrawl backoff".
- allowed-tools
- Read, Write, Edit
- compatible-with
- claude-code, codex, openclaw
03 SKILL.md 正文
Firecrawl Rate Limits
Overview
Firecrawl enforces rate limits per API key measured in requests per minute and concurrent connections. When exceeded, the API returns 429 Too Many Requests with a Retry-After header. This skill covers backoff strategies, request queuing, and proactive throttling.
Rate Limit Tiers
| Plan | Scrape RPM | Crawl Concurrency | Credits/Month |
|---|---|---|---|
| Free | 10 | 2 | 500 |
| Hobby | 20 | 3 | 3,000 |
| Standard | 50 | 5 | 50,000 |
| Growth | 100 | 10 | 500,000 |
| Scale | 500+ | 50+ | Custom |
Concurrent crawl jobs count against concurrency limits. If the queue is full, new jobs are rejected with 429.
Instructions
Step 1: Exponential Backoff with Jitter
import FirecrawlApp from "@mendable/firecrawl-js";
const firecrawl = new FirecrawlApp({
apiKey: process.env.FIRECRAWL_API_KEY!,
});
async function withBackoff<T>(
operation: () => Promise<T>,
config = { maxRetries: 5, baseDelayMs: 1000, maxDelayMs: 32000 }
): Promise<T> {
for (let attempt = 0; attempt <= config.maxRetries; attempt++) {
try {
return await operation();
} catch (error: any) {
if (attempt === config.maxRetries) throw error;
const status = error.statusCode || error.status;
// Only retry on 429 (rate limit) and 5xx (server error)
if (status && status !== 429 && status < 500) throw error;
// Exponential delay with random jitter to prevent thundering herd
const exponentialDelay = config.baseDelayMs * Math.pow(2, attempt);
const jitter = Math.random() * 500;
const delay = Math.min(exponentialDelay + jitter, config.maxDelayMs);
console.warn(`Rate limited (${status}). Retry ${attempt + 1}/${config.maxRetries} in ${delay.toFixed(0)}ms`);
await new Promise(r => setTimeout(r, delay));
}
}
throw new Error("Unreachable");
}
// Usage
const result = await withBackoff(() =>
firecrawl.scrapeUrl("https://example.com", { formats: ["markdown"] })
);
Step 2: Queue-Based Rate Limiting with p-queue
import PQueue from "p-queue";
// Limit to 5 concurrent requests, max 10 per second
const scrapeQueue = new PQueue({
concurrency: 5,
interval: 1000,
intervalCap: 10,
});
async function queuedScrape(url: string) {
return scrapeQueue.add(() =>
withBackoff(() =>
firecrawl.scrapeUrl(url, { formats: ["markdown"] })
)
);
}
// Scrape many URLs respecting rate limits
const urls = ["https://a.com", "https://b.com", "https://c.com"];
const results = await Promise.all(urls.map(url => queuedScrape(url)));
console.log(`Queue: ${scrapeQueue.pending} pending, ${scrapeQueue.size} queued`);
Step 3: Proactive Throttling (Pre-emptive)
class RateLimitTracker {
private requestTimes: number[] = [];
private windowMs: number;
private maxRequests: number;
constructor(maxRequests = 50, windowMs = 60000) {
this.maxRequests = maxRequests;
this.windowMs = windowMs;
}
async waitIfNeeded(): Promise<void> {
const now = Date.now();
this.requestTimes = this.requestTimes.filter(t => now - t < this.windowMs);
if (this.requestTimes.length >= this.maxRequests) {
const oldestInWindow = this.requestTimes[0];
const waitMs = this.windowMs - (now - oldestInWindow) + 100;
console.log(`Proactive throttle: waiting ${waitMs}ms to stay under ${this.maxRequests} RPM`);
await new Promise(r => setTimeout(r, waitMs));
}
this.requestTimes.push(Date.now());
}
}
const throttle = new RateLimitTracker(50, 60000); // 50 requests per minute
async function throttledScrape(url: string) {
await throttle.waitIfNeeded();
return firecrawl.scrapeUrl(url, { formats: ["markdown"] });
}
Step 4: Batch Scrape for Efficiency
// batchScrapeUrls is more efficient than individual scrapes
// It handles internal rate limiting and is cheaper on credits
const urls = [
"https://example.com/page1",
"https://example.com/page2",
"https://example.com/page3",
];
// Single API call instead of 3 separate scrapes
const batchResult = await firecrawl.batchScrapeUrls(urls, {
formats: ["markdown"],
});
console.log(`Batch scraped ${batchResult.data?.length} pages`);
Error Handling
| Header | Description | Action |
|---|---|---|
Retry-After |
Seconds to wait | Honor this exact value |
X-RateLimit-Limit |
Max requests per window | Use for proactive throttling |
X-RateLimit-Remaining |
Remaining in window | Slow down when < 5 |
X-RateLimit-Reset |
Reset timestamp | Wait until this time |
Examples
Monitor Rate Limit Usage
class RateLimitMonitor {
private remaining = Infinity;
private resetAt = new Date();
update(status: number, headers: Record<string, string>) {
if (headers["x-ratelimit-remaining"]) {
this.remaining = parseInt(headers["x-ratelimit-remaining"]);
}
if (headers["x-ratelimit-reset"]) {
this.resetAt = new Date(parseInt(headers["x-ratelimit-reset"]) * 1000);
}
if (this.remaining < 5) {
console.warn(`Low rate limit: ${this.remaining} remaining, resets at ${this.resetAt.toISOString()}`);
}
}
}
Resources
Next Steps
For security configuration, see firecrawl-security-basics.
04 同仓库更多 Skills
zustand-store-creator
Create zustand store creator operations. Auto-activating skill for Frontend Development. Triggers on: zustand store creator, zustand store creator Part of the Frontend Development skill category. Use when working with zustand store creator functionality. Trigger with phrases like "zustand store creator", "zustand creator", "zustand".
zero-trust-config-helper
Configure with zero trust config helper operations. Auto-activating skill for Security Advanced. Triggers on: zero trust config helper, zero trust config helper Part of the Security Advanced skill category. Use when configuring systems or services. Trigger with phrases like "zero trust config helper", "zero helper", "zero".
zapier-integration-helper
Assist with zapier integration helper operations. Auto-activating skill for Business Automation. Triggers on: zapier integration helper, zapier integration helper Part of the Business Automation skill category. Use when working with APIs or building integrations. Trigger with phrases like "zapier integration helper", "zapier helper", "zapier".
yaml-config-validator
Validate yaml config validator operations. Auto-activating skill for DevOps Basics. Triggers on: yaml config validator, yaml config validator Part of the DevOps Basics skill category. Use when configuring systems or services. Trigger with phrases like "yaml config validator", "yaml validator", "yaml".
xvary-stock-research
Thesis-driven equity analysis from public SEC EDGAR and market data; /analyze, /score, /compare workflows with bundled Python tools (Claude Code, Cursor, Codex).
xss-vulnerability-scanner
Scan xss vulnerability scanner operations. Auto-activating skill for Security Fundamentals. Triggers on: xss vulnerability scanner, xss vulnerability scanner Part of the Security Fundamentals skill category. Use when working with xss vulnerability scanner functionality. Trigger with phrases like "xss vulnerability scanner", "xss scanner", "xss".