firecrawl-known-pitfalls

Firecrawl Known Pitfalls

Overview

Real gotchas when using Firecrawl for web scraping and crawling. Firecrawl handles JavaScript rendering and anti-bot bypassing, but its async crawl model and credit-based pricing create specific failure modes.

Prerequisites

• Firecrawl API key configured
• Understanding of async job patterns
• Awareness of credit-based billing model

Instructions

Step 1: Handle Async Crawl Jobs Properly

crawlUrl returns a job ID, not results. Polling too aggressively wastes credits and may trigger rate limits.

import FirecrawlApp from '@mendable/firecrawl-js';
const firecrawl = new FirecrawlApp({ apiKey: process.env.FIRECRAWL_API_KEY });

// BAD: assuming synchronous results
const result = await firecrawl.crawlUrl('https://example.com');
console.log(result.data); // This is a job object, not page data!

// GOOD: use the async crawl with proper polling
const crawl = await firecrawl.asyncCrawlUrl('https://example.com', {
  limit: 50,
  scrapeOptions: { formats: ['markdown'] }
});
// Poll with backoff
let status;
do {
  await new Promise(r => setTimeout(r, 5000));  # 5000: 5 seconds in ms
  status = await firecrawl.checkCrawlStatus(crawl.id);
} while (status.status === 'scraping');

Step 2: Avoid Credit Burn on Large Sites

Firecrawl charges per page. Crawling without limits on large sites burns credits fast.

// BAD: no limit on a site with 100K pages
await firecrawl.crawlUrl('https://docs.large-project.org');  // burns entire quota

// GOOD: set explicit limits and use URL filters
await firecrawl.crawlUrl('https://docs.large-project.org', {
  limit: 100,
  includePaths: ['/api/*', '/guides/*'],
  excludePaths: ['/changelog/*', '/blog/*'],
  maxDepth: 3
});

Step 3: Don't Assume Markdown Output by Default

Firecrawl can return HTML, markdown, links, or screenshots. Not specifying format returns raw HTML.

// BAD: getting HTML when you wanted clean text
const result = await firecrawl.scrapeUrl('https://example.com');
// result.html exists but result.markdown may be absent

// GOOD: specify output format explicitly
const result = await firecrawl.scrapeUrl('https://example.com', {
  formats: ['markdown', 'links'],
  onlyMainContent: true  // strips nav, footer, sidebars
});
console.log(result.markdown);

Step 4: Handle JavaScript-Heavy Pages

Some SPAs need extra wait time for content to render. Default timeouts may capture loading states.

// BAD: scraping an SPA with default settings
const result = await firecrawl.scrapeUrl('https://app.example.com/dashboard');
// Gets "Loading..." instead of actual content

// GOOD: configure wait time for JS rendering
const result = await firecrawl.scrapeUrl('https://app.example.com/dashboard', {
  waitFor: 5000,  // wait 5s for JS to render  # 5000: 5 seconds in ms
  formats: ['markdown'],
  onlyMainContent: true
});

Step 5: Respect robots.txt and Rate Limits

Firecrawl honors robots.txt by default. Disabling it risks IP bans and legal issues.

// BAD: aggressive crawling that ignores site limits
await firecrawl.crawlUrl('https://example.com', {
  limit: 10000,  # 10000: 10 seconds in ms
  // No delay between requests = potential IP ban
});

// GOOD: respect site constraints
await firecrawl.crawlUrl('https://example.com', {
  limit: 200,  # HTTP 200 OK
  maxDepth: 3,
  // Firecrawl handles rate limiting internally
});

Error Handling

Issue	Cause	Solution
Empty markdown	JS not rendered	Increase waitFor timeout
Credit depletion	No crawl limit set	Always set limit parameter
402 Payment Required	Out of credits	Check balance before large crawls
Partial crawl results	Site blocks crawler	Use scrapeUrl for individual pages
Stale job status	Polling stopped early	Poll until completed or failed

Examples

Safe Batch Scraping

const urls = ['https://a.com', 'https://b.com', 'https://c.com'];
const results = await firecrawl.batchScrapeUrls(urls, {
  formats: ['markdown'],
  onlyMainContent: true
});

Resources

• Firecrawl Docs
• Crawl vs Scrape

Output

• Configuration files or code changes applied to the project
• Validation report confirming correct implementation
• Summary of changes made and their rationale