AGENT SKILL

firecrawl-advanced-troubleshooting

Debug hard-to-diagnose Firecrawl issues with systematic isolation and evidence collection. Use when standard troubleshooting fails, investigating why scrapes return empty content, crawl jobs hang, or webhooks don't fire. Trigger with phrases like "firecrawl hard bug", "firecrawl mystery error", "firecrawl impossible to debug", "firecrawl deep debug", "firecrawl not scraping".

License 许可
MIT
最近更新
2026-07-13
Stars
★ 1
Forks
0

01 前言 Front Matter

name
firecrawl-advanced-troubleshooting
tags
[ "saas", "firecrawl", "debugging", "scaling" ]
author
Jeremy Longshore <jeremy@intentsolutions.io>
license
MIT
version
1.0.0
description
Debug hard-to-diagnose Firecrawl issues with systematic isolation and evidence collection. Use when standard troubleshooting fails, investigating why scrapes return empty content, crawl jobs hang, or webhooks don't fire. Trigger with phrases like "firecrawl hard bug", "firecrawl mystery error", "firecrawl impossible to debug", "firecrawl deep debug", "firecrawl not scraping".
allowed-tools
Read, Grep, Bash(curl:*), Bash(node:*)
compatible-with
claude-code, codex, openclaw

03 SKILL.md 正文

Firecrawl Advanced Troubleshooting

Overview

Deep debugging techniques for complex Firecrawl issues: empty scrapes on certain domains, crawl jobs that never complete, inconsistent extraction results, and webhook delivery failures. Uses systematic layer-by-layer isolation.

Instructions

Step 1: Minimal Reproduction

import FirecrawlApp from "@mendable/firecrawl-js";

// Strip everything down to the simplest failing case
async function minimalRepro() {
  const firecrawl = new FirecrawlApp({
    apiKey: process.env.FIRECRAWL_API_KEY!,
  });

  // Test 1: Can we scrape at all?
  console.log("Test 1: Basic scrape");
  const basic = await firecrawl.scrapeUrl("https://example.com", {
    formats: ["markdown"],
  });
  console.log(`  Success: ${basic.success}, Length: ${basic.markdown?.length}`);

  // Test 2: Does the target URL work?
  console.log("Test 2: Target URL");
  const target = await firecrawl.scrapeUrl("https://YOUR-FAILING-URL.com", {
    formats: ["markdown"],
  });
  console.log(`  Success: ${target.success}, Length: ${target.markdown?.length}`);

  // Test 3: With waitFor for JS rendering
  console.log("Test 3: With JS wait");
  const withWait = await firecrawl.scrapeUrl("https://YOUR-FAILING-URL.com", {
    formats: ["markdown"],
    waitFor: 10000,
    onlyMainContent: true,
  });
  console.log(`  Success: ${withWait.success}, Length: ${withWait.markdown?.length}`);

  // Test 4: With actions
  console.log("Test 4: With actions");
  const withActions = await firecrawl.scrapeUrl("https://YOUR-FAILING-URL.com", {
    formats: ["markdown", "screenshot"],
    actions: [
      { type: "wait", milliseconds: 3000 },
      { type: "scroll", direction: "down" },
      { type: "wait", milliseconds: 2000 },
    ],
  });
  console.log(`  Success: ${withActions.success}, Length: ${withActions.markdown?.length}`);
  // Screenshot will show what Firecrawl actually sees
}

Step 2: Layer-by-Layer Isolation

async function diagnose(url: string) {
  const firecrawl = new FirecrawlApp({ apiKey: process.env.FIRECRAWL_API_KEY! });
  const results: Array<{ test: string; pass: boolean; detail: string }> = [];

  // Layer 1: API connectivity
  try {
    await firecrawl.scrapeUrl("https://example.com", { formats: ["markdown"] });
    results.push({ test: "API connectivity", pass: true, detail: "OK" });
  } catch (e: any) {
    results.push({ test: "API connectivity", pass: false, detail: `${e.statusCode}: ${e.message}` });
    return results; // can't continue
  }

  // Layer 2: Target URL accessibility
  try {
    const result = await firecrawl.scrapeUrl(url, { formats: ["markdown"] });
    const hasContent = (result.markdown?.length || 0) > 50;
    results.push({
      test: "Target scrape",
      pass: result.success && hasContent,
      detail: `Success: ${result.success}, Chars: ${result.markdown?.length}, Status: ${result.metadata?.statusCode}`,
    });
  } catch (e: any) {
    results.push({ test: "Target scrape", pass: false, detail: e.message });
  }

  // Layer 3: Content quality
  try {
    const result = await firecrawl.scrapeUrl(url, {
      formats: ["markdown", "html"],
      onlyMainContent: true,
      waitFor: 5000,
    });
    const md = result.markdown || "";
    const isErrorPage = /404|403|access denied|captcha|blocked/i.test(md);
    results.push({
      test: "Content quality",
      pass: md.length > 100 && !isErrorPage,
      detail: `Chars: ${md.length}, Error page: ${isErrorPage}, Has headings: ${/^#{1,3}\s/m.test(md)}`,
    });
  } catch (e: any) {
    results.push({ test: "Content quality", pass: false, detail: e.message });
  }

  // Layer 4: Map endpoint (URL discovery)
  try {
    const map = await firecrawl.mapUrl(url);
    results.push({
      test: "Map endpoint",
      pass: (map.links?.length || 0) > 0,
      detail: `Found ${map.links?.length} URLs`,
    });
  } catch (e: any) {
    results.push({ test: "Map endpoint", pass: false, detail: e.message });
  }

  return results;
}

// Run diagnosis
const results = await diagnose("https://YOUR-URL.com");
console.table(results);

Step 3: Debug Empty Scrapes

// When scrapeUrl returns empty or thin markdown:
async function debugEmptyScrape(url: string) {
  const firecrawl = new FirecrawlApp({ apiKey: process.env.FIRECRAWL_API_KEY! });

  // Get all formats to understand what Firecrawl sees
  const result = await firecrawl.scrapeUrl(url, {
    formats: ["markdown", "html", "screenshot"],
    waitFor: 10000,
  });

  console.log("=== Scrape Debug ===");
  console.log(`URL: ${result.metadata?.sourceURL}`);
  console.log(`Status: ${result.metadata?.statusCode}`);
  console.log(`Markdown length: ${result.markdown?.length || 0}`);
  console.log(`HTML length: ${result.html?.length || 0}`);
  console.log(`Title: ${result.metadata?.title}`);

  // Check if HTML has content but markdown doesn't
  if ((result.html?.length || 0) > 1000 && (result.markdown?.length || 0) < 100) {
    console.log("DIAGNOSIS: HTML has content but markdown extraction failed");
    console.log("FIX: Content may be in iframes or shadow DOM. Try with actions.");
  }

  // Check for bot detection
  if (/captcha|cloudflare|access denied|please verify/i.test(result.html || "")) {
    console.log("DIAGNOSIS: Bot detection / CAPTCHA detected");
    console.log("FIX: Site blocks automated scraping. Contact Firecrawl support.");
  }

  return result;
}

Step 4: Debug Stuck Crawl Jobs

async function debugCrawlJob(jobId: string) {
  const firecrawl = new FirecrawlApp({ apiKey: process.env.FIRECRAWL_API_KEY! });

  const status = await firecrawl.checkCrawlStatus(jobId);
  console.log("=== Crawl Job Debug ===");
  console.log(`Status: ${status.status}`);
  console.log(`Completed: ${status.completed}/${status.total}`);
  console.log(`Error: ${status.error || "none"}`);

  if (status.status === "scraping" && status.completed === status.total) {
    console.log("DIAGNOSIS: All pages scraped but job not marked complete");
    console.log("FIX: This is a Firecrawl backend issue. Wait or start a new crawl.");
  }

  if (status.completed === 0 && status.status === "scraping") {
    console.log("DIAGNOSIS: Crawl started but no pages scraped");
    console.log("FIX: Check if start URL returns content. Try scrapeUrl first.");
  }
}

Step 5: Timing Analysis

async function timeScrape(url: string, iterations = 5) {
  const firecrawl = new FirecrawlApp({ apiKey: process.env.FIRECRAWL_API_KEY! });
  const times: number[] = [];

  for (let i = 0; i < iterations; i++) {
    const start = Date.now();
    await firecrawl.scrapeUrl(url, { formats: ["markdown"] });
    times.push(Date.now() - start);
  }

  times.sort((a, b) => a - b);
  console.log(`p50: ${times[Math.floor(times.length * 0.5)]}ms`);
  console.log(`p95: ${times[Math.floor(times.length * 0.95)]}ms`);
  console.log(`min: ${times[0]}ms, max: ${times[times.length - 1]}ms`);
}

Error Handling

Issue Cause Solution
Empty markdown, HTML exists Shadow DOM or iframes Use actions to interact with page
Scrape returns CAPTCHA Bot detection Try with mobile: true, contact Firecrawl
Crawl stuck at 0 pages Start URL blocked Verify URL loads in browser first
Inconsistent results JS rendering timing Increase waitFor, use selector-based wait
Webhook never fires URL unreachable Test with curl to your endpoint first

Support Escalation Template

Subject: [P1/P2/P3] [Brief description]

URL: [failing URL]
API Key prefix: fc-xxx (first 6 chars)
Timestamp: [ISO 8601]

Expected: [what should happen]
Actual: [what happens]

Diagnostic output: [paste from diagnose() above]
Screenshot: [if available from screenshot format]

Workarounds tried:
1. [what you tried] — result: [outcome]

Resources

Next Steps

For load testing, see firecrawl-load-scale.

04 同仓库更多 Skills

jukeyman
jukeyman/jukeyman-skills

zustand-store-creator

Create zustand store creator operations. Auto-activating skill for Frontend Development. Triggers on: zustand store creator, zustand store creator Part of the Frontend Development skill category. Use when working with zustand store creator functionality. Trigger with phrases like "zustand store creator", "zustand creator", "zustand".

★ 1
jukeyman
jukeyman/jukeyman-skills

zero-trust-config-helper

Configure with zero trust config helper operations. Auto-activating skill for Security Advanced. Triggers on: zero trust config helper, zero trust config helper Part of the Security Advanced skill category. Use when configuring systems or services. Trigger with phrases like "zero trust config helper", "zero helper", "zero".

★ 1
jukeyman
jukeyman/jukeyman-skills

zapier-integration-helper

Assist with zapier integration helper operations. Auto-activating skill for Business Automation. Triggers on: zapier integration helper, zapier integration helper Part of the Business Automation skill category. Use when working with APIs or building integrations. Trigger with phrases like "zapier integration helper", "zapier helper", "zapier".

★ 1
jukeyman
jukeyman/jukeyman-skills

yaml-config-validator

Validate yaml config validator operations. Auto-activating skill for DevOps Basics. Triggers on: yaml config validator, yaml config validator Part of the DevOps Basics skill category. Use when configuring systems or services. Trigger with phrases like "yaml config validator", "yaml validator", "yaml".

★ 1
jukeyman
jukeyman/jukeyman-skills

xvary-stock-research

Thesis-driven equity analysis from public SEC EDGAR and market data; /analyze, /score, /compare workflows with bundled Python tools (Claude Code, Cursor, Codex).

★ 1
jukeyman
jukeyman/jukeyman-skills

xss-vulnerability-scanner

Scan xss vulnerability scanner operations. Auto-activating skill for Security Fundamentals. Triggers on: xss vulnerability scanner, xss vulnerability scanner Part of the Security Fundamentals skill category. Use when working with xss vulnerability scanner functionality. Trigger with phrases like "xss vulnerability scanner", "xss scanner", "xss".

★ 1