[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7BNPSCAPZ9GJVTBC4VR7CDF":3,"skill-same:01M7BNPSCAPZ9GJVTBC4VR7CDF":115},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":24,"commitSha":25,"skillDirectory":26,"forksCount":14,"repositoryHtmlUrl":27,"repositoryHomepageUrl":6,"ownerHtmlUrl":28,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":29,"install":30,"resources":43},"01M7BNPSCAPZ9GJVTBC4VR7CDF","web-scrape",null,"Search the web semantically and pull clean content from pages, including JavaScript-heavy ones. Use whenever you need live web data during a task: finding docs, API references, code examples, articles, papers, or current info (Exa semantic search), or extracting the readable content of a specific URL — especially a JS-rendered, dynamic, or paywalled-by-script page a plain fetch returns empty (Firecrawl). Trigger even when the tools aren't named: \"scrape this page\", \"search the web for\", \"pull the content from this URL\", \"find docs\u002Farticles about\", \"extract data from this site\", \"what does this page say\", \"research X online\", \"get me the latest on\". Also bundles source clients — keyless Reddit (practitioner sentiment), Academic (OpenAlex papers + Unpaywall free PDFs, citations), Substack (newsletters, long-form), and paid Supadata (YouTube search + multi-platform transcripts) — plus a multi-source deep-research method: reach here for \"what do people on Reddit say\", \"find research papers \u002F citations on\", \"read this Substack \u002F newsletter\", \"YouTube transcript\", \"video transcript\", or \"do a rigorous deep research pass on X\".","skyremote\u002Fskool-navaigate-install","skyremote","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F136350643?v=4","01M7BNKJA14Z25S7KHGFQ149FK","SkyRemote","01M7BNKJAWYJKDVNP8N33RWF04",0,"MIT","https:\u002F\u002Fgithub.com\u002Fskyremote\u002Fskool-navaigate-install\u002Fblob\u002Fmain\u002Fplugins\u002Fshelf-deep-research\u002Fskills\u002Fweb-scrape\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: web-scrape\ndescription: >-\n  Search the web semantically and pull clean content from pages, including\n  JavaScript-heavy ones. Use whenever you need live web data during a task:\n  finding docs, API references, code examples, articles, papers, or current info\n  (Exa semantic search), or extracting the readable content of a specific URL —\n  especially a JS-rendered, dynamic, or paywalled-by-script page a plain fetch\n  returns empty (Firecrawl). Trigger even when the tools aren't named: \"scrape\n  this page\", \"search the web for\", \"pull the content from this URL\", \"find\n  docs\u002Farticles about\", \"extract data from this site\", \"what does this page say\",\n  \"research X online\", \"get me the latest on\". Also bundles source clients —\n  keyless Reddit (practitioner sentiment), Academic (OpenAlex papers + Unpaywall\n  free PDFs, citations), Substack (newsletters, long-form), and paid Supadata\n  (YouTube search + multi-platform transcripts) — plus a multi-source\n  deep-research method: reach here for \"what do people on Reddit say\", \"find\n  research papers \u002F citations on\", \"read this Substack \u002F newsletter\", \"YouTube\n  transcript\", \"video transcript\", or \"do a rigorous deep research pass on X\".\n---\n\n# \u002Fweb-scrape\n\nCommunity copy of how we read the web today. **Their** keys in\n`~\u002F.claude\u002F.env` — never Daniel's. Reddit via Composio uses\n`COMPOSIO_USER_ID` from that file (falls back to the machine user). After\nthis command, they type `\u002Faios` and pick the next module.\n\nTwo providers behind one helper script, picked by job:\n\n- **Exa** — *semantic* web search. You describe what you want in natural\n  language and get back the most relevant pages, optionally with excerpts,\n  full text, summaries, or a grounded structured answer. Use it to **find**\n  things and to answer \"where is X \u002F what's the best source on Y\".\n- **Firecrawl** — a real headless-browser **scraper**. It executes JavaScript\n  and returns clean markdown. Use it to **read** a specific URL, especially a\n  single-page app, infinite-scroll page, or anything where the raw HTML is a\n  shell that fills in via JS.\n\nRule of thumb: **Exa to discover, Firecrawl to extract.** If you already have\nthe URL and just need its rendered text, go straight to `scrape`. If you need\nthe *content of pages Exa already found*, Exa can return it inline (`--text`)\n— you don't always need a second Firecrawl call.\n\n## Setup (one-time)\n\nKeys live in `~\u002F.claude\u002F.env`, one `KEY=VALUE` per line:\n\n```\nEXA_API_KEY=...\nFIRECRAWL_API_KEY=fc-...\n```\n\nThe helper auto-loads that file, so you don't need to export anything. It uses\n`requests` over REST — no `exa-py` or `firecrawl` SDK install required. If a\nkey is missing the script tells you exactly which one and where to put it.\n\n## The helper\n\nAll web work goes through one script. It lives in the `scripts\u002F` folder next to this SKILL.md\n(`scripts\u002Fwebscrape.py`). Set `S` to that folder once, then run it with `python3`:\n\n```bash\nS=\"\u003Cthe scripts\u002F folder next to this SKILL.md>\"\npython3 \"$S\u002Fwebscrape.py\" \u003Ccommand> [options]\n```\n\nAdd `--json` to any command for the raw API response (use when you need\nfields the human view drops, e.g. publish dates, scores, screenshots, or\n`output`\u002F`grounding` from a structured search). Run any command with\n`--help` for its full option list.\n\n### Exa — search\n\n```bash\n# Default: balanced semantic search, query-relevant highlights per result.\npython3 ...\u002Fwebscrape.py search \"Next.js route handler authentication example\"\n\n# Full page text capped to control tokens; restrict to authoritative sources.\npython3 ...\u002Fwebscrape.py search \"supabase row level security policies\" \\\n  --text --max-chars 6000 --include-domains supabase.com github.com\n\n# Research-grade synthesis (slower) with a grounded structured payload.\npython3 ...\u002Fwebscrape.py search \"compare vector databases for RAG in 2026\" \\\n  --type deep --system-prompt \"Prefer official docs; collapse duplicates.\" \\\n  --output-schema '{\"type\":\"object\",\"required\":[\"options\"],\"properties\":{\"options\":{\"type\":\"array\",\"items\":{\"type\":\"object\",\"properties\":{\"name\":{\"type\":\"string\"},\"tradeoff\":{\"type\":\"string\"}}}}}}' \\\n  --json\n```\n\nPicking `--type` (see `references\u002Fexa.md` for the full table):\n\n| Want | Use |\n|------|-----|\n| Most queries — balanced relevance\u002Fspeed | `auto` *(default)* |\n| Chat\u002Fautocomplete\u002Fquick lookup | `instant` or `fast` |\n| Research, enrichment, hard comparisons | `deep` or `deep-reasoning` |\n\nContent flags (`--highlights` default, `--text`, `--summary`): start with\n**highlights** for agent workflows — they keep token usage predictable. Reach\nfor `--text --max-chars N` only when downstream reasoning needs broad page\ncontext, and always cap it.\n\n### Exa — contents \u002F answer\n\n```bash\n# You already have URLs and want their clean content (no search step).\npython3 ...\u002Fwebscrape.py contents https:\u002F\u002Fa.com\u002Fpost https:\u002F\u002Fb.com\u002Fdoc --text --max-chars 8000\n\n# Question-first: a grounded answer with citations.\npython3 ...\u002Fwebscrape.py answer \"What changed in the React 19 compiler?\"\n```\n\n### Firecrawl — scrape (JS-heavy pages)\n\n```bash\n# Render a page and print clean markdown (main content only).\npython3 ...\u002Fwebscrape.py scrape \"https:\u002F\u002Fexample.com\u002Fspa-dashboard\"\n\n# Give a slow SPA time to hydrate, and keep the full page (nav\u002Ffooter too).\npython3 ...\u002Fwebscrape.py scrape \"https:\u002F\u002Fexample.com\u002Fapp\" --wait 3000 --full\n\n# Other formats when you need them.\npython3 ...\u002Fwebscrape.py scrape \"https:\u002F\u002Fexample.com\" --formats markdown links\n```\n\n`scrape` already executes JavaScript, so it handles most dynamic pages on its\nown. If a page needs **clicks, form fills, or login** before the content\nexists, that's Firecrawl's `actions` feature — not wired into this helper;\nsee `references\u002Ffirecrawl.md` for how to add an `actions` array, or fall back\nto the project's interactive-browser tooling.\n\n### Firecrawl — search\n\n```bash\n# Firecrawl's own web search; --scrape also pulls each result to markdown.\npython3 ...\u002Fwebscrape.py crawl-search \"best practices structured logging\" --num 5 --scrape\n```\n\nExa's `search` is the better default for *finding* things (true semantic\nranking). Use `crawl-search` when you specifically want Firecrawl's\nindex + one-shot scrape of the hits.\n\n## Source clients (Reddit \u002F Academic \u002F Substack free; Supadata paid)\n\nBeyond the open web, the source clients live in `sources\u002F` inside\nthe `scripts\u002F` folder next to this SKILL.md (`$S\u002Fsources\u002F`). Each is a\nself-contained CLI with the same conventions as `webscrape.py` (subcommands,\n`--json` before or after the subcommand, human-readable default output). Academic\nand Substack are keyless; **Reddit** routes through Composio's managed toolkit\n(read-only OAuth, `COMPOSIO_API_KEY` in `~\u002F.claude\u002F.env`) with the keyless direct\npublic-JSON client as an automatic fallback; **Supadata** is **paid\n(credit-metered)** and needs `SUPADATA_API_KEY`. Reach for them when the *kind* of\nsource matters:\n\n| Source | Script | Reach for it when you want… |\n|--------|--------|------------------------------|\n| **Reddit** | `scripts\u002Fsources\u002Freddit.py` | practitioner sentiment, lived failure modes, \"what actually broke \u002F what did people pay\" |\n| **Academic** | `scripts\u002Fsources\u002Facademic.py` | primary research, citation-weighted credibility, the research-vs-production gap (OpenAlex + Unpaywall free PDFs) |\n| **Substack** | `scripts\u002Fsources\u002Fsubstack.py` | named-author long-form analysis, operator essays, thought leadership |\n| **Supadata** *(paid, credit-metered)* | `scripts\u002Fsources\u002Fsupadata.py` | **video\u002Ftranscript** evidence — YouTube search + multi-platform transcripts (YouTube\u002FTikTok\u002FIG\u002FX\u002FFB\u002Ffile): \"what are people saying on YouTube\", the actual words of a talk\u002Finterview\u002Fdemo |\n| **X-Search** *(paid; dormant until xAI funded)* | `scripts\u002Fsources\u002Fxsearch.py` | **X\u002FTwitter real-time** — \"what is X\u002FTwitter saying about …\", real-time X sentiment, recent posts from specific handles (via xAI Grok's X Search tool). **Wired but dormant** until the xAI account has billing\u002Fcredit (console.x.ai) |\n\n```bash\n# Reddit — via Composio managed toolkit (read-only OAuth, runs off Composio infra\n# so no IP block); auto-falls back to the direct public-JSON client if unconnected\npython3 scripts\u002Fsources\u002Freddit.py research \"AI coding agents in production\" --threads 5\npython3 scripts\u002Fsources\u002Freddit.py thread \"\u003Creddit-url>\" --comments 25\n\n# Academic — OpenAlex + Unpaywall, free (set OPENALEX_EMAIL in ~\u002F.claude\u002F.env, optional)\npython3 scripts\u002Fsources\u002Facademic.py search \"RAG evaluation\" --from-year 2023 --sort cites\npython3 scripts\u002Fsources\u002Facademic.py pdf 10.1038\u002Fs41586-023-06291-2   # free PDF via Unpaywall\n\n# Substack — discovery via Exa\u002FFirecrawl, extraction via Substack's public API\npython3 scripts\u002Fsources\u002Fsubstack.py search \"AI agents in production\" --limit 10\npython3 scripts\u002Fsources\u002Fsubstack.py post \u003Cpublication> \u003Cslug>\n\n# Supadata — PAID\u002Fmetered: YouTube search + transcript; research is conservative by default\npython3 scripts\u002Fsources\u002Fsupadata.py search \"retrieval augmented generation\" --limit 10\npython3 scripts\u002Fsources\u002Fsupadata.py transcript \"https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=...\" --lang en\npython3 scripts\u002Fsources\u002Fsupadata.py research \"AI coding agents\" --videos 3   # metered — use when video evidence is wanted\n\n# X-Search — PAID, DORMANT until xAI funded (console.x.ai): real-time X\u002FTwitter via Grok's X Search tool\npython3 scripts\u002Fsources\u002Fxsearch.py search \"what is X saying about Cloudflare Workers\" --max-results 5\npython3 scripts\u002Fsources\u002Fxsearch.py research \"AI coding agents\" --max-results 10   # same bundle shape as the others\n```\n\nNotes: **Academic** uses no key for search\u002Fcitations\u002FPDF; `--full-text` and\n**Substack** discovery reuse the Exa\u002FFirecrawl keys already in `~\u002F.claude\u002F.env`.\n**Reddit** routes through Composio's managed Reddit toolkit (managed OAuth,\nread-only, `COMPOSIO_API_KEY` in `~\u002F.claude\u002F.env`, `user_id from COMPOSIO_USER_ID in ~\u002F.claude\u002F.env`) so it runs\noff Composio's infra and isn't hit by Reddit's outbound-IP block; if the Reddit\nconnection isn't authorised it auto-falls back to the direct public-JSON client\n(which can 403 on cloud\u002FCI networks). **Supadata** is the only **paid** source\n(`SUPADATA_API_KEY`, credit-metered) — handles both sync and async (job-id\npolling) transcripts; keep `research` small and reach for it only when\nvideo\u002Ftranscript evidence is specifically wanted, not on every pass. Full details\nper source: `references\u002Freddit.md`, `references\u002Facademic.md`,\n`references\u002Fsubstack.md`, `references\u002Fsupadata.md`.\n\n*Source status \u002F future stubs:* **Supadata** is **live** (paid). **X-Search**\n(`XAI_API_KEY`, paid) is now **wired but dormant** — the code is complete and the dry\ncall reaches xAI correctly; it activates the moment the xAI account has billing\u002Fcredit\n(console.x.ai), with **no code change** (details: `references\u002Fxsearch.md`). **Podcast**\nhas **no dedicated endpoint** and is covered by Supadata's YouTube search + transcript\n(or a direct media-file URL). Everything except Supadata and X-Search stays free.\n\n## Running a rigorous multi-source research pass\n\nWhen the task is \"research X properly\" rather than one lookup, don't ad-hoc it —\nfollow `references\u002Fresearch-method.md`. It captures a disciplined flow (scope →\nrecon with HOT\u002FWARM\u002FCOLD per source → human checkpoint → parallel topic agents →\ncritic pass → confidence-labelled synthesis), a **source-scoring rubric**\n(recency \u002F source type \u002F specificity \u002F independence) and ready prompt templates,\nall driving **our existing `deep-research` harness and `chief-of-staff` fan-out**\nover the sources above. It adds no new orchestrator and no database.\n\n## Choosing a path\n\n```\nNeed live web data\n├─ Have a specific URL to read?\n│   ├─ Page is plain\u002Fstatic and small  → a normal fetch is fine\n│   └─ Page is JS-heavy \u002F dynamic \u002F SPA → scrape\n├─ Need to FIND pages by meaning       → search   (Exa)\n├─ Have URLs, just want their content  → contents (Exa)  or scrape (Firecrawl)\n└─ Want a direct answer with citations → answer   (Exa)\n```\n\n## Common mistakes (and the fix)\n\nThese are real API gotchas — avoid them:\n\n- On **Exa `\u002Fsearch`**, `text`\u002F`highlights`\u002F`summary` must be **nested under\n  `contents`** (the script does this). On **`\u002Fcontents`** they're **top-level**.\n- `useAutoprompt`, `numSentences`, `highlightsPerUrl`, `tokensNum`,\n  `livecrawl: \"always\"`, `includeUrls`\u002F`excludeUrls` are **deprecated or\n  non-existent** — use `--max-age-hours 0` for freshness and\n  `--include-domains`\u002F`--exclude-domains` for filtering.\n- Uncapped `--text` can blow up context. Prefer `--highlights`, or\n  `--text --max-chars N`.\n- Slow\u002Fempty `scrape` result on a dynamic page → add `--wait 2000`-`5000` to\n  let JS render before extraction.\n- `401` → the matching key is missing or wrong in `~\u002F.claude\u002F.env`.\n\n## Deeper reference\n\n- `references\u002Fexa.md` — search types, content config, `outputSchema`,\n  `maxAgeHours`, `\u002Fcontents` & `\u002Fanswer`, troubleshooting.\n- `references\u002Ffirecrawl.md` — scrape body fields, `formats`, `actions` for\n  interaction, the search endpoint, response shape.\n- `references\u002Freddit.md` — Reddit via Composio managed toolkit (read tools, execute contract), CLI, the direct public-JSON fallback and its 403\u002FIP gotcha.\n- `references\u002Facademic.md` — OpenAlex + Unpaywall CLI, paper fields, citation graph.\n- `references\u002Fsubstack.md` — Substack discovery + extraction CLI, response shape.\n- `references\u002Fsupadata.md` — Supadata video\u002Ftranscript CLI (YouTube search, multi-platform transcripts, sync-vs-async, cost note).\n- `references\u002Fxsearch.md` — X-Search CLI (xAI Grok X Search tool: Responses API shape, `x_search` fields, model id, cost note, the dormant-until-funded gate).\n- `references\u002Fresearch-method.md` — the full multi-source research flow, source-scoring\n  rubric, and recon\u002Ftopic\u002Fcritic\u002Fsynthesis prompt templates.\n\n**Canonical sources of truth** (fetch these if behaviour ever contradicts the\nabove, and report drift back to the user):\n- Exa: https:\u002F\u002Fexa.ai\u002Fdocs\u002Freference\u002Fsearch-api-guide-for-coding-agents\n- Firecrawl: https:\u002F\u002Fdocs.firecrawl.dev\n","# \u002Fweb-scrape\n\nCommunity copy of how we read the web today. **Their** keys in\n`~\u002F.claude\u002F.env` — never Daniel's. Reddit via Composio uses\n`COMPOSIO_USER_ID` from that file (falls back to the machine user). After\nthis command, they type `\u002Faios` and pick the next module.\n\nTwo providers behind one helper script, picked by job:\n\n- **Exa** — *semantic* web search. You describe what you want in natural\n  language and get back the most relevant pages, optionally with excerpts,\n  full text, summaries, or a grounded structured answer. Use it to **find**\n  things and to answer \"where is X \u002F what's the best source on Y\".\n- **Firecrawl** — a real headless-browser **scraper**. It executes JavaScript\n  and returns clean markdown. Use it to **read** a specific URL, especially a\n  single-page app, infinite-scroll page, or anything where the raw HTML is a\n  shell that fills in via JS.\n\nRule of thumb: **Exa to discover, Firecrawl to extract.** If you already have\nthe URL and just need its rendered text, go straight to `scrape`. If you need\nthe *content of pages Exa already found*, Exa can return it inline (`--text`)\n— you don't always need a second Firecrawl call.\n\n## Setup (one-time)\n\nKeys live in `~\u002F.claude\u002F.env`, one `KEY=VALUE` per line:\n\n```\nEXA_API_KEY=...\nFIRECRAWL_API_KEY=fc-...\n```\n\nThe helper auto-loads that file, so you don't need to export anything. It uses\n`requests` over REST — no `exa-py` or `firecrawl` SDK install required. If a\nkey is missing the script tells you exactly which one and where to put it.\n\n## The helper\n\nAll web work goes through one script. It lives in the `scripts\u002F` folder next to this SKILL.md\n(`scripts\u002Fwebscrape.py`). Set `S` to that folder once, then run it with `python3`:\n\n```bash\nS=\"\u003Cthe scripts\u002F folder next to this SKILL.md>\"\npython3 \"$S\u002Fwebscrape.py\" \u003Ccommand> [options]\n```\n\nAdd `--json` to any command for the raw API response (use when you need\nfields the human view drops, e.g. publish dates, scores, screenshots, or\n`output`\u002F`grounding` from a structured search). Run any command with\n`--help` for its full option list.\n\n### Exa — search\n\n```bash\n# Default: balanced semantic search, query-relevant highlights per result.\npython3 ...\u002Fwebscrape.py search \"Next.js route handler authentication example\"\n\n# Full page text capped to control tokens; restrict to authoritative sources.\npython3 ...\u002Fwebscrape.py search \"supabase row level security policies\" \\\n  --text --max-chars 6000 --include-domains supabase.com github.com\n\n# Research-grade synthesis (slower) with a grounded structured payload.\npython3 ...\u002Fwebscrape.py search \"compare vector databases for RAG in 2026\" \\\n  --type deep --system-prompt \"Prefer official docs; collapse duplicates.\" \\\n  --output-schema '{\"type\":\"object\",\"required\":[\"options\"],\"properties\":{\"options\":{\"type\":\"array\",\"items\":{\"type\":\"object\",\"properties\":{\"name\":{\"type\":\"string\"},\"tradeoff\":{\"type\":\"string\"}}}}}}' \\\n  --json\n```\n\nPicking `--type` (see `references\u002Fexa.md` for the full table):\n\n| Want | Use |\n|------|-----|\n| Most queries — balanced relevance\u002Fspeed | `auto` *(default)* |\n| Chat\u002Fautocomplete\u002Fquick lookup | `instant` or `fast` |\n| Research, enrichment, hard comparisons | `deep` or `deep-reasoning` |\n\nContent flags (`--highlights` default, `--text`, `--summary`): start with\n**highlights** for agent workflows — they keep token usage predictable. Reach\nfor `--text --max-chars N` only when downstream reasoning needs broad page\ncontext, and always cap it.\n\n### Exa — contents \u002F answer\n\n```bash\n# You already have URLs and want their clean content (no search step).\npython3 ...\u002Fwebscrape.py contents https:\u002F\u002Fa.com\u002Fpost https:\u002F\u002Fb.com\u002Fdoc --text --max-chars 8000\n\n# Question-first: a grounded answer with citations.\npython3 ...\u002Fwebscrape.py answer \"What changed in the React 19 compiler?\"\n```\n\n### Firecrawl — scrape (JS-heavy pages)\n\n```bash\n# Render a page and print clean markdown (main content only).\npython3 ...\u002Fwebscrape.py scrape \"https:\u002F\u002Fexample.com\u002Fspa-dashboard\"\n\n# Give a slow SPA time to hydrate, and keep the full page (nav\u002Ffooter too).\npython3 ...\u002Fwebscrape.py scrape \"https:\u002F\u002Fexample.com\u002Fapp\" --wait 3000 --full\n\n# Other formats when you need them.\npython3 ...\u002Fwebscrape.py scrape \"https:\u002F\u002Fexample.com\" --formats markdown links\n```\n\n`scrape` already executes JavaScript, so it handles most dynamic pages on its\nown. If a page needs **clicks, form fills, or login** before the content\nexists, that's Firecrawl's `actions` feature — not wired into this helper;\nsee `references\u002Ffirecrawl.md` for how to add an `actions` array, or fall back\nto the project's interactive-browser tooling.\n\n### Firecrawl — search\n\n```bash\n# Firecrawl's own web search; --scrape also pulls each result to markdown.\npython3 ...\u002Fwebscrape.py crawl-search \"best practices structured logging\" --num 5 --scrape\n```\n\nExa's `search` is the better default for *finding* things (true semantic\nranking). Use `crawl-search` when you specifically want Firecrawl's\nindex + one-shot scrape of the hits.\n\n## Source clients (Reddit \u002F Academic \u002F Substack free; Supadata paid)\n\nBeyond the open web, the source clients live in `sources\u002F` inside\nthe `scripts\u002F` folder next to this SKILL.md (`$S\u002Fsources\u002F`). Each is a\nself-contained CLI with the same conventions as `webscrape.py` (subcommands,\n`--json` before or after the subcommand, human-readable default output). Academic\nand Substack are keyless; **Reddit** routes through Composio's managed toolkit\n(read-only OAuth, `COMPOSIO_API_KEY` in `~\u002F.claude\u002F.env`) with the keyless direct\npublic-JSON client as an automatic fallback; **Supadata** is **paid\n(credit-metered)** and needs `SUPADATA_API_KEY`. Reach for them when the *kind* of\nsource matters:\n\n| Source | Script | Reach for it when you want… |\n|--------|--------|------------------------------|\n| **Reddit** | `scripts\u002Fsources\u002Freddit.py` | practitioner sentiment, lived failure modes, \"what actually broke \u002F what did people pay\" |\n| **Academic** | `scripts\u002Fsources\u002Facademic.py` | primary research, citation-weighted credibility, the research-vs-production gap (OpenAlex + Unpaywall free PDFs) |\n| **Substack** | `scripts\u002Fsources\u002Fsubstack.py` | named-author long-form analysis, operator essays, thought leadership |\n| **Supadata** *(paid, credit-metered)* | `scripts\u002Fsources\u002Fsupadata.py` | **video\u002Ftranscript** evidence — YouTube search + multi-platform transcripts (YouTube\u002FTikTok\u002FIG\u002FX\u002FFB\u002Ffile): \"what are people saying on YouTube\", the actual words of a talk\u002Finterview\u002Fdemo |\n| **X-Search** *(paid; dormant until xAI funded)* | `scripts\u002Fsources\u002Fxsearch.py` | **X\u002FTwitter real-time** — \"what is X\u002FTwitter saying about …\", real-time X sentiment, recent posts from specific handles (via xAI Grok's X Search tool). **Wired but dormant** until the xAI account has billing\u002Fcredit (console.x.ai) |\n\n```bash\n# Reddit — via Composio managed toolkit (read-only OAuth, runs off Composio infra\n# so no IP block); auto-falls back to the direct public-JSON client if unconnected\npython3 scripts\u002Fsources\u002Freddit.py research \"AI coding agents in production\" --threads 5\npython3 scripts\u002Fsources\u002Freddit.py thread \"\u003Creddit-url>\" --comments 25\n\n# Academic — OpenAlex + Unpaywall, free (set OPENALEX_EMAIL in ~\u002F.claude\u002F.env, optional)\npython3 scripts\u002Fsources\u002Facademic.py search \"RAG evaluation\" --from-year 2023 --sort cites\npython3 scripts\u002Fsources\u002Facademic.py pdf 10.1038\u002Fs41586-023-06291-2   # free PDF via Unpaywall\n\n# Substack — discovery via Exa\u002FFirecrawl, extraction via Substack's public API\npython3 scripts\u002Fsources\u002Fsubstack.py search \"AI agents in production\" --limit 10\npython3 scripts\u002Fsources\u002Fsubstack.py post \u003Cpublication> \u003Cslug>\n\n# Supadata — PAID\u002Fmetered: YouTube search + transcript; research is conservative by default\npython3 scripts\u002Fsources\u002Fsupadata.py search \"retrieval augmented generation\" --limit 10\npython3 scripts\u002Fsources\u002Fsupadata.py transcript \"https:\u002F\u002Fwww.youtube.com\u002Fwatch?v=...\" --lang en\npython3 scripts\u002Fsources\u002Fsupadata.py research \"AI coding agents\" --videos 3   # metered — use when video evidence is wanted\n\n# X-Search — PAID, DORMANT until xAI funded (console.x.ai): real-time X\u002FTwitter via Grok's X Search tool\npython3 scripts\u002Fsources\u002Fxsearch.py search \"what is X saying about Cloudflare Workers\" --max-results 5\npython3 scripts\u002Fsources\u002Fxsearch.py research \"AI coding agents\" --max-results 10   # same bundle shape as the others\n```\n\nNotes: **Academic** uses no key for search\u002Fcitations\u002FPDF; `--full-text` and\n**Substack** discovery reuse the Exa\u002FFirecrawl keys already in `~\u002F.claude\u002F.env`.\n**Reddit** routes through Composio's managed Reddit toolkit (managed OAuth,\nread-only, `COMPOSIO_API_KEY` in `~\u002F.claude\u002F.env`, `user_id from COMPOSIO_USER_ID in ~\u002F.claude\u002F.env`) so it runs\noff Composio's infra and isn't hit by Reddit's outbound-IP block; if the Reddit\nconnection isn't authorised it auto-falls back to the direct public-JSON client\n(which can 403 on cloud\u002FCI networks). **Supadata** is the only **paid** source\n(`SUPADATA_API_KEY`, credit-metered) — handles both sync and async (job-id\npolling) transcripts; keep `research` small and reach for it only when\nvideo\u002Ftranscript evidence is specifically wanted, not on every pass. Full details\nper source: `references\u002Freddit.md`, `references\u002Facademic.md`,\n`references\u002Fsubstack.md`, `references\u002Fsupadata.md`.\n\n*Source status \u002F future stubs:* **Supadata** is **live** (paid). **X-Search**\n(`XAI_API_KEY`, paid) is now **wired but dormant** — the code is complete and the dry\ncall reaches xAI correctly; it activates the moment the xAI account has billing\u002Fcredit\n(console.x.ai), with **no code change** (details: `references\u002Fxsearch.md`). **Podcast**\nhas **no dedicated endpoint** and is covered by Supadata's YouTube search + transcript\n(or a direct media-file URL). Everything except Supadata and X-Search stays free.\n\n## Running a rigorous multi-source research pass\n\nWhen the task is \"research X properly\" rather than one lookup, don't ad-hoc it —\nfollow `references\u002Fresearch-method.md`. It captures a disciplined flow (scope →\nrecon with HOT\u002FWARM\u002FCOLD per source → human checkpoint → parallel topic agents →\ncritic pass → confidence-labelled synthesis), a **source-scoring rubric**\n(recency \u002F source type \u002F specificity \u002F independence) and ready prompt templates,\nall driving **our existing `deep-research` harness and `chief-of-staff` fan-out**\nover the sources above. It adds no new orchestrator and no database.\n\n## Choosing a path\n\n```\nNeed live web data\n├─ Have a specific URL to read?\n│   ├─ Page is plain\u002Fstatic and small  → a normal fetch is fine\n│   └─ Page is JS-heavy \u002F dynamic \u002F SPA → scrape\n├─ Need to FIND pages by meaning       → search   (Exa)\n├─ Have URLs, just want their content  → contents (Exa)  or scrape (Firecrawl)\n└─ Want a direct answer with citations → answer   (Exa)\n```\n\n## Common mistakes (and the fix)\n\nThese are real API gotchas — avoid them:\n\n- On **Exa `\u002Fsearch`**, `text`\u002F`highlights`\u002F`summary` must be **nested under\n  `contents`** (the script does this). On **`\u002Fcontents`** they're **top-level**.\n- `useAutoprompt`, `numSentences`, `highlightsPerUrl`, `tokensNum`,\n  `livecrawl: \"always\"`, `includeUrls`\u002F`excludeUrls` are **deprecated or\n  non-existent** — use `--max-age-hours 0` for freshness and\n  `--include-domains`\u002F`--exclude-domains` for filtering.\n- Uncapped `--text` can blow up context. Prefer `--highlights`, or\n  `--text --max-chars N`.\n- Slow\u002Fempty `scrape` result on a dynamic page → add `--wait 2000`-`5000` to\n  let JS render before extraction.\n- `401` → the matching key is missing or wrong in `~\u002F.claude\u002F.env`.\n\n## Deeper reference\n\n- `references\u002Fexa.md` — search types, content config, `outputSchema`,\n  `maxAgeHours`, `\u002Fcontents` & `\u002Fanswer`, troubleshooting.\n- `references\u002Ffirecrawl.md` — scrape body fields, `formats`, `actions` for\n  interaction, the search endpoint, response shape.\n- `references\u002Freddit.md` — Reddit via Composio managed toolkit (read tools, execute contract), CLI, the direct public-JSON fallback and its 403\u002FIP gotcha.\n- `references\u002Facademic.md` — OpenAlex + Unpaywall CLI, paper fields, citation graph.\n- `references\u002Fsubstack.md` — Substack discovery + extraction CLI, response shape.\n- `references\u002Fsupadata.md` — Supadata video\u002Ftranscript CLI (YouTube search, multi-platform transcripts, sync-vs-async, cost note).\n- `references\u002Fxsearch.md` — X-Search CLI (xAI Grok X Search tool: Responses API shape, `x_search` fields, model id, cost note, the dormant-until-funded gate).\n- `references\u002Fresearch-method.md` — the full multi-source research flow, source-scoring\n  rubric, and recon\u002Ftopic\u002Fcritic\u002Fsynthesis prompt templates.\n\n**Canonical sources of truth** (fetch these if behaviour ever contradicts the\nabove, and report drift back to the user):\n- Exa: https:\u002F\u002Fexa.ai\u002Fdocs\u002Freference\u002Fsearch-api-guide-for-coding-agents\n- Firecrawl: https:\u002F\u002Fdocs.firecrawl.dev\n",{"name":5,"description":7},"main","b6a401bedc8f704ad123d7e068f4fbb45d125f1d","plugins\u002Fshelf-deep-research\u002Fskills\u002Fweb-scrape","https:\u002F\u002Fgithub.com\u002Fskyremote\u002Fskool-navaigate-install","https:\u002F\u002Fgithub.com\u002Fskyremote","2026-11-06 06:04:17",{"source":8,"skillName":5,"commands":31,"prompt":41,"zipAvailable":42},[32,35,38],{"tool":33,"command":34},"npx","npx skills add skyremote\u002Fskool-navaigate-install --skill web-scrape",{"tool":36,"command":37},"bunx","bunx skills add skyremote\u002Fskool-navaigate-install --skill web-scrape",{"tool":39,"command":40},"pnpm","pnpm dlx skills add skyremote\u002Fskool-navaigate-install --skill web-scrape","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fskyremote\u002Fskool-navaigate-install\u002Fblob\u002Fmain\u002Fplugins\u002Fshelf-deep-research\u002Fskills\u002Fweb-scrape\u002FSKILL.md",true,[44,50,54,58,62,66,70,74,78,82,87,91,95,99,103,107,111],{"path":45,"resourceType":46,"resourceTypeDesc":47,"sizeBytes":48,"sourceHtmlUrl":49},"references\u002Facademic.md","REFERENCE","参考文档",2428,"https:\u002F\u002Fgithub.com\u002Fskyremote\u002Fskool-navaigate-install\u002Fblob\u002Fmain\u002Fplugins\u002Fshelf-deep-research\u002Fskills\u002Fweb-scrape\u002Freferences\u002Facademic.md",{"path":51,"resourceType":46,"resourceTypeDesc":47,"sizeBytes":52,"sourceHtmlUrl":53},"references\u002Fexa.md",5858,"https:\u002F\u002Fgithub.com\u002Fskyremote\u002Fskool-navaigate-install\u002Fblob\u002Fmain\u002Fplugins\u002Fshelf-deep-research\u002Fskills\u002Fweb-scrape\u002Freferences\u002Fexa.md",{"path":55,"resourceType":46,"resourceTypeDesc":47,"sizeBytes":56,"sourceHtmlUrl":57},"references\u002Ffirecrawl.md",4286,"https:\u002F\u002Fgithub.com\u002Fskyremote\u002Fskool-navaigate-install\u002Fblob\u002Fmain\u002Fplugins\u002Fshelf-deep-research\u002Fskills\u002Fweb-scrape\u002Freferences\u002Ffirecrawl.md",{"path":59,"resourceType":46,"resourceTypeDesc":47,"sizeBytes":60,"sourceHtmlUrl":61},"references\u002Freddit.md",5204,"https:\u002F\u002Fgithub.com\u002Fskyremote\u002Fskool-navaigate-install\u002Fblob\u002Fmain\u002Fplugins\u002Fshelf-deep-research\u002Fskills\u002Fweb-scrape\u002Freferences\u002Freddit.md",{"path":63,"resourceType":46,"resourceTypeDesc":47,"sizeBytes":64,"sourceHtmlUrl":65},"references\u002Fresearch-method.md",13219,"https:\u002F\u002Fgithub.com\u002Fskyremote\u002Fskool-navaigate-install\u002Fblob\u002Fmain\u002Fplugins\u002Fshelf-deep-research\u002Fskills\u002Fweb-scrape\u002Freferences\u002Fresearch-method.md",{"path":67,"resourceType":46,"resourceTypeDesc":47,"sizeBytes":68,"sourceHtmlUrl":69},"references\u002Fresearch-sources.md",3224,"https:\u002F\u002Fgithub.com\u002Fskyremote\u002Fskool-navaigate-install\u002Fblob\u002Fmain\u002Fplugins\u002Fshelf-deep-research\u002Fskills\u002Fweb-scrape\u002Freferences\u002Fresearch-sources.md",{"path":71,"resourceType":46,"resourceTypeDesc":47,"sizeBytes":72,"sourceHtmlUrl":73},"references\u002Fsubstack.md",2358,"https:\u002F\u002Fgithub.com\u002Fskyremote\u002Fskool-navaigate-install\u002Fblob\u002Fmain\u002Fplugins\u002Fshelf-deep-research\u002Fskills\u002Fweb-scrape\u002Freferences\u002Fsubstack.md",{"path":75,"resourceType":46,"resourceTypeDesc":47,"sizeBytes":76,"sourceHtmlUrl":77},"references\u002Fsupadata.md",4844,"https:\u002F\u002Fgithub.com\u002Fskyremote\u002Fskool-navaigate-install\u002Fblob\u002Fmain\u002Fplugins\u002Fshelf-deep-research\u002Fskills\u002Fweb-scrape\u002Freferences\u002Fsupadata.md",{"path":79,"resourceType":46,"resourceTypeDesc":47,"sizeBytes":80,"sourceHtmlUrl":81},"references\u002Fxsearch.md",6307,"https:\u002F\u002Fgithub.com\u002Fskyremote\u002Fskool-navaigate-install\u002Fblob\u002Fmain\u002Fplugins\u002Fshelf-deep-research\u002Fskills\u002Fweb-scrape\u002Freferences\u002Fxsearch.md",{"path":83,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":14,"sourceHtmlUrl":86},"scripts\u002Fsources\u002F__init__.py","SCRIPT","脚本","https:\u002F\u002Fgithub.com\u002Fskyremote\u002Fskool-navaigate-install\u002Fblob\u002Fmain\u002Fplugins\u002Fshelf-deep-research\u002Fskills\u002Fweb-scrape\u002Fscripts\u002Fsources\u002F__init__.py",{"path":88,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":89,"sourceHtmlUrl":90},"scripts\u002Fsources\u002F_common.py",2851,"https:\u002F\u002Fgithub.com\u002Fskyremote\u002Fskool-navaigate-install\u002Fblob\u002Fmain\u002Fplugins\u002Fshelf-deep-research\u002Fskills\u002Fweb-scrape\u002Fscripts\u002Fsources\u002F_common.py",{"path":92,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":93,"sourceHtmlUrl":94},"scripts\u002Fsources\u002Facademic.py",13019,"https:\u002F\u002Fgithub.com\u002Fskyremote\u002Fskool-navaigate-install\u002Fblob\u002Fmain\u002Fplugins\u002Fshelf-deep-research\u002Fskills\u002Fweb-scrape\u002Fscripts\u002Fsources\u002Facademic.py",{"path":96,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":97,"sourceHtmlUrl":98},"scripts\u002Fsources\u002Freddit.py",29073,"https:\u002F\u002Fgithub.com\u002Fskyremote\u002Fskool-navaigate-install\u002Fblob\u002Fmain\u002Fplugins\u002Fshelf-deep-research\u002Fskills\u002Fweb-scrape\u002Fscripts\u002Fsources\u002Freddit.py",{"path":100,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":101,"sourceHtmlUrl":102},"scripts\u002Fsources\u002Fsubstack.py",12990,"https:\u002F\u002Fgithub.com\u002Fskyremote\u002Fskool-navaigate-install\u002Fblob\u002Fmain\u002Fplugins\u002Fshelf-deep-research\u002Fskills\u002Fweb-scrape\u002Fscripts\u002Fsources\u002Fsubstack.py",{"path":104,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":105,"sourceHtmlUrl":106},"scripts\u002Fsources\u002Fsupadata.py",14851,"https:\u002F\u002Fgithub.com\u002Fskyremote\u002Fskool-navaigate-install\u002Fblob\u002Fmain\u002Fplugins\u002Fshelf-deep-research\u002Fskills\u002Fweb-scrape\u002Fscripts\u002Fsources\u002Fsupadata.py",{"path":108,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":109,"sourceHtmlUrl":110},"scripts\u002Fsources\u002Fxsearch.py",19255,"https:\u002F\u002Fgithub.com\u002Fskyremote\u002Fskool-navaigate-install\u002Fblob\u002Fmain\u002Fplugins\u002Fshelf-deep-research\u002Fskills\u002Fweb-scrape\u002Fscripts\u002Fsources\u002Fxsearch.py",{"path":112,"resourceType":84,"resourceTypeDesc":85,"sizeBytes":113,"sourceHtmlUrl":114},"scripts\u002Fwebscrape.py",12566,"https:\u002F\u002Fgithub.com\u002Fskyremote\u002Fskool-navaigate-install\u002Fblob\u002Fmain\u002Fplugins\u002Fshelf-deep-research\u002Fskills\u002Fweb-scrape\u002Fscripts\u002Fwebscrape.py",[116,123,130,137,144,149],{"skillNo":117,"name":118,"nameZh":6,"description":119,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":120,"categories":121,"keywords":122,"publishedAt":19},"01M7BP00HT65AC5QW5K7Z3QHHH","your-voice","Help them build THEIR voice skill from their recordings and line edits. Use when they type \u002Fyour-voice. Never install daniel-voice.","https:\u002F\u002Fgithub.com\u002Fskyremote\u002Fskool-navaigate-install\u002Fblob\u002Fmain\u002Fskills\u002Fyour-voice\u002FSKILL.md",[],[],{"skillNo":124,"name":125,"nameZh":6,"description":126,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":127,"categories":128,"keywords":129,"publishedAt":19},"01M7BNZW4FKYC8ANHQE4H2WGH2","your-ops","Help them componentize THEIR invoice or ops template from a real document they sent. Use when they type \u002Fyour-ops. Never ship NavAIgate letterhead or KG register facts.","https:\u002F\u002Fgithub.com\u002Fskyremote\u002Fskool-navaigate-install\u002Fblob\u002Fmain\u002Fskills\u002Fyour-ops\u002FSKILL.md",[],[],{"skillNo":131,"name":132,"nameZh":6,"description":133,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":134,"categories":135,"keywords":136,"publishedAt":19},"01M7BNZSTWW9TB2ADVNSBEKDTC","your-brain","Help them start THEIR memory and keep it in step across machines. Use when they type \u002Fyour-brain. Never install brain-sync or turso-search.","https:\u002F\u002Fgithub.com\u002Fskyremote\u002Fskool-navaigate-install\u002Fblob\u002Fmain\u002Fskills\u002Fyour-brain\u002FSKILL.md",[],[],{"skillNo":138,"name":139,"nameZh":6,"description":140,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":141,"categories":142,"keywords":143,"publishedAt":19},"01M7BNZQ7AVN1G1AZGKJ4ZEEZB","writing-style","Scrub AI slop from prose they are about to ship. Use when they type \u002Fwriting-style, or when they ask to de-slop a draft. Voice itself is \u002Fyour-voice — this skill is the editor.","https:\u002F\u002Fgithub.com\u002Fskyremote\u002Fskool-navaigate-install\u002Fblob\u002Fmain\u002Fskills\u002Fwriting-style\u002FSKILL.md",[],[],{"skillNo":145,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":146,"categories":147,"keywords":148,"publishedAt":19},"01M7BNZBMES31K9FYSF8Z43RX7","https:\u002F\u002Fgithub.com\u002Fskyremote\u002Fskool-navaigate-install\u002Fblob\u002Fmain\u002Fskills\u002Fweb-scrape\u002FSKILL.md",[],[],{"skillNo":150,"name":151,"nameZh":6,"description":152,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":153,"categories":154,"keywords":155,"publishedAt":19},"01M7BNZ7XN9SQ1GKYF88KH9GR3","transcripts","Pull YouTube, TikTok, Instagram or file transcripts via Supadata. Use when they type \u002Ftranscripts or paste a video URL and want the words.","https:\u002F\u002Fgithub.com\u002Fskyremote\u002Fskool-navaigate-install\u002Fblob\u002Fmain\u002Fskills\u002Ftranscripts\u002FSKILL.md",[],[]]