AGENT SKILL

blocked-page-recovery

Use when a fetch fails: 403/429, paywall, WAF, bot wall. Recovers the page via archives and provenance-checked fallbacks.

License 许可
MIT
所属仓库
iberi22/skills
最近更新
2026-10-12
Stars
★ 0
Forks
0

01 前言 Front Matter

name
blocked-page-recovery
license
MIT
version
1.1.0
platforms
[ "linux" ]
description
Use when a fetch fails: 403/429, paywall, WAF, bot wall. Recovers the page via archives and provenance-checked fallbacks.

02 资源文件 1

03 SKILL.md 正文

Blocked-Page Recovery

When a page won't fetch — 403/429, Cloudflare "Just a moment...", a paywall, or a bot-detection interstitial — don't give up and don't loop on the same URL. Third-party services often hold a copy of the page. Work down this ladder, cheapest first.

If the obscura skill is installed, try obscura fetch --stealth <URL> first: it renders JS and many WAF/bot-wall pages locally for free. Use this ladder when Obscura still fails.

The ladder

1. Wayback Machine  — archive.org "available" API  (snapshot + timestamp)
2. archive.today    — domain rotation: archive.ph → .md → .li → .is
3. Jina Reader      — only if JINA_API_KEY is set  (live server-side render)
4. Firecrawl Scrape — only if FIRECRAWL_API_KEY is set (JS render + WAF bypass, paid per page)
5. API-first pivot  — look for /api/, /graphql, .json, or RSS on the same host
6. Real browser     — browser automation tool as the last, most expensive resort

Run routes 1-3 in one shot with the bundled script (stdlib Python 3, resolve the path relative to this skill directory):

python3 scripts/recover_page.py "https://example.com/blocked-article" --json

Options: --out FILE saves the body, --timeout N (default 25). Exit codes: 0 recovered, 1 nothing worked, 2 bad invocation.

The script tries each route in order, validates every body (see "Fake successes" below), and prints the first genuine hit with its provenance. Firecrawl is a MANUAL route (the script covers 1-3 only): use it when 1-3 fail and the page needs JS rendering or WAF bypass.

Provenance discipline (non-negotiable)

Every recovered copy carries a provenance you MUST preserve when citing:

Route Provenance How to cite
Wayback / archive.today snapshot Cite WITH the snapshot date: "as archived 2026-08-06". Never present a snapshot as the live page — it may be stale.
Jina Reader live Server-side re-render of the live page; cite normally.
Firecrawl / live fetch / browser live Cite normally.

If the user needs current data (prices, availability, breaking news), a snapshot is context, not an answer — say so explicitly and note its age.

Manual routes

1. Wayback Machine (best provenance, try first)

# Discovery: returns closest snapshot URL + timestamp as JSON
curl -sL "https://archive.org/wayback/available?url={URL}"
# Then fetch archived_snapshots.closest.url

For enumerating many snapshots (or recovering deleted pages), the CDX index:

curl -sL "https://web.archive.org/cdx/search/cdx?url={URL}&output=json&limit=10"

CDX intermittently returns 503 under load — if it does, fall back to the available API; don't retry-hammer it.

Works for: any publicly crawled URL. Fails for: robots-blocked sites, never-crawled URLs, JS-only SPAs (snapshots don't render).

2. archive.today (paywalls, deleted content)

User-submitted archives — often has paywalled news articles Wayback lacks. Rate-limits aggressively (429) and rotates domains, so iterate:

for d in archive.ph archive.md archive.li archive.is; do
  curl -sL --max-time 20 "https://$d/newest/{URL}" -o /tmp/page.html \
    -w "%{http_code}" && break
done

Validate the body, not the status code — a 429 still ships several KB of rate-limit HTML that looks like a success to a size check alone.

3. Jina Reader (requires JINA_API_KEY)

r.jina.ai re-renders the live page in a real browser server-side and returns markdown. Anonymous access is dead (401 → Turnstile); a key is required. Replace <JINA_API_KEY> with your key:

curl -s -H "Authorization: Bearer <JINA_API_KEY>" "https://r.jina.ai/{URL}"

Handles JS SPAs that archives can't. Skip this route entirely when the env var is unset.

4. Firecrawl Scrape (requires FIRECRAWL_API_KEY)

api.firecrawl.dev renders the live page (JS + WAF bypass) and returns markdown. Billed per page against your plan's credits — check current pricing. Replace <FIRECRAWL_API_KEY> with your key:

curl -s -X POST "https://api.firecrawl.dev/v2/scrape" \
  -H "Authorization: Bearer <FIRECRAWL_API_KEY>" \
  -H "Content-Type: application/json" \
  -d "{\"url\": \"{URL}\", \"formats\": [\"markdown\"]}"

Provenance: live — cite normally. Costs credits per page: use AFTER free routes 1-3.

5. API-first pivot

WAFs protect the HTML surface far more aggressively than the data endpoints behind it. After 2-3 blocked attempts on a site, stop fighting the HTML and look for:

  • /api/..., /graphql, or .json variants of the page URL
  • An RSS/Atom feed (/feed, /rss, <link rel="alternate"> in any copy you did recover)
  • A sitemap (/sitemap.xml) revealing canonical URLs that may not be gated

Fake successes — routes that LIE

These return HTTP 200 with a plausible body that is NOT the page. The script rejects them automatically; reject them manually too:

  • Google Cache is dead (since mid-2024). webcache.googleusercontent.com returns 200 + tens of KB, but it's a Google Search interstitial with a JS redirect, not a cache. Never use it.
  • AMP caches (*.cdn.ampproject.org) mostly return a ~300-byte <title>Redirecting</title> meta-refresh stub pointing back at the original (blocked) URL. Treating that as success creates a fetch loop.
  • Rate-limit bodies: archive.today 429 pages are multi-KB HTML. Check for the target's actual content (title words, expected strings), not just size.

Detection heuristics the script applies: body under a per-route byte floor; meta-refresh/JS-redirect stubs whose target is the original host; interstitial titles ("Just a moment", "Redirecting", "Google Search", "Attention Required").

Proxy relays: don't

Generic "web proxy" relays are man-in-the-middle by construction. Never send cookies or Authorization headers through one, and don't use them for anything the user will rely on — provenance is unverifiable. Prefer archives, which at least timestamp their copies.

Related skills

  • grounded-citations — citation rules for anything recovered here.
  • obscura — free local renderer, tried first when installed.

04 同仓库更多 Skills