blocked-page-recovery
Use when a fetch fails: 403/429, paywall, WAF, bot wall.
- License 许可
- MIT
- 最近更新
- 2026-10-20
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- blocked-page-recovery
- author
- Hermes Agent
- license
- MIT
- version
- 1.0.0
- metadata
- { "hermes": { "tags": [ "Research", "Archives", "Wayback", "Paywall", "WAF", "Fallback" ], "related_skills": [ "grounded-citations" ] } }
- platforms
- [ "linux", "macos", "windows" ]
- description
- Use when a fetch fails: 403/429, paywall, WAF, bot wall.
02 资源文件 2
- 参考文档references/hermes-docker-curl-playbook.md3.0 KB
- 脚本scripts/recover_page.py8.5 KB
03 SKILL.md 正文
Blocked-Page Recovery
When a page won't fetch — 403/429, Cloudflare "Just a moment...", a paywall, or a bot-detection interstitial — don't give up and don't loop on the same URL. Third-party services often hold a copy of the page. Work down this ladder, cheapest first.
The ladder
1. Wayback Machine — archive.org "available" API (snapshot + timestamp)
2. archive.today — domain rotation: archive.ph → .md → .li → .is
3. Jina Reader — only if JINA_API_KEY is set (live server-side render)
4. API-first pivot — look for /api/, /graphql, .json, or RSS on the same host
5. Real browser — browser tool as the last, most expensive resort
Run it in one shot with the bundled script:
python3 scripts/recover_page.py "https://example.com/blocked-article" --json
The script tries each route in order, validates every body (see "Fake successes" below), and prints the first genuine hit with its provenance.
Provenance discipline (non-negotiable)
Every recovered copy carries a provenance you MUST preserve when citing:
| Route | Provenance | How to cite |
|---|---|---|
| Wayback / archive.today | snapshot |
Cite WITH the snapshot date: "as archived 2026-08-06". Never present a snapshot as the live page — it may be stale. |
| Jina Reader | live |
Server-side re-render of the live page; cite normally. |
| Live fetch / browser | live |
Cite normally. |
If the user needs current data (prices, availability, breaking news), a snapshot is context, not an answer — say so explicitly and note its age.
Manual routes
1. Wayback Machine (best provenance, try first)
# Discovery: returns closest snapshot URL + timestamp as JSON
curl -sL "https://archive.org/wayback/available?url={URL}"
# Then fetch archived_snapshots.closest.url
For enumerating many snapshots (or recovering deleted pages), the CDX index:
curl -sL "https://web.archive.org/cdx/search/cdx?url={URL}&output=json&limit=10"
CDX intermittently returns 503 under load — if it does, fall back to the
available API; don't retry-hammer it.
Works for: any publicly crawled URL. Fails for: robots-blocked sites, never-crawled URLs, JS-only SPAs (snapshots don't render).
2. archive.today (paywalls, deleted content)
User-submitted archives — often has paywalled news articles Wayback lacks. Rate-limits aggressively (429) and rotates domains, so iterate:
for d in archive.ph archive.md archive.li archive.is; do
curl -sL --max-time 20 "https://$d/newest/{URL}" -o /tmp/page.html \
-w "%{http_code}" && break
done
Validate the body, not the status code — a 429 still ships several KB of rate-limit HTML that looks like a success to a size check alone.
3. Jina Reader (requires JINA_API_KEY)
r.jina.ai re-renders the live page in a real browser server-side and
returns markdown. Anonymous access is dead (401 → Turnstile); a key is
required:
curl -s -H "Authorization: Bearer $JINA_API_KEY" "https://r.jina.ai/{URL}"
Handles JS SPAs that archives can't. Skip this route entirely when the env var is unset.
4. API-first pivot
WAFs protect the HTML surface far more aggressively than the data endpoints behind it. After 2-3 blocked attempts on a site, stop fighting the HTML and look for:
/api/...,/graphql, or.jsonvariants of the page URL- An RSS/Atom feed (
/feed,/rss,<link rel="alternate">in any copy you did recover) - A sitemap (
/sitemap.xml) revealing canonical URLs that may not be gated
Fake successes — routes that LIE
These return HTTP 200 with a plausible body that is NOT the page. The script rejects them automatically; reject them manually too:
- Google Cache is dead (since mid-2024).
webcache.googleusercontent.comreturns 200 + tens of KB, but it's a Google Search interstitial with a JS redirect, not a cache. Never use it. - AMP caches (
*.cdn.ampproject.org) mostly return a ~300-byte<title>Redirecting</title>meta-refresh stub pointing back at the original (blocked) URL. Treating that as success creates a fetch loop. - Rate-limit bodies: archive.today 429 pages are multi-KB HTML. Check for the target's actual content (title words, expected strings), not just size.
Detection heuristics the script applies: body under a per-route byte floor; meta-refresh/JS-redirect stubs whose target is the original host; interstitial titles ("Just a moment", "Redirecting", "Google Search", "Attention Required").
Proxy relays: don't
Generic "web proxy" relays are man-in-the-middle by construction. Never send cookies or Authorization headers through one, and don't use them for anything the user will rely on — provenance is unverifiable. Prefer archives, which at least timestamp their copies.
When the web TOOLS themselves fail
If web_search/web_extract time out (or the system browser is busy) but
the terminal has network, switch to terminal curl against search engines —
see references/hermes-docker-curl-playbook.md for engine-specific recipes,
output-handling rules (terminal stdout truncates; write to files instead),
and Chinese-site verification patterns.
04 同仓库更多 Skills
web-retrieval-resilience
Web tools fail or search rules break: repair or fall back.
verification-loop
Run a six-phase verification of a Claude Code session's work — build, type check, lint, tests with coverage, security grep, and diff review — then produce a PASS/FAIL verification report. Use when verifying work after completing a feature or refactor, before creating a PR, or when quality gates must pass.
unreal-world-builder
开放世界与环境专家——精通 UE5 World Partition、Landscape、程序化植被、HLOD 和大规模关卡流式加载,打造无缝开放世界体验
unreal-technical-artist
Unreal Engine 视觉管线专家——精通材质编辑器、Niagara 特效、程序化内容生成和 UE5 项目的美术到引擎管线
unreal-systems-engineer
性能与混合架构专家——精通 C++/Blueprint 边界、Nanite 几何体、Lumen GI 和 Gameplay Ability System,面向 AAA 级 Unreal Engine 项目
unreal-multiplayer-architect
Unreal Engine 网络专家——精通 Actor 复制、GameMode/GameState 架构、服务端权威玩法、网络预测和 UE5 专用服务器配置