[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7F8FP46QESQMCZ1KBT3TW3R":3,"skill-same:01M7F8FP46QESQMCZ1KBT3TW3R":75},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":41,"commitSha":42,"skillDirectory":43,"forksCount":14,"repositoryHtmlUrl":44,"repositoryHomepageUrl":6,"ownerHtmlUrl":45,"ownerBlogUrl":46,"ownerDescription":47,"updatedAt":48,"install":49,"resources":62},"01M7F8FP46QESQMCZ1KBT3TW3R","blocked-page-recovery",null,"Use when a fetch fails: 403\u002F429, paywall, WAF, bot wall.","chaorenex1\u002Fhermes-home-content","chaorenex1","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F23194925?v=4","01M7F5NA1D1D02R2MEMYQN2NZT","chan chan","01M7F5NA20KTHMTY2015CYM1TZ",0,"MIT","https:\u002F\u002Fgithub.com\u002Fchaorenex1\u002Fhermes-home-content\u002Fblob\u002Fmain\u002Fskills\u002Fweb\u002Fblocked-page-recovery\u002FSKILL.md",[],[],"2026-11-16 10:05:34",false,"---\r\nname: blocked-page-recovery\r\ndescription: \"Use when a fetch fails: 403\u002F429, paywall, WAF, bot wall.\"\r\nversion: 1.0.0\r\nauthor: Hermes Agent\r\nlicense: MIT\r\nplatforms: [linux, macos, windows]\r\nmetadata:\r\n  hermes:\r\n    tags: [Research, Archives, Wayback, Paywall, WAF, Fallback]\r\n    related_skills: [grounded-citations]\r\n---\r\n\r\n# Blocked-Page Recovery\r\n\r\nWhen a page won't fetch — 403\u002F429, Cloudflare \"Just a moment...\", a paywall,\r\nor a bot-detection interstitial — don't give up and don't loop on the same\r\nURL. Third-party services often hold a **copy** of the page. Work down this\r\nladder, cheapest first.\r\n\r\n## The ladder\r\n\r\n```\r\n1. Wayback Machine  — archive.org \"available\" API  (snapshot + timestamp)\r\n2. archive.today    — domain rotation: archive.ph → .md → .li → .is\r\n3. Jina Reader      — only if JINA_API_KEY is set  (live server-side render)\r\n4. API-first pivot  — look for \u002Fapi\u002F, \u002Fgraphql, .json, or RSS on the same host\r\n5. Real browser     — browser tool as the last, most expensive resort\r\n```\r\n\r\nRun it in one shot with the bundled script:\r\n\r\n```bash\r\npython3 scripts\u002Frecover_page.py \"https:\u002F\u002Fexample.com\u002Fblocked-article\" --json\r\n```\r\n\r\nThe script tries each route in order, validates every body (see \"Fake\r\nsuccesses\" below), and prints the first genuine hit with its provenance.\r\n\r\n## Provenance discipline (non-negotiable)\r\n\r\nEvery recovered copy carries a provenance you MUST preserve when citing:\r\n\r\n| Route | Provenance | How to cite |\r\n|-------|-----------|-------------|\r\n| Wayback \u002F archive.today | `snapshot` | Cite WITH the snapshot date: \"as archived 2026-08-06\". Never present a snapshot as the live page — it may be stale. |\r\n| Jina Reader | `live` | Server-side re-render of the live page; cite normally. |\r\n| Live fetch \u002F browser | `live` | Cite normally. |\r\n\r\nIf the user needs *current* data (prices, availability, breaking news), a\r\nsnapshot is context, not an answer — say so explicitly and note its age.\r\n\r\n## Manual routes\r\n\r\n### 1. Wayback Machine (best provenance, try first)\r\n\r\n```bash\r\n# Discovery: returns closest snapshot URL + timestamp as JSON\r\ncurl -sL \"https:\u002F\u002Farchive.org\u002Fwayback\u002Favailable?url={URL}\"\r\n# Then fetch archived_snapshots.closest.url\r\n```\r\n\r\nFor enumerating many snapshots (or recovering deleted pages), the CDX index:\r\n\r\n```bash\r\ncurl -sL \"https:\u002F\u002Fweb.archive.org\u002Fcdx\u002Fsearch\u002Fcdx?url={URL}&output=json&limit=10\"\r\n```\r\n\r\nCDX intermittently returns 503 under load — if it does, fall back to the\r\n`available` API; don't retry-hammer it.\r\n\r\nWorks for: any publicly crawled URL. Fails for: robots-blocked sites,\r\nnever-crawled URLs, JS-only SPAs (snapshots don't render).\r\n\r\n### 2. archive.today (paywalls, deleted content)\r\n\r\nUser-submitted archives — often has paywalled news articles Wayback lacks.\r\nRate-limits aggressively (429) and rotates domains, so iterate:\r\n\r\n```bash\r\nfor d in archive.ph archive.md archive.li archive.is; do\r\n  curl -sL --max-time 20 \"https:\u002F\u002F$d\u002Fnewest\u002F{URL}\" -o \u002Ftmp\u002Fpage.html \\\r\n    -w \"%{http_code}\" && break\r\ndone\r\n```\r\n\r\n**Validate the body, not the status code** — a 429 still ships several KB of\r\nrate-limit HTML that looks like a success to a size check alone.\r\n\r\n### 3. Jina Reader (requires JINA_API_KEY)\r\n\r\n`r.jina.ai` re-renders the live page in a real browser server-side and\r\nreturns markdown. Anonymous access is dead (401 → Turnstile); a key is\r\nrequired:\r\n\r\n```bash\r\ncurl -s -H \"Authorization: Bearer $JINA_API_KEY\" \"https:\u002F\u002Fr.jina.ai\u002F{URL}\"\r\n```\r\n\r\nHandles JS SPAs that archives can't. Skip this route entirely when the env\r\nvar is unset.\r\n\r\n### 4. API-first pivot\r\n\r\nWAFs protect the HTML surface far more aggressively than the data endpoints\r\nbehind it. After 2-3 blocked attempts on a site, stop fighting the HTML and\r\nlook for:\r\n\r\n- `\u002Fapi\u002F...`, `\u002Fgraphql`, or `.json` variants of the page URL\r\n- An RSS\u002FAtom feed (`\u002Ffeed`, `\u002Frss`, `\u003Clink rel=\"alternate\">` in any copy\r\n  you did recover)\r\n- A sitemap (`\u002Fsitemap.xml`) revealing canonical URLs that may not be gated\r\n\r\n## Fake successes — routes that LIE\r\n\r\nThese return HTTP 200 with a plausible body that is NOT the page. The script\r\nrejects them automatically; reject them manually too:\r\n\r\n- **Google Cache is dead** (since mid-2024). `webcache.googleusercontent.com`\r\n  returns 200 + tens of KB, but it's a Google Search interstitial with a JS\r\n  redirect, not a cache. Never use it.\r\n- **AMP caches** (`*.cdn.ampproject.org`) mostly return a ~300-byte\r\n  `\u003Ctitle>Redirecting\u003C\u002Ftitle>` meta-refresh stub pointing back at the\r\n  original (blocked) URL. Treating that as success creates a fetch loop.\r\n- **Rate-limit bodies**: archive.today 429 pages are multi-KB HTML. Check for\r\n  the target's actual content (title words, expected strings), not just size.\r\n\r\nDetection heuristics the script applies: body under a per-route byte floor;\r\nmeta-refresh\u002FJS-redirect stubs whose target is the original host; interstitial\r\ntitles (\"Just a moment\", \"Redirecting\", \"Google Search\", \"Attention Required\").\r\n\r\n## Proxy relays: don't\r\n\r\nGeneric \"web proxy\" relays are man-in-the-middle by construction. Never send\r\ncookies or Authorization headers through one, and don't use them for anything\r\nthe user will rely on — provenance is unverifiable. Prefer archives, which at\r\nleast timestamp their copies.\r\n\r\n## When the web TOOLS themselves fail\r\n\r\nIf `web_search`\u002F`web_extract` time out (or the system browser is busy) but\r\nthe terminal has network, switch to terminal `curl` against search engines —\r\nsee `references\u002Fhermes-docker-curl-playbook.md` for engine-specific recipes,\r\noutput-handling rules (terminal stdout truncates; write to files instead),\r\nand Chinese-site verification patterns.\r\n","# Blocked-Page Recovery\r\n\r\nWhen a page won't fetch — 403\u002F429, Cloudflare \"Just a moment...\", a paywall,\r\nor a bot-detection interstitial — don't give up and don't loop on the same\r\nURL. Third-party services often hold a **copy** of the page. Work down this\r\nladder, cheapest first.\r\n\r\n## The ladder\r\n\r\n```\r\n1. Wayback Machine  — archive.org \"available\" API  (snapshot + timestamp)\r\n2. archive.today    — domain rotation: archive.ph → .md → .li → .is\r\n3. Jina Reader      — only if JINA_API_KEY is set  (live server-side render)\r\n4. API-first pivot  — look for \u002Fapi\u002F, \u002Fgraphql, .json, or RSS on the same host\r\n5. Real browser     — browser tool as the last, most expensive resort\r\n```\r\n\r\nRun it in one shot with the bundled script:\r\n\r\n```bash\r\npython3 scripts\u002Frecover_page.py \"https:\u002F\u002Fexample.com\u002Fblocked-article\" --json\r\n```\r\n\r\nThe script tries each route in order, validates every body (see \"Fake\r\nsuccesses\" below), and prints the first genuine hit with its provenance.\r\n\r\n## Provenance discipline (non-negotiable)\r\n\r\nEvery recovered copy carries a provenance you MUST preserve when citing:\r\n\r\n| Route | Provenance | How to cite |\r\n|-------|-----------|-------------|\r\n| Wayback \u002F archive.today | `snapshot` | Cite WITH the snapshot date: \"as archived 2026-08-06\". Never present a snapshot as the live page — it may be stale. |\r\n| Jina Reader | `live` | Server-side re-render of the live page; cite normally. |\r\n| Live fetch \u002F browser | `live` | Cite normally. |\r\n\r\nIf the user needs *current* data (prices, availability, breaking news), a\r\nsnapshot is context, not an answer — say so explicitly and note its age.\r\n\r\n## Manual routes\r\n\r\n### 1. Wayback Machine (best provenance, try first)\r\n\r\n```bash\r\n# Discovery: returns closest snapshot URL + timestamp as JSON\r\ncurl -sL \"https:\u002F\u002Farchive.org\u002Fwayback\u002Favailable?url={URL}\"\r\n# Then fetch archived_snapshots.closest.url\r\n```\r\n\r\nFor enumerating many snapshots (or recovering deleted pages), the CDX index:\r\n\r\n```bash\r\ncurl -sL \"https:\u002F\u002Fweb.archive.org\u002Fcdx\u002Fsearch\u002Fcdx?url={URL}&output=json&limit=10\"\r\n```\r\n\r\nCDX intermittently returns 503 under load — if it does, fall back to the\r\n`available` API; don't retry-hammer it.\r\n\r\nWorks for: any publicly crawled URL. Fails for: robots-blocked sites,\r\nnever-crawled URLs, JS-only SPAs (snapshots don't render).\r\n\r\n### 2. archive.today (paywalls, deleted content)\r\n\r\nUser-submitted archives — often has paywalled news articles Wayback lacks.\r\nRate-limits aggressively (429) and rotates domains, so iterate:\r\n\r\n```bash\r\nfor d in archive.ph archive.md archive.li archive.is; do\r\n  curl -sL --max-time 20 \"https:\u002F\u002F$d\u002Fnewest\u002F{URL}\" -o \u002Ftmp\u002Fpage.html \\\r\n    -w \"%{http_code}\" && break\r\ndone\r\n```\r\n\r\n**Validate the body, not the status code** — a 429 still ships several KB of\r\nrate-limit HTML that looks like a success to a size check alone.\r\n\r\n### 3. Jina Reader (requires JINA_API_KEY)\r\n\r\n`r.jina.ai` re-renders the live page in a real browser server-side and\r\nreturns markdown. Anonymous access is dead (401 → Turnstile); a key is\r\nrequired:\r\n\r\n```bash\r\ncurl -s -H \"Authorization: Bearer $JINA_API_KEY\" \"https:\u002F\u002Fr.jina.ai\u002F{URL}\"\r\n```\r\n\r\nHandles JS SPAs that archives can't. Skip this route entirely when the env\r\nvar is unset.\r\n\r\n### 4. API-first pivot\r\n\r\nWAFs protect the HTML surface far more aggressively than the data endpoints\r\nbehind it. After 2-3 blocked attempts on a site, stop fighting the HTML and\r\nlook for:\r\n\r\n- `\u002Fapi\u002F...`, `\u002Fgraphql`, or `.json` variants of the page URL\r\n- An RSS\u002FAtom feed (`\u002Ffeed`, `\u002Frss`, `\u003Clink rel=\"alternate\">` in any copy\r\n  you did recover)\r\n- A sitemap (`\u002Fsitemap.xml`) revealing canonical URLs that may not be gated\r\n\r\n## Fake successes — routes that LIE\r\n\r\nThese return HTTP 200 with a plausible body that is NOT the page. The script\r\nrejects them automatically; reject them manually too:\r\n\r\n- **Google Cache is dead** (since mid-2024). `webcache.googleusercontent.com`\r\n  returns 200 + tens of KB, but it's a Google Search interstitial with a JS\r\n  redirect, not a cache. Never use it.\r\n- **AMP caches** (`*.cdn.ampproject.org`) mostly return a ~300-byte\r\n  `\u003Ctitle>Redirecting\u003C\u002Ftitle>` meta-refresh stub pointing back at the\r\n  original (blocked) URL. Treating that as success creates a fetch loop.\r\n- **Rate-limit bodies**: archive.today 429 pages are multi-KB HTML. Check for\r\n  the target's actual content (title words, expected strings), not just size.\r\n\r\nDetection heuristics the script applies: body under a per-route byte floor;\r\nmeta-refresh\u002FJS-redirect stubs whose target is the original host; interstitial\r\ntitles (\"Just a moment\", \"Redirecting\", \"Google Search\", \"Attention Required\").\r\n\r\n## Proxy relays: don't\r\n\r\nGeneric \"web proxy\" relays are man-in-the-middle by construction. Never send\r\ncookies or Authorization headers through one, and don't use them for anything\r\nthe user will rely on — provenance is unverifiable. Prefer archives, which at\r\nleast timestamp their copies.\r\n\r\n## When the web TOOLS themselves fail\r\n\r\nIf `web_search`\u002F`web_extract` time out (or the system browser is busy) but\r\nthe terminal has network, switch to terminal `curl` against search engines —\r\nsee `references\u002Fhermes-docker-curl-playbook.md` for engine-specific recipes,\r\noutput-handling rules (terminal stdout truncates; write to files instead),\r\nand Chinese-site verification patterns.\r\n",{"name":5,"author":24,"license":15,"version":25,"metadata":26,"platforms":37,"description":7},"Hermes Agent","1.0.0",{"hermes":27},{"tags":28,"related_skills":35},[29,30,31,32,33,34],"Research","Archives","Wayback","Paywall","WAF","Fallback",[36],"grounded-citations",[38,39,40],"linux","macos","windows","main","a3f3199bb87ec3c91248f68a39d88527e1847ae9","skills\u002Fweb\u002Fblocked-page-recovery","https:\u002F\u002Fgithub.com\u002Fchaorenex1\u002Fhermes-home-content","https:\u002F\u002Fgithub.com\u002Fchaorenex1","https:\u002F\u002Fblog.aduib.com","A programmer whose only hobby in life is coding","2026-10-20 00:34:32",{"source":8,"skillName":5,"commands":50,"prompt":60,"zipAvailable":61},[51,54,57],{"tool":52,"command":53},"npx","npx skills add chaorenex1\u002Fhermes-home-content --skill blocked-page-recovery",{"tool":55,"command":56},"bunx","bunx skills add chaorenex1\u002Fhermes-home-content --skill blocked-page-recovery",{"tool":58,"command":59},"pnpm","pnpm dlx skills add chaorenex1\u002Fhermes-home-content --skill blocked-page-recovery","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fchaorenex1\u002Fhermes-home-content\u002Fblob\u002Fmain\u002Fskills\u002Fweb\u002Fblocked-page-recovery\u002FSKILL.md",true,[63,69],{"path":64,"resourceType":65,"resourceTypeDesc":66,"sizeBytes":67,"sourceHtmlUrl":68},"references\u002Fhermes-docker-curl-playbook.md","REFERENCE","参考文档",3052,"https:\u002F\u002Fgithub.com\u002Fchaorenex1\u002Fhermes-home-content\u002Fblob\u002Fmain\u002Fskills\u002Fweb\u002Fblocked-page-recovery\u002Freferences\u002Fhermes-docker-curl-playbook.md",{"path":70,"resourceType":71,"resourceTypeDesc":72,"sizeBytes":73,"sourceHtmlUrl":74},"scripts\u002Frecover_page.py","SCRIPT","脚本",8722,"https:\u002F\u002Fgithub.com\u002Fchaorenex1\u002Fhermes-home-content\u002Fblob\u002Fmain\u002Fskills\u002Fweb\u002Fblocked-page-recovery\u002Fscripts\u002Frecover_page.py",[76,83,90,97,104,111],{"skillNo":77,"name":78,"nameZh":6,"description":79,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":80,"categories":81,"keywords":82,"publishedAt":19},"01M7F8GF32YD5MBQW2TGSRH1YP","web-retrieval-resilience","Web tools fail or search rules break: repair or fall back.","https:\u002F\u002Fgithub.com\u002Fchaorenex1\u002Fhermes-home-content\u002Fblob\u002Fmain\u002Fskills\u002Fweb\u002Fweb-retrieval-resilience\u002FSKILL.md",[],[],{"skillNo":84,"name":85,"nameZh":6,"description":86,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":87,"categories":88,"keywords":89,"publishedAt":19},"01M7F8ER6VXHBNVRQBZS7X08SE","verification-loop","Run a six-phase verification of a Claude Code session's work — build, type check, lint, tests with coverage, security grep, and diff review — then produce a PASS\u002FFAIL verification report. Use when verifying work after completing a feature or refactor, before creating a PR, or when quality gates must pass.","https:\u002F\u002Fgithub.com\u002Fchaorenex1\u002Fhermes-home-content\u002Fblob\u002Fmain\u002Fskills\u002Fverification-loop\u002FSKILL.md",[],[],{"skillNo":91,"name":92,"nameZh":6,"description":93,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":94,"categories":95,"keywords":96,"publishedAt":19},"01M7F8E4BQJ38KVQVM3MT20PYV","unreal-world-builder","开放世界与环境专家——精通 UE5 World Partition、Landscape、程序化植被、HLOD 和大规模关卡流式加载，打造无缝开放世界体验","https:\u002F\u002Fgithub.com\u002Fchaorenex1\u002Fhermes-home-content\u002Fblob\u002Fmain\u002Fskills\u002Funreal-engine\u002Funreal-world-builder\u002FSKILL.md",[],[],{"skillNo":98,"name":99,"nameZh":6,"description":100,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":101,"categories":102,"keywords":103,"publishedAt":19},"01M7F8D747SQW2FTP7GBSX38BR","unreal-technical-artist","Unreal Engine 视觉管线专家——精通材质编辑器、Niagara 特效、程序化内容生成和 UE5 项目的美术到引擎管线","https:\u002F\u002Fgithub.com\u002Fchaorenex1\u002Fhermes-home-content\u002Fblob\u002Fmain\u002Fskills\u002Funreal-engine\u002Funreal-technical-artist\u002FSKILL.md",[],[],{"skillNo":105,"name":106,"nameZh":6,"description":107,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":108,"categories":109,"keywords":110,"publishedAt":19},"01M7F8CKHYJVZVBTN0321J88T2","unreal-systems-engineer","性能与混合架构专家——精通 C++\u002FBlueprint 边界、Nanite 几何体、Lumen GI 和 Gameplay Ability System，面向 AAA 级 Unreal Engine 项目","https:\u002F\u002Fgithub.com\u002Fchaorenex1\u002Fhermes-home-content\u002Fblob\u002Fmain\u002Fskills\u002Funreal-engine\u002Funreal-systems-engineer\u002FSKILL.md",[],[],{"skillNo":112,"name":113,"nameZh":6,"description":114,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":115,"categories":116,"keywords":117,"publishedAt":19},"01M7F8BXVZMG70SFNS3R3KET42","unreal-multiplayer-architect","Unreal Engine 网络专家——精通 Actor 复制、GameMode\u002FGameState 架构、服务端权威玩法、网络预测和 UE5 专用服务器配置","https:\u002F\u002Fgithub.com\u002Fchaorenex1\u002Fhermes-home-content\u002Fblob\u002Fmain\u002Fskills\u002Funreal-engine\u002Funreal-multiplayer-architect\u002FSKILL.md",[],[]]