[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7BJSF663WP6ZYJMZKM91TXK":3,"skill-same:01M7BJSF663WP6ZYJMZKM91TXK":67},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":18,"linkOnly":19,"markdownContent":20,"markdownBody":21,"frontMatter":22,"refName":40,"commitSha":41,"skillDirectory":42,"forksCount":13,"repositoryHtmlUrl":43,"repositoryHomepageUrl":6,"ownerHtmlUrl":44,"ownerBlogUrl":45,"ownerDescription":6,"updatedAt":46,"install":47,"resources":60},"01M7BJSF663WP6ZYJMZKM91TXK","blocked-page-recovery",null,"Recover blocked\u002Fpaywalled\u002FWAF'd pages via fallbacks.","darkmatter\u002Fhermes","darkmatter","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F17834193?v=4","01M7BHSAW99WRAMTSSHNHX73V0","01M7BHSAXDR4PY4NA2DY5PN6W8",0,"MIT","https:\u002F\u002Fgithub.com\u002Fdarkmatter\u002Fhermes\u002Fblob\u002Fmain\u002Fdevbox\u002Fskills\u002Fresearch\u002Fblocked-page-recovery\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: blocked-page-recovery\ndescription: \"Recover blocked\u002Fpaywalled\u002FWAF'd pages via fallbacks.\"\nversion: 1.0.0\nauthor: Hermes Agent\nlicense: MIT\nplatforms: [linux, macos, windows]\nmetadata:\n  hermes:\n    tags: [Research, Archives, Wayback, Paywall, WAF, Fallback]\n    related_skills: [grounded-citations]\n---\n\n# Blocked-Page Recovery\n\nWhen a page won't fetch — 403\u002F429, Cloudflare \"Just a moment...\", a paywall,\nor a bot-detection interstitial — don't give up and don't loop on the same\nURL. Third-party services often hold a **copy** of the page. Work down this\nladder, cheapest first.\n\n## The ladder\n\n```\n1. Wayback Machine  — archive.org \"available\" API  (snapshot + timestamp)\n2. archive.today    — domain rotation: archive.ph → .md → .li → .is\n3. Jina Reader      — only if JINA_API_KEY is set  (live server-side render)\n4. API-first pivot  — look for \u002Fapi\u002F, \u002Fgraphql, .json, or RSS on the same host\n5. Real browser     — browser tool as the last, most expensive resort\n```\n\nRun it in one shot with the bundled script:\n\n```bash\npython3 scripts\u002Frecover_page.py \"https:\u002F\u002Fexample.com\u002Fblocked-article\" --json\n```\n\nThe script tries each route in order, validates every body (see \"Fake\nsuccesses\" below), and prints the first genuine hit with its provenance.\n\n## Provenance discipline (non-negotiable)\n\nEvery recovered copy carries a provenance you MUST preserve when citing:\n\n| Route | Provenance | How to cite |\n|-------|-----------|-------------|\n| Wayback \u002F archive.today | `snapshot` | Cite WITH the snapshot date: \"as archived 2026-08-06\". Never present a snapshot as the live page — it may be stale. |\n| Jina Reader | `live` | Server-side re-render of the live page; cite normally. |\n| Live fetch \u002F browser | `live` | Cite normally. |\n\nIf the user needs *current* data (prices, availability, breaking news), a\nsnapshot is context, not an answer — say so explicitly and note its age.\n\n## Manual routes\n\n### 1. Wayback Machine (best provenance, try first)\n\n```bash\n# Discovery: returns closest snapshot URL + timestamp as JSON\ncurl -sL \"https:\u002F\u002Farchive.org\u002Fwayback\u002Favailable?url={URL}\"\n# Then fetch archived_snapshots.closest.url\n```\n\nFor enumerating many snapshots (or recovering deleted pages), the CDX index:\n\n```bash\ncurl -sL \"https:\u002F\u002Fweb.archive.org\u002Fcdx\u002Fsearch\u002Fcdx?url={URL}&output=json&limit=10\"\n```\n\nCDX intermittently returns 503 under load — if it does, fall back to the\n`available` API; don't retry-hammer it.\n\nWorks for: any publicly crawled URL. Fails for: robots-blocked sites,\nnever-crawled URLs, JS-only SPAs (snapshots don't render).\n\n### 2. archive.today (paywalls, deleted content)\n\nUser-submitted archives — often has paywalled news articles Wayback lacks.\nRate-limits aggressively (429) and rotates domains, so iterate:\n\n```bash\nfor d in archive.ph archive.md archive.li archive.is; do\n  curl -sL --max-time 20 \"https:\u002F\u002F$d\u002Fnewest\u002F{URL}\" -o \u002Ftmp\u002Fpage.html \\\n    -w \"%{http_code}\" && break\ndone\n```\n\n**Validate the body, not the status code** — a 429 still ships several KB of\nrate-limit HTML that looks like a success to a size check alone.\n\n### 3. Jina Reader (requires JINA_API_KEY)\n\n`r.jina.ai` re-renders the live page in a real browser server-side and\nreturns markdown. Anonymous access is dead (401 → Turnstile); a key is\nrequired:\n\n```bash\ncurl -s -H \"Authorization: Bearer \u003CREDACTED>\" \"https:\u002F\u002Fr.jina.ai\u002F{URL}\"\n```\n\nHandles JS SPAs that archives can't. Skip this route entirely when the env\nvar is unset.\n\n### 4. API-first pivot\n\nWAFs protect the HTML surface far more aggressively than the data endpoints\nbehind it. After 2-3 blocked attempts on a site, stop fighting the HTML and\nlook for:\n\n- `\u002Fapi\u002F...`, `\u002Fgraphql`, or `.json` variants of the page URL\n- An RSS\u002FAtom feed (`\u002Ffeed`, `\u002Frss`, `\u003Clink rel=\"alternate\">` in any copy\n  you did recover)\n- A sitemap (`\u002Fsitemap.xml`) revealing canonical URLs that may not be gated\n\n## Fake successes — routes that LIE\n\nThese return HTTP 200 with a plausible body that is NOT the page. The script\nrejects them automatically; reject them manually too:\n\n- **Google Cache is dead** (since mid-2024). `webcache.googleusercontent.com`\n  returns 200 + tens of KB, but it's a Google Search interstitial with a JS\n  redirect, not a cache. Never use it.\n- **AMP caches** (`*.cdn.ampproject.org`) mostly return a ~300-byte\n  `\u003Ctitle>Redirecting\u003C\u002Ftitle>` meta-refresh stub pointing back at the\n  original (blocked) URL. Treating that as success creates a fetch loop.\n- **Rate-limit bodies**: archive.today 429 pages are multi-KB HTML. Check for\n  the target's actual content (title words, expected strings), not just size.\n\nDetection heuristics the script applies: body under a per-route byte floor;\nmeta-refresh\u002FJS-redirect stubs whose target is the original host; interstitial\ntitles (\"Just a moment\", \"Redirecting\", \"Google Search\", \"Attention Required\").\n\n## Proxy relays: don't\n\nGeneric \"web proxy\" relays are man-in-the-middle by construction. Never send\ncookies or Authorization headers through one, and don't use them for anything\nthe user will rely on — provenance is unverifiable. Prefer archives, which at\nleast timestamp their copies.\n","# Blocked-Page Recovery\n\nWhen a page won't fetch — 403\u002F429, Cloudflare \"Just a moment...\", a paywall,\nor a bot-detection interstitial — don't give up and don't loop on the same\nURL. Third-party services often hold a **copy** of the page. Work down this\nladder, cheapest first.\n\n## The ladder\n\n```\n1. Wayback Machine  — archive.org \"available\" API  (snapshot + timestamp)\n2. archive.today    — domain rotation: archive.ph → .md → .li → .is\n3. Jina Reader      — only if JINA_API_KEY is set  (live server-side render)\n4. API-first pivot  — look for \u002Fapi\u002F, \u002Fgraphql, .json, or RSS on the same host\n5. Real browser     — browser tool as the last, most expensive resort\n```\n\nRun it in one shot with the bundled script:\n\n```bash\npython3 scripts\u002Frecover_page.py \"https:\u002F\u002Fexample.com\u002Fblocked-article\" --json\n```\n\nThe script tries each route in order, validates every body (see \"Fake\nsuccesses\" below), and prints the first genuine hit with its provenance.\n\n## Provenance discipline (non-negotiable)\n\nEvery recovered copy carries a provenance you MUST preserve when citing:\n\n| Route | Provenance | How to cite |\n|-------|-----------|-------------|\n| Wayback \u002F archive.today | `snapshot` | Cite WITH the snapshot date: \"as archived 2026-08-06\". Never present a snapshot as the live page — it may be stale. |\n| Jina Reader | `live` | Server-side re-render of the live page; cite normally. |\n| Live fetch \u002F browser | `live` | Cite normally. |\n\nIf the user needs *current* data (prices, availability, breaking news), a\nsnapshot is context, not an answer — say so explicitly and note its age.\n\n## Manual routes\n\n### 1. Wayback Machine (best provenance, try first)\n\n```bash\n# Discovery: returns closest snapshot URL + timestamp as JSON\ncurl -sL \"https:\u002F\u002Farchive.org\u002Fwayback\u002Favailable?url={URL}\"\n# Then fetch archived_snapshots.closest.url\n```\n\nFor enumerating many snapshots (or recovering deleted pages), the CDX index:\n\n```bash\ncurl -sL \"https:\u002F\u002Fweb.archive.org\u002Fcdx\u002Fsearch\u002Fcdx?url={URL}&output=json&limit=10\"\n```\n\nCDX intermittently returns 503 under load — if it does, fall back to the\n`available` API; don't retry-hammer it.\n\nWorks for: any publicly crawled URL. Fails for: robots-blocked sites,\nnever-crawled URLs, JS-only SPAs (snapshots don't render).\n\n### 2. archive.today (paywalls, deleted content)\n\nUser-submitted archives — often has paywalled news articles Wayback lacks.\nRate-limits aggressively (429) and rotates domains, so iterate:\n\n```bash\nfor d in archive.ph archive.md archive.li archive.is; do\n  curl -sL --max-time 20 \"https:\u002F\u002F$d\u002Fnewest\u002F{URL}\" -o \u002Ftmp\u002Fpage.html \\\n    -w \"%{http_code}\" && break\ndone\n```\n\n**Validate the body, not the status code** — a 429 still ships several KB of\nrate-limit HTML that looks like a success to a size check alone.\n\n### 3. Jina Reader (requires JINA_API_KEY)\n\n`r.jina.ai` re-renders the live page in a real browser server-side and\nreturns markdown. Anonymous access is dead (401 → Turnstile); a key is\nrequired:\n\n```bash\ncurl -s -H \"Authorization: Bearer \u003CREDACTED>\" \"https:\u002F\u002Fr.jina.ai\u002F{URL}\"\n```\n\nHandles JS SPAs that archives can't. Skip this route entirely when the env\nvar is unset.\n\n### 4. API-first pivot\n\nWAFs protect the HTML surface far more aggressively than the data endpoints\nbehind it. After 2-3 blocked attempts on a site, stop fighting the HTML and\nlook for:\n\n- `\u002Fapi\u002F...`, `\u002Fgraphql`, or `.json` variants of the page URL\n- An RSS\u002FAtom feed (`\u002Ffeed`, `\u002Frss`, `\u003Clink rel=\"alternate\">` in any copy\n  you did recover)\n- A sitemap (`\u002Fsitemap.xml`) revealing canonical URLs that may not be gated\n\n## Fake successes — routes that LIE\n\nThese return HTTP 200 with a plausible body that is NOT the page. The script\nrejects them automatically; reject them manually too:\n\n- **Google Cache is dead** (since mid-2024). `webcache.googleusercontent.com`\n  returns 200 + tens of KB, but it's a Google Search interstitial with a JS\n  redirect, not a cache. Never use it.\n- **AMP caches** (`*.cdn.ampproject.org`) mostly return a ~300-byte\n  `\u003Ctitle>Redirecting\u003C\u002Ftitle>` meta-refresh stub pointing back at the\n  original (blocked) URL. Treating that as success creates a fetch loop.\n- **Rate-limit bodies**: archive.today 429 pages are multi-KB HTML. Check for\n  the target's actual content (title words, expected strings), not just size.\n\nDetection heuristics the script applies: body under a per-route byte floor;\nmeta-refresh\u002FJS-redirect stubs whose target is the original host; interstitial\ntitles (\"Just a moment\", \"Redirecting\", \"Google Search\", \"Attention Required\").\n\n## Proxy relays: don't\n\nGeneric \"web proxy\" relays are man-in-the-middle by construction. Never send\ncookies or Authorization headers through one, and don't use them for anything\nthe user will rely on — provenance is unverifiable. Prefer archives, which at\nleast timestamp their copies.\n",{"name":5,"author":23,"license":14,"version":24,"metadata":25,"platforms":36,"description":7},"Hermes Agent","1.0.0",{"hermes":26},{"tags":27,"related_skills":34},[28,29,30,31,32,33],"Research","Archives","Wayback","Paywall","WAF","Fallback",[35],"grounded-citations",[37,38,39],"linux","macos","windows","main","81189f70e1089f9b3a944ffc480e6887efd831b9","devbox\u002Fskills\u002Fresearch\u002Fblocked-page-recovery","https:\u002F\u002Fgithub.com\u002Fdarkmatter\u002Fhermes","https:\u002F\u002Fgithub.com\u002Fdarkmatter","dm.sh","2026-09-14 08:11:52",{"source":8,"skillName":5,"commands":48,"prompt":58,"zipAvailable":59},[49,52,55],{"tool":50,"command":51},"npx","npx skills add darkmatter\u002Fhermes --skill blocked-page-recovery",{"tool":53,"command":54},"bunx","bunx skills add darkmatter\u002Fhermes --skill blocked-page-recovery",{"tool":56,"command":57},"pnpm","pnpm dlx skills add darkmatter\u002Fhermes --skill blocked-page-recovery","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fdarkmatter\u002Fhermes\u002Fblob\u002Fmain\u002Fdevbox\u002Fskills\u002Fresearch\u002Fblocked-page-recovery\u002FSKILL.md",true,[61],{"path":62,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":65,"sourceHtmlUrl":66},"scripts\u002Frecover_page.py","SCRIPT","脚本",8486,"https:\u002F\u002Fgithub.com\u002Fdarkmatter\u002Fhermes\u002Fblob\u002Fmain\u002Fdevbox\u002Fskills\u002Fresearch\u002Fblocked-page-recovery\u002Fscripts\u002Frecover_page.py",[68,75,82,89,95,102],{"skillNo":69,"name":70,"nameZh":6,"description":71,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":72,"categories":73,"keywords":74,"publishedAt":18},"01M7BMC2PYBVXM6FQEZMBQZB4G","plan","Write a markdown plan to .hermes\u002Fplans\u002F; no execution.","https:\u002F\u002Fgithub.com\u002Fdarkmatter\u002Fhermes\u002Fblob\u002Fmain\u002Fskills\u002Fsoftware-development\u002Fplan\u002FSKILL.md",[],[],{"skillNo":76,"name":77,"nameZh":6,"description":78,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":79,"categories":80,"keywords":81,"publishedAt":18},"01M7BMBSEKDVQ09Z666RCWT7BR","monorepo-fleet-sync","Sync docs and configs when a Flue monorepo fleet changes.","https:\u002F\u002Fgithub.com\u002Fdarkmatter\u002Fhermes\u002Fblob\u002Fmain\u002Fskills\u002Fsoftware-development\u002Fmonorepo-fleet-sync\u002FSKILL.md",[],[],{"skillNo":83,"name":84,"nameZh":6,"description":85,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":86,"categories":87,"keywords":88,"publishedAt":18},"01M7BMBNWFJT2TJZ2AKA64GEV9","inspecting-hermes-desktop-dom","Read the live Hermes desktop DOM\u002FCSS over CDP.","https:\u002F\u002Fgithub.com\u002Fdarkmatter\u002Fhermes\u002Fblob\u002Fmain\u002Fskills\u002Fsoftware-development\u002Finspecting-hermes-desktop-dom\u002FSKILL.md",[],[],{"skillNo":90,"name":35,"nameZh":6,"description":91,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":92,"categories":93,"keywords":94,"publishedAt":18},"01M7BMAZM4VJ97K7JD0FDN58M8","Ground answers and documents in cited, verifiable sources.","https:\u002F\u002Fgithub.com\u002Fdarkmatter\u002Fhermes\u002Fblob\u002Fmain\u002Fskills\u002Fresearch\u002Fgrounded-citations\u002FSKILL.md",[],[],{"skillNo":96,"name":97,"nameZh":6,"description":98,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":99,"categories":100,"keywords":101,"publishedAt":18},"01M7BMATPEDZ4BPP67Y4ZS0BMM","competitor-news-monitor","Watch named companies for material news; cited digests.","https:\u002F\u002Fgithub.com\u002Fdarkmatter\u002Fhermes\u002Fblob\u002Fmain\u002Fskills\u002Fresearch\u002Fcompetitor-news-monitor\u002FSKILL.md",[],[],{"skillNo":103,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":9,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":104,"categories":105,"keywords":106,"publishedAt":18},"01M7BMAQZV7YFA2SFPT7HCVCTX","https:\u002F\u002Fgithub.com\u002Fdarkmatter\u002Fhermes\u002Fblob\u002Fmain\u002Fskills\u002Fresearch\u002Fblocked-page-recovery\u002FSKILL.md",[],[]]