[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7CV9DJS1D885Y4MNV30BYYV":3,"skill-same:01M7CV9DJS1D885Y4MNV30BYYV":67},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":18,"linkOnly":19,"markdownContent":20,"markdownBody":21,"frontMatter":22,"refName":41,"commitSha":42,"skillDirectory":43,"forksCount":13,"repositoryHtmlUrl":44,"repositoryHomepageUrl":6,"ownerHtmlUrl":45,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":46,"install":47,"resources":60},"01M7CV9DJS1D885Y4MNV30BYYV","blocked-page-recovery",null,"Use when a fetch fails: 403\u002F429, paywall, WAF, bot wall.","dwikanugraha13\u002FProject-Anara","dwikanugraha13","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F63535036?v=4","01M7CTBN0DG8GDNP4QZ68W8M0A","01M7CTBN10Q015JDBYJS6AJRGN",0,"MIT","https:\u002F\u002Fgithub.com\u002Fdwikanugraha13\u002FProject-Anara\u002Fblob\u002Fmain\u002Fbackend\u002Fskills\u002Fblocked-page-recovery\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\ncategory: web\nname: blocked-page-recovery\ndescription: \"Use when a fetch fails: 403\u002F429, paywall, WAF, bot wall.\"\nversion: 1.0.0\nauthor: Anara Agent\nlicense: MIT\nplatforms: [linux, macos, windows]\nmetadata:\n  anara:\n    tags: [Research, Archives, Wayback, Paywall, WAF, Fallback]\n    related_skills: [grounded-citations]\n---\n\n# Blocked-Page Recovery\n\nWhen a page won't fetch — 403\u002F429, Cloudflare \"Just a moment...\", a paywall,\nor a bot-detection interstitial — don't give up and don't loop on the same\nURL. Third-party services often hold a **copy** of the page. Work down this\nladder, cheapest first.\n\n## The ladder\n\n```\n1. Wayback Machine  — archive.org \"available\" API  (snapshot + timestamp)\n2. archive.today    — domain rotation: archive.ph → .md → .li → .is\n3. Jina Reader      — only if JINA_API_KEY is set  (live server-side render)\n4. API-first pivot  — look for \u002Fapi\u002F, \u002Fgraphql, .json, or RSS on the same host\n5. Real browser     — browser tool as the last, most expensive resort\n```\n\nRun it in one shot with the bundled script:\n\n```bash\npython3 scripts\u002Frecover_page.py \"https:\u002F\u002Fexample.com\u002Fblocked-article\" --json\n```\n\nThe script tries each route in order, validates every body (see \"Fake\nsuccesses\" below), and prints the first genuine hit with its provenance.\n\n## Provenance discipline (non-negotiable)\n\nEvery recovered copy carries a provenance you MUST preserve when citing:\n\n| Route | Provenance | How to cite |\n|-------|-----------|-------------|\n| Wayback \u002F archive.today | `snapshot` | Cite WITH the snapshot date: \"as archived 2026-08-06\". Never present a snapshot as the live page — it may be stale. |\n| Jina Reader | `live` | Server-side re-render of the live page; cite normally. |\n| Live fetch \u002F browser | `live` | Cite normally. |\n\nIf the user needs *current* data (prices, availability, breaking news), a\nsnapshot is context, not an answer — say so explicitly and note its age.\n\n## Manual routes\n\n### 1. Wayback Machine (best provenance, try first)\n\n```bash\n# Discovery: returns closest snapshot URL + timestamp as JSON\ncurl -sL \"https:\u002F\u002Farchive.org\u002Fwayback\u002Favailable?url={URL}\"\n# Then fetch archived_snapshots.closest.url\n```\n\nFor enumerating many snapshots (or recovering deleted pages), the CDX index:\n\n```bash\ncurl -sL \"https:\u002F\u002Fweb.archive.org\u002Fcdx\u002Fsearch\u002Fcdx?url={URL}&output=json&limit=10\"\n```\n\nCDX intermittently returns 503 under load — if it does, fall back to the\n`available` API; don't retry-hammer it.\n\nWorks for: any publicly crawled URL. Fails for: robots-blocked sites,\nnever-crawled URLs, JS-only SPAs (snapshots don't render).\n\n### 2. archive.today (paywalls, deleted content)\n\nUser-submitted archives — often has paywalled news articles Wayback lacks.\nRate-limits aggressively (429) and rotates domains, so iterate:\n\n```bash\nfor d in archive.ph archive.md archive.li archive.is; do\n  curl -sL --max-time 20 \"https:\u002F\u002F$d\u002Fnewest\u002F{URL}\" -o ~\u002F.anara\u002Fcache\u002Fscratch\u002Fpage.html \\\n    -w \"%{http_code}\" && break\ndone\n```\n\n**Validate the body, not the status code** — a 429 still ships several KB of\nrate-limit HTML that looks like a success to a size check alone.\n\n### 3. Jina Reader (requires JINA_API_KEY)\n\n`r.jina.ai` re-renders the live page in a real browser server-side and\nreturns markdown. Anonymous access is dead (401 → Turnstile); a key is\nrequired:\n\n```bash\ncurl -s -H \"Authorization: Bearer $JINA_API_KEY\" \"https:\u002F\u002Fr.jina.ai\u002F{URL}\"\n```\n\nHandles JS SPAs that archives can't. Skip this route entirely when the env\nvar is unset.\n\n### 4. API-first pivot\n\nWAFs protect the HTML surface far more aggressively than the data endpoints\nbehind it. After 2-3 blocked attempts on a site, stop fighting the HTML and\nlook for:\n\n- `\u002Fapi\u002F...`, `\u002Fgraphql`, or `.json` variants of the page URL\n- An RSS\u002FAtom feed (`\u002Ffeed`, `\u002Frss`, `\u003Clink rel=\"alternate\">` in any copy\n  you did recover)\n- A sitemap (`\u002Fsitemap.xml`) revealing canonical URLs that may not be gated\n\n## Fake successes — routes that LIE\n\nThese return HTTP 200 with a plausible body that is NOT the page. The script\nrejects them automatically; reject them manually too:\n\n- **Google Cache is dead** (since mid-2024). `webcache.googleusercontent.com`\n  returns 200 + tens of KB, but it's a Google Search interstitial with a JS\n  redirect, not a cache. Never use it.\n- **AMP caches** (`*.cdn.ampproject.org`) mostly return a ~300-byte\n  `\u003Ctitle>Redirecting\u003C\u002Ftitle>` meta-refresh stub pointing back at the\n  original (blocked) URL. Treating that as success creates a fetch loop.\n- **Rate-limit bodies**: archive.today 429 pages are multi-KB HTML. Check for\n  the target's actual content (title words, expected strings), not just size.\n\nDetection heuristics the script applies: body under a per-route byte floor;\nmeta-refresh\u002FJS-redirect stubs whose target is the original host; interstitial\ntitles (\"Just a moment\", \"Redirecting\", \"Google Search\", \"Attention Required\").\n\n## Proxy relays: don't\n\nGeneric \"web proxy\" relays are man-in-the-middle by construction. Never send\ncookies or Authorization headers through one, and don't use them for anything\nthe user will rely on — provenance is unverifiable. Prefer archives, which at\nleast timestamp their copies.\n","# Blocked-Page Recovery\n\nWhen a page won't fetch — 403\u002F429, Cloudflare \"Just a moment...\", a paywall,\nor a bot-detection interstitial — don't give up and don't loop on the same\nURL. Third-party services often hold a **copy** of the page. Work down this\nladder, cheapest first.\n\n## The ladder\n\n```\n1. Wayback Machine  — archive.org \"available\" API  (snapshot + timestamp)\n2. archive.today    — domain rotation: archive.ph → .md → .li → .is\n3. Jina Reader      — only if JINA_API_KEY is set  (live server-side render)\n4. API-first pivot  — look for \u002Fapi\u002F, \u002Fgraphql, .json, or RSS on the same host\n5. Real browser     — browser tool as the last, most expensive resort\n```\n\nRun it in one shot with the bundled script:\n\n```bash\npython3 scripts\u002Frecover_page.py \"https:\u002F\u002Fexample.com\u002Fblocked-article\" --json\n```\n\nThe script tries each route in order, validates every body (see \"Fake\nsuccesses\" below), and prints the first genuine hit with its provenance.\n\n## Provenance discipline (non-negotiable)\n\nEvery recovered copy carries a provenance you MUST preserve when citing:\n\n| Route | Provenance | How to cite |\n|-------|-----------|-------------|\n| Wayback \u002F archive.today | `snapshot` | Cite WITH the snapshot date: \"as archived 2026-08-06\". Never present a snapshot as the live page — it may be stale. |\n| Jina Reader | `live` | Server-side re-render of the live page; cite normally. |\n| Live fetch \u002F browser | `live` | Cite normally. |\n\nIf the user needs *current* data (prices, availability, breaking news), a\nsnapshot is context, not an answer — say so explicitly and note its age.\n\n## Manual routes\n\n### 1. Wayback Machine (best provenance, try first)\n\n```bash\n# Discovery: returns closest snapshot URL + timestamp as JSON\ncurl -sL \"https:\u002F\u002Farchive.org\u002Fwayback\u002Favailable?url={URL}\"\n# Then fetch archived_snapshots.closest.url\n```\n\nFor enumerating many snapshots (or recovering deleted pages), the CDX index:\n\n```bash\ncurl -sL \"https:\u002F\u002Fweb.archive.org\u002Fcdx\u002Fsearch\u002Fcdx?url={URL}&output=json&limit=10\"\n```\n\nCDX intermittently returns 503 under load — if it does, fall back to the\n`available` API; don't retry-hammer it.\n\nWorks for: any publicly crawled URL. Fails for: robots-blocked sites,\nnever-crawled URLs, JS-only SPAs (snapshots don't render).\n\n### 2. archive.today (paywalls, deleted content)\n\nUser-submitted archives — often has paywalled news articles Wayback lacks.\nRate-limits aggressively (429) and rotates domains, so iterate:\n\n```bash\nfor d in archive.ph archive.md archive.li archive.is; do\n  curl -sL --max-time 20 \"https:\u002F\u002F$d\u002Fnewest\u002F{URL}\" -o ~\u002F.anara\u002Fcache\u002Fscratch\u002Fpage.html \\\n    -w \"%{http_code}\" && break\ndone\n```\n\n**Validate the body, not the status code** — a 429 still ships several KB of\nrate-limit HTML that looks like a success to a size check alone.\n\n### 3. Jina Reader (requires JINA_API_KEY)\n\n`r.jina.ai` re-renders the live page in a real browser server-side and\nreturns markdown. Anonymous access is dead (401 → Turnstile); a key is\nrequired:\n\n```bash\ncurl -s -H \"Authorization: Bearer $JINA_API_KEY\" \"https:\u002F\u002Fr.jina.ai\u002F{URL}\"\n```\n\nHandles JS SPAs that archives can't. Skip this route entirely when the env\nvar is unset.\n\n### 4. API-first pivot\n\nWAFs protect the HTML surface far more aggressively than the data endpoints\nbehind it. After 2-3 blocked attempts on a site, stop fighting the HTML and\nlook for:\n\n- `\u002Fapi\u002F...`, `\u002Fgraphql`, or `.json` variants of the page URL\n- An RSS\u002FAtom feed (`\u002Ffeed`, `\u002Frss`, `\u003Clink rel=\"alternate\">` in any copy\n  you did recover)\n- A sitemap (`\u002Fsitemap.xml`) revealing canonical URLs that may not be gated\n\n## Fake successes — routes that LIE\n\nThese return HTTP 200 with a plausible body that is NOT the page. The script\nrejects them automatically; reject them manually too:\n\n- **Google Cache is dead** (since mid-2024). `webcache.googleusercontent.com`\n  returns 200 + tens of KB, but it's a Google Search interstitial with a JS\n  redirect, not a cache. Never use it.\n- **AMP caches** (`*.cdn.ampproject.org`) mostly return a ~300-byte\n  `\u003Ctitle>Redirecting\u003C\u002Ftitle>` meta-refresh stub pointing back at the\n  original (blocked) URL. Treating that as success creates a fetch loop.\n- **Rate-limit bodies**: archive.today 429 pages are multi-KB HTML. Check for\n  the target's actual content (title words, expected strings), not just size.\n\nDetection heuristics the script applies: body under a per-route byte floor;\nmeta-refresh\u002FJS-redirect stubs whose target is the original host; interstitial\ntitles (\"Just a moment\", \"Redirecting\", \"Google Search\", \"Attention Required\").\n\n## Proxy relays: don't\n\nGeneric \"web proxy\" relays are man-in-the-middle by construction. Never send\ncookies or Authorization headers through one, and don't use them for anything\nthe user will rely on — provenance is unverifiable. Prefer archives, which at\nleast timestamp their copies.\n",{"name":5,"author":23,"license":14,"version":24,"category":25,"metadata":26,"platforms":37,"description":7},"Anara Agent","1.0.0","web",{"anara":27},{"tags":28,"related_skills":35},[29,30,31,32,33,34],"Research","Archives","Wayback","Paywall","WAF","Fallback",[36],"grounded-citations",[38,39,40],"linux","macos","windows","main","3e6d6b7ffb6ab43ddf3991509e5cfb5f8c90cd1a","backend\u002Fskills\u002Fblocked-page-recovery","https:\u002F\u002Fgithub.com\u002Fdwikanugraha13\u002FProject-Anara","https:\u002F\u002Fgithub.com\u002Fdwikanugraha13","2026-10-13 07:57:21",{"source":8,"skillName":5,"commands":48,"prompt":58,"zipAvailable":59},[49,52,55],{"tool":50,"command":51},"npx","npx skills add dwikanugraha13\u002FProject-Anara --skill blocked-page-recovery",{"tool":53,"command":54},"bunx","bunx skills add dwikanugraha13\u002FProject-Anara --skill blocked-page-recovery",{"tool":56,"command":57},"pnpm","pnpm dlx skills add dwikanugraha13\u002FProject-Anara --skill blocked-page-recovery","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fdwikanugraha13\u002FProject-Anara\u002Fblob\u002Fmain\u002Fbackend\u002Fskills\u002Fblocked-page-recovery\u002FSKILL.md",true,[61],{"path":62,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":65,"sourceHtmlUrl":66},"scripts\u002Frecover_page.py","SCRIPT","脚本",8481,"https:\u002F\u002Fgithub.com\u002Fdwikanugraha13\u002FProject-Anara\u002Fblob\u002Fmain\u002Fbackend\u002Fskills\u002Fblocked-page-recovery\u002Fscripts\u002Frecover_page.py",[68,75,82,90,97,104],{"skillNo":69,"name":70,"nameZh":6,"description":71,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":72,"categories":73,"keywords":74,"publishedAt":18},"01M7CVK8W5MPQ8H45A6XE6RPK1","youtube-content","YouTube transcripts to summaries, threads, blogs.","https:\u002F\u002Fgithub.com\u002Fdwikanugraha13\u002FProject-Anara\u002Fblob\u002Fmain\u002Fbackend\u002Fskills\u002Fyoutube-content\u002FSKILL.md",[],[],{"skillNo":76,"name":77,"nameZh":6,"description":78,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":79,"categories":80,"keywords":81,"publishedAt":18},"01M7CVK56KG9399ZZVE9KM54WS","xurl","X\u002FTwitter via xurl CLI: raw post search, posting, DM, media.","https:\u002F\u002Fgithub.com\u002Fdwikanugraha13\u002FProject-Anara\u002Fblob\u002Fmain\u002Fbackend\u002Fskills\u002Fxurl\u002FSKILL.md",[],[],{"skillNo":83,"name":84,"nameZh":6,"description":85,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":86,"sourceHtmlUrl":87,"categories":88,"keywords":89,"publishedAt":18},"01M7CVK22A44A88NPPBAC1FVQP","xlsx","Create, read, edit Excel .xlsx workbooks and CSVs.","https:\u002F\u002Fgithub.com\u002Fdwikanugraha13\u002FProject-Anara\u002Fblob\u002Fmain\u002Fbackend\u002Fskills\u002Fxlsx\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fdwikanugraha13\u002FProject-Anara\u002Fblob\u002Fmain\u002Fbackend\u002Fskills\u002Fxlsx\u002FSKILL.md",[],[],{"skillNo":91,"name":92,"nameZh":6,"description":93,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":94,"categories":95,"keywords":96,"publishedAt":18},"01M7CVJX88C8PY2T6RQZ5GCG15","weights-and-biases","W&B: log ML experiments, sweeps, model registry, dashboards.","https:\u002F\u002Fgithub.com\u002Fdwikanugraha13\u002FProject-Anara\u002Fblob\u002Fmain\u002Fbackend\u002Fskills\u002Fweights-and-biases\u002FSKILL.md",[],[],{"skillNo":98,"name":99,"nameZh":6,"description":100,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":101,"categories":102,"keywords":103,"publishedAt":18},"01M7CVJR3W1FBRGC56DG6BJ1Y3","weekly-review-planning","Weekly reset: commitments, stalled work, next-week plan.","https:\u002F\u002Fgithub.com\u002Fdwikanugraha13\u002FProject-Anara\u002Fblob\u002Fmain\u002Fbackend\u002Fskills\u002Fweekly-review-planning\u002FSKILL.md",[],[],{"skillNo":105,"name":106,"nameZh":6,"description":107,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":108,"categories":109,"keywords":110,"publishedAt":18},"01M7CVJGPJNPQDM8S1KTTMJN7M","touchdesigner-mcp","Control TouchDesigner via twozero MCP.","https:\u002F\u002Fgithub.com\u002Fdwikanugraha13\u002FProject-Anara\u002Fblob\u002Fmain\u002Fbackend\u002Fskills\u002Ftouchdesigner-mcp\u002FSKILL.md",[],[]]