[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7AVZ029D94C9HNET2TYTHNZ":3,"skill-same:01M7AVZ029D94C9HNET2TYTHNZ":80},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":43,"commitSha":44,"skillDirectory":45,"forksCount":46,"repositoryHtmlUrl":47,"repositoryHomepageUrl":6,"ownerHtmlUrl":48,"ownerBlogUrl":49,"ownerDescription":50,"updatedAt":51,"install":52,"resources":65},"01M7AVZ029D94C9HNET2TYTHNZ","technical-docs-corpora",null,"Use when bulk-crawling public tech docs into agent skills.","swcstudiospace\u002Faimeecodes","swcstudiospace","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F192491759?v=4","01M7AVGY9HD2GXDY0H77EBGRGQ","@swcstudio'(space)","01M7AVGYAKRDHFDMPG97W4D5M2",1,"MIT","https:\u002F\u002Fgithub.com\u002Fswcstudiospace\u002Faimeecodes\u002Fblob\u002Fmain\u002F.aimee\u002Fskills\u002Ftechnical-docs-corpora\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: technical-docs-corpora\ndescription: \"Use when bulk-crawling public tech docs into agent skills.\"\nversion: 1.1.1\nauthor: Hermes Agent\nlicense: MIT\nplatforms: [linux, macos, windows]\nmetadata:\n  hermes:\n    tags: [docs, corpus, scrape, skills, sqlite, fts, batch, offline]\n    related_skills: [hermes-agent-skill-authoring]\n---\n\n# Technical docs corpora → agent skills\n\n## Overview\n\nTurn public software documentation sites into **queryable local corpora** and **shared agent skills** (SKILL.md + `references\u002F`). Prefer the project `docs-scraper` when present; this skill encodes the durable workflow, allowlist\u002Frobots pitfalls, and dual-app install pattern from multi-site batch builds.\n\n**Project (this VPS):** `\u002Froot\u002F.openclaw\u002Fworkspace\u002Fprojects\u002Fdocs-scraper`  \n**Shared install root:** `~\u002Fagent-skills\u002F`  \n**Operator skill (external):** `docs-scraper` — if it needs curator patches, run `hermes curator adopt docs-scraper` (it lives under `skills.external_dirs`).\n\n## When to use\n\n- Offline skills for many eng docs sites (React, Stripe, k8s, …)\n- Bulk crawl + FTS\u002Fhybrid index + export skill packages\n- Full-site \u002F “unlimited pages” corpora (see §1b + `references\u002Ffull-corpus-crawl.md`)\n- Dual Hermes + OpenClaw sharing for doc packages\n- Empty crawls (allowlist, robots, llms.txt, sitemaps)\n\nDon't use for: private\u002Fauth docs, CAPTCHA bypass, or ignoring robots\u002FToS.\n\n## Default stack\n\n| Layer | Choice |\n|---|---|\n| SoR | SQLite + FTS5 |\n| Vectors (optional) | fastembed + sqlite-vec, RRF hybrid |\n| Fetch | httpx + markdownify (Crawl4AI optional) |\n| Shared skills | `~\u002Fagent-skills\u002F\u003Cname>\u002F` |\n\n## Workflow\n\n### 1. Single site (polite slice, default)\n\n```bash\ncd \u002Froot\u002F.openclaw\u002Fworkspace\u002Fprojects\u002Fdocs-scraper && source .venv\u002Fbin\u002Factivate\n# configs\u002F\u003Csite>.yaml — tight allowed_prefixes, delay_ms ≥ 300, honor_robots: true\npython scripts\u002Fcrawl.py --config configs\u002F\u003Csite>.yaml --max-pages 80\npython scripts\u002Fembed_index.py --db data\u002Fdb\u002F\u003Csite>.sqlite   # optional; FTS works without\npython scripts\u002Fexport_skill.py --db data\u002Fdb\u002F\u003Csite>.sqlite --name docs-\u003Csite> --max-pages 80\nln -sfn \"$PWD\u002Fdata\u002Fexports\u002Fskills\u002Fdocs-\u003Csite>\" ~\u002Fagent-skills\u002Fdocs-\u003Csite>\n# optional Hermes visibility:\nln -sfn ~\u002Fagent-skills\u002Fdocs-\u003Csite> ~\u002F.hermes\u002Fskills\u002Fdocs-\u003Csite>\n```\n\n**Done when:** `pages > 0`, `chunks > 0`, export has `references\u002F*.md`, skill listed in Hermes\u002FOpenClaw.\n\n### 1b. Full-site \u002F “unlimited pages” crawl\n\nWhen the user wants the whole docs site (not an 80-page skill slice):\n\n1. Set `policy.max_pages` very high (e.g. `100000`) — no true unlimited int; the scraper always caps the frontier.\n2. Prefer sitemap **and** `llms.txt`; do **not** rely on llms list order alone (often alpha → API\u002FOS first, core SDK late).\n3. Run crawl in **background** with a log; multi-k Mintlify sites take tens of minutes.\n4. After fetch, **index in batches**. Single end-of-run commit loses all chunks on kill\u002Ftimeout. Use batch commits in `indexer.py` and\u002For `scripts\u002Findex_missing.py` for pages still missing chunks.\n5. **Delete mega-pages before heavy FTS**: `\u002Fllms-full.txt`, giant OpenAPI YAML, or any `length(markdown) > ~200k`. One 12MB page can balloon SQLite WAL to tens of GB.\n6. Skill package ≠ full DB: SQLite is SoR; export **curated** `references\u002F` (~100–200 core pages) and point SKILL.md at `scripts\u002Fquery.py` for full-corpus FTS.\n7. Spot-check FTS with short queries (`create Agent tools`); long AND-of-many-terms queries often return empty under FTS5.\n\n```bash\npython scripts\u002Fcrawl.py --config configs\u002Fagno-docs.yaml   # example: max_pages=100000\npython -u scripts\u002Findex_missing.py --db data\u002Fdb\u002Fskills\u002Fdocs-agno.sqlite --batch 50\npython scripts\u002Fquery.py \"create Agent tools\" --db data\u002Fdb\u002Fskills\u002Fdocs-agno.sqlite -k 5 --mode fts\n```\n\nDetails: `references\u002Ffull-corpus-crawl.md` (Agno Mintlify case study).\n\n### 2. Multi-site batch (5+ sites)\n\n```bash\n# Catalog: configs\u002Fcatalog\u002Ftop20-eng-docs.yaml\npython scripts\u002Fbatch_docs_skills.py --catalog configs\u002Fcatalog\u002Ftop20-eng-docs.yaml \\\n  --max-pages 80 --skip-embed --force\n# FTS-first for speed; embed later if hybrid needed:\n# for db in data\u002Fdb\u002Fskills\u002Fdocs-*.sqlite; do python scripts\u002Fembed_index.py --db \"$db\"; done\npython scripts\u002Fbatch_docs_skills.py --only react,fastapi --max-pages 150 --force\npython scripts\u002Fbatch_docs_skills.py --resume-failed\n```\n\nState: `data\u002Fbatch\u002Ftop20-state.json`.  \n**Done when:** each source `status=ok`, pages ≥ floor, installed under `~\u002Fagent-skills\u002F`.\n\n### 3. Dual-app install (Hermes + OpenClaw)\n\n1. Canonical body under `~\u002Fagent-skills\u002F\u003Cskill>\u002F`\n2. Hermes: `skills.external_dirs: [\u002Froot\u002Fagent-skills]` as a **real YAML list** (not a quoted JSON string)\n3. OpenClaw: `skills.load.extraDirs: [\"\u002Froot\u002Fagent-skills\"]` in `openclaw.json`\n4. Optional symlinks under `~\u002F.hermes\u002Fskills\u002F` and `~\u002F.openclaw\u002Fskills\u002F`\n\n## Discovery & allowlist pitfalls\n\n1. **Trailing slash mismatch.** Normalized paths drop trailing `\u002F`. Prefix `\u002F3\u002F` must match `\u002F3` and `\u002F3\u002Flibrary\u002F...` via “equals stripped pref OR startswith stripped + `\u002F`”.\n2. **llms.txt is markdown.** Parse `[text](url)` and bare URLs. Skip `*`, `{templates}`, and junk that breaks `urlsplit` (Invalid IPv6 URL).\n3. **llms.txt order is not importance.** Many Mintlify sites list paths alphabetically. Early cap (`frontier[:max_pages]`) then yields only `agent-os` \u002F `api-reference` and starves `agents` \u002F `models` \u002F `tools`. For balanced slices: score+quota by top-level prefix, or crawl full site then curate export. `seed_frontier=` on `run_crawl` accepts a pre-ranked list.\n4. **Host drift.** Redirecting roots (docs.anthropic.com → platform.claude.com, docs.gitbook.com → gitbook.com\u002Fdocs — no `www`): set `root_url` to the **final** host so allowlist host checks pass.\n5. **robots.txt path games.** nodejs.org `Disallow: \u002Fdocs\u002F` but `Allow: \u002Fapi\u002F`. Prefer robots-allowed paths; don't default to turning robots off.\n6. **Sitemap sparse → BFS.** One seed is fine if crawl expands under allowlist; raise `max_depth` for handbook sites.\n7. **discovered > 0 but fetched_ok = 0.** Check `skipped_robots` before blaming fetch\u002Fengine.\n8. **Duplicate HTML + `.md` URLs.** Mintlify and GitBook often serve both; full crawls can nearly double page count. Harmless for FTS; **dedupe in skill export** (prefer the non-`.md` URL; strip `.md` as the uniqueness key). Shipping both wastes the ~100–200 reference budget.\n9. **Locale sitemap mirrors.** GitBook-style sitemap indexes list `documentation\u002Ffr|zh|ja-*` copies of the same EN tree. Deny those prefixes or you triple-count. Probe `sitemap.xml` children before setting `max_pages`.\n10. **llms.txt lists non-pages.** Mintlify indexes can include `\u002Fdocs\u002Fbin\u002F*.sh` and `\u002Fdocs\u002Fscripts\u002F*.sh` that 404. Deny `\u002Fdocs\u002Fbin` and `\u002Fdocs\u002Fscripts`, or treat those `fetched_fail` as expected.\n11. **`llms-full.txt` pollutes FTS even when it is “only” ~200–300k.** The concat dump often steals another page’s title (Greptile: titled “Analytics Dashboard”) and outranks real pages. Deny `\u002Fdocs\u002Fllms-full.txt` (and `\u002Fdocs\u002Fllms-full.txt\u002F1`) in config; delete matching rows **before** trusting FTS or export. This is separate from WAL-death on multi-MB dumps.\n\n## Skill package shape\n\n- Frontmatter `name` + trigger-focused `description`\n- Runbook: cite `references\u002F` Source URLs; don't invent APIs\n- Optional SQLite path for hybrid\u002FFTS query of the full corpus\n- **No secrets in SKILL.md.** Avoid OpenClaw `requires.env` for optional keys (marks skill “needs setup”)\n\n## Query\n\n```bash\npython scripts\u002Fquery.py \"question\" --db data\u002Fdb\u002Fskills\u002Fdocs-\u003Cx>.sqlite -k 8 --mode auto\n# modes: auto | fts | vector | hybrid\n# Prefer short queries (2–4 tokens). Long AND chains often return zero under FTS5.\n```\n\n## Common Pitfalls\n\n1. **Vectors on first batch pass** — slow; ship FTS with `--skip-embed`, embed offline later.\n2. **Forking skills per app** — drift; always symlink from `~\u002Fagent-skills`.\n3. **external_dirs as a string** — tree never loads; fix to YAML sequence.\n4. **Patching external_dirs skills via agent** — refused; edit project + re-export, or `hermes curator adopt \u003Cname>`.\n5. **Loose allowlists on MDN-scale sites** — keep caps polite (≤100 unless asked for full corpus).\n6. **Single end-of-run FTS commit on multi-k pages** — kill\u002Ftimeout → `chunks=0` despite pages stored. Batch commits or `index_missing.py`; `PRAGMA wal_checkpoint(TRUNCATE)` after kills.\n7. **Mega-page WAL death** — indexing `llms-full.txt` \u002F huge OpenAPI as one page can grow WAL to 30GB+ and hit “database or disk is full”. Delete those rows before indexing; monitor `*.sqlite-wal` size.\n8. **`llms-full.txt` FTS title theft** — a 200–300k concat dump can stay under the WAL threshold and still dominate FTS with a wrong title. After crawl, `DELETE` rows where `url LIKE '%llms-full%'` or `length(markdown) > 200000`, then spot-check a short FTS query before export.\n9. **Export `--query` with many AND terms** — FTS5 may return 1 page or none; export without query or with a short query, or build curated references via SQL prefix quotas. Default `export_skill.py` is recency-ordered and will include `.md` dupes + `llms.txt` unless you filter.\n10. **API-heavy frontiers** — GitBook `developers\u002Fgitbook-api` can be ~60% of discovered URLs. Crawl full EN if asked; curate the skill package with prefix quotas (getting-started \u002F create-content \u002F publish \u002F git-sync first). Point SKILL.md at `scripts\u002Fquery.py` for the API tail.\n11. **`docs-scraper` is external** — durable project code fixes go in the docs-scraper repo; this skill only carries operator workflow.\n12. **Probe robots \u002F llms \u002F sitemap with curl** (plain-text endpoints). Do not use the browser stack or a paid extract API for `robots.txt`, `llms.txt`, or `sitemap.xml`.\n\n## Verification Checklist\n\n- [ ] Dry-run discover counts before full crawl\n- [ ] robots honored; empty runs checked for skipped_robots\n- [ ] Each skill: pages > 0, chunks > 0, references + SKILL.md\n- [ ] Full corpus: `indexed_pages == pages`, no `length(markdown) > 200k`, WAL not multi-GB\n- [ ] Under `~\u002Fagent-skills` and visible to Hermes + OpenClaw\n- [ ] Spot FTS query returns on-host citations (short query); first hits are not `llms-full.txt` or a stolen title\n- [ ] Skill `references\u002F` are unique HTML pages (no `.md` duplicate, no `llms*.txt`)\n\n## Support files\n\n- `references\u002Fbatch-catalog-notes.md` — site quirks from the top-20 eng docs batch\n- `references\u002Ffull-corpus-crawl.md` — unlimited\u002Ffull-site crawl runbook + Agno Mintlify case study\n- `references\u002Fgitbook-greptile-crawl.md` — Greptile Mintlify + GitBook EN corpus recipe (locale deny, llms-full, unique-HTML export)\n","# Technical docs corpora → agent skills\n\n## Overview\n\nTurn public software documentation sites into **queryable local corpora** and **shared agent skills** (SKILL.md + `references\u002F`). Prefer the project `docs-scraper` when present; this skill encodes the durable workflow, allowlist\u002Frobots pitfalls, and dual-app install pattern from multi-site batch builds.\n\n**Project (this VPS):** `\u002Froot\u002F.openclaw\u002Fworkspace\u002Fprojects\u002Fdocs-scraper`  \n**Shared install root:** `~\u002Fagent-skills\u002F`  \n**Operator skill (external):** `docs-scraper` — if it needs curator patches, run `hermes curator adopt docs-scraper` (it lives under `skills.external_dirs`).\n\n## When to use\n\n- Offline skills for many eng docs sites (React, Stripe, k8s, …)\n- Bulk crawl + FTS\u002Fhybrid index + export skill packages\n- Full-site \u002F “unlimited pages” corpora (see §1b + `references\u002Ffull-corpus-crawl.md`)\n- Dual Hermes + OpenClaw sharing for doc packages\n- Empty crawls (allowlist, robots, llms.txt, sitemaps)\n\nDon't use for: private\u002Fauth docs, CAPTCHA bypass, or ignoring robots\u002FToS.\n\n## Default stack\n\n| Layer | Choice |\n|---|---|\n| SoR | SQLite + FTS5 |\n| Vectors (optional) | fastembed + sqlite-vec, RRF hybrid |\n| Fetch | httpx + markdownify (Crawl4AI optional) |\n| Shared skills | `~\u002Fagent-skills\u002F\u003Cname>\u002F` |\n\n## Workflow\n\n### 1. Single site (polite slice, default)\n\n```bash\ncd \u002Froot\u002F.openclaw\u002Fworkspace\u002Fprojects\u002Fdocs-scraper && source .venv\u002Fbin\u002Factivate\n# configs\u002F\u003Csite>.yaml — tight allowed_prefixes, delay_ms ≥ 300, honor_robots: true\npython scripts\u002Fcrawl.py --config configs\u002F\u003Csite>.yaml --max-pages 80\npython scripts\u002Fembed_index.py --db data\u002Fdb\u002F\u003Csite>.sqlite   # optional; FTS works without\npython scripts\u002Fexport_skill.py --db data\u002Fdb\u002F\u003Csite>.sqlite --name docs-\u003Csite> --max-pages 80\nln -sfn \"$PWD\u002Fdata\u002Fexports\u002Fskills\u002Fdocs-\u003Csite>\" ~\u002Fagent-skills\u002Fdocs-\u003Csite>\n# optional Hermes visibility:\nln -sfn ~\u002Fagent-skills\u002Fdocs-\u003Csite> ~\u002F.hermes\u002Fskills\u002Fdocs-\u003Csite>\n```\n\n**Done when:** `pages > 0`, `chunks > 0`, export has `references\u002F*.md`, skill listed in Hermes\u002FOpenClaw.\n\n### 1b. Full-site \u002F “unlimited pages” crawl\n\nWhen the user wants the whole docs site (not an 80-page skill slice):\n\n1. Set `policy.max_pages` very high (e.g. `100000`) — no true unlimited int; the scraper always caps the frontier.\n2. Prefer sitemap **and** `llms.txt`; do **not** rely on llms list order alone (often alpha → API\u002FOS first, core SDK late).\n3. Run crawl in **background** with a log; multi-k Mintlify sites take tens of minutes.\n4. After fetch, **index in batches**. Single end-of-run commit loses all chunks on kill\u002Ftimeout. Use batch commits in `indexer.py` and\u002For `scripts\u002Findex_missing.py` for pages still missing chunks.\n5. **Delete mega-pages before heavy FTS**: `\u002Fllms-full.txt`, giant OpenAPI YAML, or any `length(markdown) > ~200k`. One 12MB page can balloon SQLite WAL to tens of GB.\n6. Skill package ≠ full DB: SQLite is SoR; export **curated** `references\u002F` (~100–200 core pages) and point SKILL.md at `scripts\u002Fquery.py` for full-corpus FTS.\n7. Spot-check FTS with short queries (`create Agent tools`); long AND-of-many-terms queries often return empty under FTS5.\n\n```bash\npython scripts\u002Fcrawl.py --config configs\u002Fagno-docs.yaml   # example: max_pages=100000\npython -u scripts\u002Findex_missing.py --db data\u002Fdb\u002Fskills\u002Fdocs-agno.sqlite --batch 50\npython scripts\u002Fquery.py \"create Agent tools\" --db data\u002Fdb\u002Fskills\u002Fdocs-agno.sqlite -k 5 --mode fts\n```\n\nDetails: `references\u002Ffull-corpus-crawl.md` (Agno Mintlify case study).\n\n### 2. Multi-site batch (5+ sites)\n\n```bash\n# Catalog: configs\u002Fcatalog\u002Ftop20-eng-docs.yaml\npython scripts\u002Fbatch_docs_skills.py --catalog configs\u002Fcatalog\u002Ftop20-eng-docs.yaml \\\n  --max-pages 80 --skip-embed --force\n# FTS-first for speed; embed later if hybrid needed:\n# for db in data\u002Fdb\u002Fskills\u002Fdocs-*.sqlite; do python scripts\u002Fembed_index.py --db \"$db\"; done\npython scripts\u002Fbatch_docs_skills.py --only react,fastapi --max-pages 150 --force\npython scripts\u002Fbatch_docs_skills.py --resume-failed\n```\n\nState: `data\u002Fbatch\u002Ftop20-state.json`.  \n**Done when:** each source `status=ok`, pages ≥ floor, installed under `~\u002Fagent-skills\u002F`.\n\n### 3. Dual-app install (Hermes + OpenClaw)\n\n1. Canonical body under `~\u002Fagent-skills\u002F\u003Cskill>\u002F`\n2. Hermes: `skills.external_dirs: [\u002Froot\u002Fagent-skills]` as a **real YAML list** (not a quoted JSON string)\n3. OpenClaw: `skills.load.extraDirs: [\"\u002Froot\u002Fagent-skills\"]` in `openclaw.json`\n4. Optional symlinks under `~\u002F.hermes\u002Fskills\u002F` and `~\u002F.openclaw\u002Fskills\u002F`\n\n## Discovery & allowlist pitfalls\n\n1. **Trailing slash mismatch.** Normalized paths drop trailing `\u002F`. Prefix `\u002F3\u002F` must match `\u002F3` and `\u002F3\u002Flibrary\u002F...` via “equals stripped pref OR startswith stripped + `\u002F`”.\n2. **llms.txt is markdown.** Parse `[text](url)` and bare URLs. Skip `*`, `{templates}`, and junk that breaks `urlsplit` (Invalid IPv6 URL).\n3. **llms.txt order is not importance.** Many Mintlify sites list paths alphabetically. Early cap (`frontier[:max_pages]`) then yields only `agent-os` \u002F `api-reference` and starves `agents` \u002F `models` \u002F `tools`. For balanced slices: score+quota by top-level prefix, or crawl full site then curate export. `seed_frontier=` on `run_crawl` accepts a pre-ranked list.\n4. **Host drift.** Redirecting roots (docs.anthropic.com → platform.claude.com, docs.gitbook.com → gitbook.com\u002Fdocs — no `www`): set `root_url` to the **final** host so allowlist host checks pass.\n5. **robots.txt path games.** nodejs.org `Disallow: \u002Fdocs\u002F` but `Allow: \u002Fapi\u002F`. Prefer robots-allowed paths; don't default to turning robots off.\n6. **Sitemap sparse → BFS.** One seed is fine if crawl expands under allowlist; raise `max_depth` for handbook sites.\n7. **discovered > 0 but fetched_ok = 0.** Check `skipped_robots` before blaming fetch\u002Fengine.\n8. **Duplicate HTML + `.md` URLs.** Mintlify and GitBook often serve both; full crawls can nearly double page count. Harmless for FTS; **dedupe in skill export** (prefer the non-`.md` URL; strip `.md` as the uniqueness key). Shipping both wastes the ~100–200 reference budget.\n9. **Locale sitemap mirrors.** GitBook-style sitemap indexes list `documentation\u002Ffr|zh|ja-*` copies of the same EN tree. Deny those prefixes or you triple-count. Probe `sitemap.xml` children before setting `max_pages`.\n10. **llms.txt lists non-pages.** Mintlify indexes can include `\u002Fdocs\u002Fbin\u002F*.sh` and `\u002Fdocs\u002Fscripts\u002F*.sh` that 404. Deny `\u002Fdocs\u002Fbin` and `\u002Fdocs\u002Fscripts`, or treat those `fetched_fail` as expected.\n11. **`llms-full.txt` pollutes FTS even when it is “only” ~200–300k.** The concat dump often steals another page’s title (Greptile: titled “Analytics Dashboard”) and outranks real pages. Deny `\u002Fdocs\u002Fllms-full.txt` (and `\u002Fdocs\u002Fllms-full.txt\u002F1`) in config; delete matching rows **before** trusting FTS or export. This is separate from WAL-death on multi-MB dumps.\n\n## Skill package shape\n\n- Frontmatter `name` + trigger-focused `description`\n- Runbook: cite `references\u002F` Source URLs; don't invent APIs\n- Optional SQLite path for hybrid\u002FFTS query of the full corpus\n- **No secrets in SKILL.md.** Avoid OpenClaw `requires.env` for optional keys (marks skill “needs setup”)\n\n## Query\n\n```bash\npython scripts\u002Fquery.py \"question\" --db data\u002Fdb\u002Fskills\u002Fdocs-\u003Cx>.sqlite -k 8 --mode auto\n# modes: auto | fts | vector | hybrid\n# Prefer short queries (2–4 tokens). Long AND chains often return zero under FTS5.\n```\n\n## Common Pitfalls\n\n1. **Vectors on first batch pass** — slow; ship FTS with `--skip-embed`, embed offline later.\n2. **Forking skills per app** — drift; always symlink from `~\u002Fagent-skills`.\n3. **external_dirs as a string** — tree never loads; fix to YAML sequence.\n4. **Patching external_dirs skills via agent** — refused; edit project + re-export, or `hermes curator adopt \u003Cname>`.\n5. **Loose allowlists on MDN-scale sites** — keep caps polite (≤100 unless asked for full corpus).\n6. **Single end-of-run FTS commit on multi-k pages** — kill\u002Ftimeout → `chunks=0` despite pages stored. Batch commits or `index_missing.py`; `PRAGMA wal_checkpoint(TRUNCATE)` after kills.\n7. **Mega-page WAL death** — indexing `llms-full.txt` \u002F huge OpenAPI as one page can grow WAL to 30GB+ and hit “database or disk is full”. Delete those rows before indexing; monitor `*.sqlite-wal` size.\n8. **`llms-full.txt` FTS title theft** — a 200–300k concat dump can stay under the WAL threshold and still dominate FTS with a wrong title. After crawl, `DELETE` rows where `url LIKE '%llms-full%'` or `length(markdown) > 200000`, then spot-check a short FTS query before export.\n9. **Export `--query` with many AND terms** — FTS5 may return 1 page or none; export without query or with a short query, or build curated references via SQL prefix quotas. Default `export_skill.py` is recency-ordered and will include `.md` dupes + `llms.txt` unless you filter.\n10. **API-heavy frontiers** — GitBook `developers\u002Fgitbook-api` can be ~60% of discovered URLs. Crawl full EN if asked; curate the skill package with prefix quotas (getting-started \u002F create-content \u002F publish \u002F git-sync first). Point SKILL.md at `scripts\u002Fquery.py` for the API tail.\n11. **`docs-scraper` is external** — durable project code fixes go in the docs-scraper repo; this skill only carries operator workflow.\n12. **Probe robots \u002F llms \u002F sitemap with curl** (plain-text endpoints). Do not use the browser stack or a paid extract API for `robots.txt`, `llms.txt`, or `sitemap.xml`.\n\n## Verification Checklist\n\n- [ ] Dry-run discover counts before full crawl\n- [ ] robots honored; empty runs checked for skipped_robots\n- [ ] Each skill: pages > 0, chunks > 0, references + SKILL.md\n- [ ] Full corpus: `indexed_pages == pages`, no `length(markdown) > 200k`, WAL not multi-GB\n- [ ] Under `~\u002Fagent-skills` and visible to Hermes + OpenClaw\n- [ ] Spot FTS query returns on-host citations (short query); first hits are not `llms-full.txt` or a stolen title\n- [ ] Skill `references\u002F` are unique HTML pages (no `.md` duplicate, no `llms*.txt`)\n\n## Support files\n\n- `references\u002Fbatch-catalog-notes.md` — site quirks from the top-20 eng docs batch\n- `references\u002Ffull-corpus-crawl.md` — unlimited\u002Ffull-site crawl runbook + Agno Mintlify case study\n- `references\u002Fgitbook-greptile-crawl.md` — Greptile Mintlify + GitBook EN corpus recipe (locale deny, llms-full, unique-HTML export)\n",{"name":5,"author":24,"license":15,"version":25,"metadata":26,"platforms":39,"description":7},"Hermes Agent","1.1.1",{"hermes":27},{"tags":28,"related_skills":37},[29,30,31,32,33,34,35,36],"docs","corpus","scrape","skills","sqlite","fts","batch","offline",[38],"hermes-agent-skill-authoring",[40,41,42],"linux","macos","windows","main","1532f6e28b7d7cc5ae38dfe70c4e886157e8f01e",".aimee\u002Fskills\u002Ftechnical-docs-corpora",0,"https:\u002F\u002Fgithub.com\u002Fswcstudiospace\u002Faimeecodes","https:\u002F\u002Fgithub.com\u002Fswcstudiospace","https:\u002F\u002Fwww.swcstudio.space\u002F","APAC's premier WEB3.ARL Studio for programming advanced best-in-class innovations'","2026-11-11 21:45:42",{"source":8,"skillName":5,"commands":53,"prompt":63,"zipAvailable":64},[54,57,60],{"tool":55,"command":56},"npx","npx skills add swcstudiospace\u002Faimeecodes --skill technical-docs-corpora",{"tool":58,"command":59},"bunx","bunx skills add swcstudiospace\u002Faimeecodes --skill technical-docs-corpora",{"tool":61,"command":62},"pnpm","pnpm dlx skills add swcstudiospace\u002Faimeecodes --skill technical-docs-corpora","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fswcstudiospace\u002Faimeecodes\u002Fblob\u002Fmain\u002F.aimee\u002Fskills\u002Ftechnical-docs-corpora\u002FSKILL.md",true,[66,72,76],{"path":67,"resourceType":68,"resourceTypeDesc":69,"sizeBytes":70,"sourceHtmlUrl":71},"references\u002Fbatch-catalog-notes.md","REFERENCE","参考文档",2986,"https:\u002F\u002Fgithub.com\u002Fswcstudiospace\u002Faimeecodes\u002Fblob\u002Fmain\u002F.aimee\u002Fskills\u002Ftechnical-docs-corpora\u002Freferences\u002Fbatch-catalog-notes.md",{"path":73,"resourceType":68,"resourceTypeDesc":69,"sizeBytes":74,"sourceHtmlUrl":75},"references\u002Ffull-corpus-crawl.md",3091,"https:\u002F\u002Fgithub.com\u002Fswcstudiospace\u002Faimeecodes\u002Fblob\u002Fmain\u002F.aimee\u002Fskills\u002Ftechnical-docs-corpora\u002Freferences\u002Ffull-corpus-crawl.md",{"path":77,"resourceType":68,"resourceTypeDesc":69,"sizeBytes":78,"sourceHtmlUrl":79},"references\u002Fgitbook-greptile-crawl.md",1983,"https:\u002F\u002Fgithub.com\u002Fswcstudiospace\u002Faimeecodes\u002Fblob\u002Fmain\u002F.aimee\u002Fskills\u002Ftechnical-docs-corpora\u002Freferences\u002Fgitbook-greptile-crawl.md",[81,88,95,102,109,116],{"skillNo":82,"name":83,"nameZh":6,"description":84,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":85,"categories":86,"keywords":87,"publishedAt":19},"01M7AW01SRKZDGAMSENA2KKV1C","xurl","X\u002FTwitter via xurl CLI: raw post search, posting, DM, media.","https:\u002F\u002Fgithub.com\u002Fswcstudiospace\u002Faimeecodes\u002Fblob\u002Fmain\u002F.aimee\u002Fskills\u002Fxurl\u002FSKILL.md",[],[],{"skillNo":89,"name":90,"nameZh":6,"description":91,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":92,"categories":93,"keywords":94,"publishedAt":19},"01M7AVZKD3FMV3ZPX50DY7XZZR","weights-and-biases","W&B: log ML experiments, sweeps, model registry, dashboards.","https:\u002F\u002Fgithub.com\u002Fswcstudiospace\u002Faimeecodes\u002Fblob\u002Fmain\u002F.aimee\u002Fskills\u002Fweights-and-biases\u002FSKILL.md",[],[],{"skillNo":96,"name":97,"nameZh":6,"description":98,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":99,"categories":100,"keywords":101,"publishedAt":19},"01M7AVZEY5AS893YVWYY4DFQKK","touchdesigner-mcp","Control TouchDesigner via twozero MCP.","https:\u002F\u002Fgithub.com\u002Fswcstudiospace\u002Faimeecodes\u002Fblob\u002Fmain\u002F.aimee\u002Fskills\u002Ftouchdesigner-mcp\u002FSKILL.md",[],[],{"skillNo":103,"name":104,"nameZh":6,"description":105,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":106,"categories":107,"keywords":108,"publishedAt":19},"01M7AVZ4K45X76C09FQQMC9GH0","test-driven-development","TDD: enforce RED-GREEN-REFACTOR, tests before code.","https:\u002F\u002Fgithub.com\u002Fswcstudiospace\u002Faimeecodes\u002Fblob\u002Fmain\u002F.aimee\u002Fskills\u002Ftest-driven-development\u002FSKILL.md",[],[],{"skillNo":110,"name":111,"nameZh":6,"description":112,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":113,"categories":114,"keywords":115,"publishedAt":19},"01M7AVYTXSJWYYD7GZMTTR9BC0","teams-meeting-pipeline","Teams meeting summaries, job replay, Graph subscriptions.","https:\u002F\u002Fgithub.com\u002Fswcstudiospace\u002Faimeecodes\u002Fblob\u002Fmain\u002F.aimee\u002Fskills\u002Fteams-meeting-pipeline\u002FSKILL.md",[],[],{"skillNo":117,"name":118,"nameZh":6,"description":119,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":120,"categories":121,"keywords":122,"publishedAt":19},"01M7AVYQM0HYH07KQ644AWV4TS","tailscale-serve-host-edge","Use when exposing apps on Tailscale HTTPS paths or k3s edge.","https:\u002F\u002Fgithub.com\u002Fswcstudiospace\u002Faimeecodes\u002Fblob\u002Fmain\u002F.aimee\u002Fskills\u002Ftailscale-serve-host-edge\u002FSKILL.md",[],[]]