[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7B0M12B80YT5XBGHHB52PBY":3,"skill-same:01M7B0M12B80YT5XBGHHB52PBY":59},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":18,"linkOnly":19,"markdownContent":20,"markdownBody":21,"frontMatter":22,"refName":39,"commitSha":40,"skillDirectory":41,"forksCount":13,"repositoryHtmlUrl":42,"repositoryHomepageUrl":6,"ownerHtmlUrl":43,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":44,"install":45,"resources":58},"01M7B0M12B80YT5XBGHHB52PBY","ci-debugging",null,"Use when a CI\u002Ftest failure seems to have one cause but the real root is different, or when a silent failure hides the real issue.","gitricko\u002F.minions","gitricko","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F18060558?v=4","01M7B0KG9EHG0CZ0QT1EN5YPCR","01M7B0KGA0ZV14R7SKBF3B5SCT",2,"MIT","https:\u002F\u002Fgithub.com\u002Fgitricko\u002F.minions\u002Fblob\u002Fmain\u002Fskills\u002Fci-debugging\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: ci-debugging\ndescription: Use when a CI\u002Ftest failure seems to have one cause but the real root is different, or when a silent failure hides the real issue.\nversion: 1.0.0\nauthor: Hermes Agent\nlicense: MIT\nplatforms: [linux]\nmetadata:\n  hermes:\n    tags: [ci, debugging, root-cause, tests, github-actions]\n    related_skills: [github-codespace, karpathy-coding-guidelines, tdd]\n---\n\n# CI Debugging — Find the Root Cause\n\nDebug CI failures by comparing passing vs failing commits, never by weakening the test.\n\n## The core rule\n\n**Fix the root cause, never weaken the test.**\n\nWhen a fix seems to work, verify it actually fixed the right thing. If a test\nis flaky or \"fails sometimes\", the failure is telling you something about the\ncode — not about the test.\n\n## The 5-step procedure\n\n1. **Get the failing commit's full log.**\n   ```bash\n   gh run view \u003Crun-id> --log | grep -E \"FAIL|Error|✗\" | head -20\n   # identify the exact step and line\n   ```\n\n2. **Compare passing vs failing commits.**\n   ```bash\n   git log --oneline \u003Cpassing>..\u003Cfailing>\n   git diff \u003Cpassing>..\u003Cfailing> -- \u003Csuspect-file>\n   ```\n   A \"silent failure\" (process killed before writing output) gives you zero\n   error info — the only signal is which commit changed the behavior.\n\n3. **Reproduce locally in isolation** (not by guessing).\n   Use the repo's test harness (`dts` \u002F `scripts\u002Fdts.sh`) to run the exact\n   failing command inside a throwaway container, so the host environment\n   can't mask the bug.\n\n4. **Fix the root cause, then re-run the original test.**\n   If your fix \"works\" but the test had to be weakened, you fixed the wrong\n   thing.\n\n5. **Verify with a clean test run** — not just the one case that failed.\n   Run the full suite; a root-cause fix should not require relaxing other\n   tests.\n\n## Step 0: Ensure logs are captured *before* debugging\n\nIf probes redirect to `\u002Fdev\u002Fnull`, there is nothing to debug with.\nBefore any debugging, verify the test suite captures failure output\nto files that get uploaded as CI artifacts.\n\nSee the `ci-log-capture` skill for the pattern: replace\n`>\u002Fdev\u002Fnull 2>&1` with file captures to `${CI_LOGS_DIR}` so that:\n- On **success**: clean `[PASS]` in CI log, full output sits quietly in artifact\n- On **failure**: `[FAIL]` + the full captured body is `cat`-ed to stderr\n  (visible in CI log immediately) AND persisted to artifact for later analysis\n\n### The `run_and_log` helper pattern\n\n```sh\n# Add after CI_LOGS_DIR setup:\nrun_and_log() {\n    local test_name=\"$1\"; shift\n    local log_file=\"${CI_LOGS_DIR}\u002F${test_name}.log\"\n    if \"$@\" >\"${log_file}\" 2>&1; then\n        log_info \"${test_name} — OK\"\n        return 0\n    else\n        log_error \"${test_name} — FAILED (see ${log_file})\"\n        cat \"${log_file}\"   # print to CI log for immediate visibility\n        return 1\n    fi\n}\n```\n\n### Which probes to capture\n\n| Category | Examples | Why |\n|----------|----------|-----|\n| **Version checks** | `hermes --version`, `9router --version` | Confirms binary installed correctly |\n| **Health\u002Fmodel checks** | `\u002Fv1\u002Fmodels` curl probes | Shows HTTP status, model count |\n| **Chat completion** | Chat completion endpoints | Shows **400\u002F401\u002F403 body** from upstream OC bug |\n| **Preconfig** | 9Router\u002FOmniRoute login\u002Fsetup | Shows which step failed in the chain |\n\nAll captured files must match the artifact upload path (`path: ${{ env.CI_LOGS_DIR }}\u002F`)\nso they're downloaded on failure. See `ci-log-capture` skill.\n\n## Silent failure patterns (hardest to debug)\n\n| Symptom | Likely cause | How to find it |\n|---------|--------------|----------------|\n| Process exits 0 but no output written | Process killed early (OOM, timeout) before flush | Compare exit code + check for killed processes in CI log |\n| Test \"passes\" but artifact is empty | `if-no-files-found: warn` masks empty upload | Check artifact size\u002Fcontent, not just job status |\n| Fail only in CI, never locally | NGINX\u002FCI PATH differs, env var unset, runner image change | Compare `env` output between local and CI |\n| Fail only sometimes (flaky) | Race condition, retry logic, provider rate-limit | Look for \"retry\" loops and timeouts; bump tolerance only if truly ephemeral |\n| `rm -rf` reports success but `verify CLEAN` fails with leftover mount | Docker volume mount (`-v host:container\u002Fpath`) blocks removal — `rm -rf` cannot delete a mountpoint | Check `mount` output for `$HOME` in failure logs; unmount with `umount -l` plus `sudo umount -l` fallback (container runs as uid 1000) before retrying removal |\n| `--keep-config` test fails \"removed X (should preserve)\" | Install never created X because its setup is disabled by default | Ensure install creates empty placeholder dirs (`mkdir -p ~\u002F.mnemon ~\u002F.ollama ~\u002F.9router`) even when disabled so the uninstall contract has something to preserve |\n| Uninstall test deletes real `~\u002F.mnemon`\u002F`~\u002F.hermes` on host\u002Fcodespace | Test ran with real `$HOME` instead of isolated temp dir — codespace mnemon is the memory layer, deleting it kills the session's model proxy | Always run uninstall\u002Fpurge tests with `HOME=$(mktemp -d)` and clean up; never `rm -rf` with real `$HOME` in tests or scripts |\n| CLI env var silently ignored (e.g. `TAILSCALE_SOCKET` ignored by v1.104.1) `fails to connect` on custom socket | `bin --socket` works but `ENV=... bin status` dials default path | Probe with both forms live: `ENV=... bin status` vs `bin --socket=... status`; if only the flag succeeds, fix all probes\u002Fwrappers to use `--socket` and ship a `bin\u002Fts` wrapper that always passes it — never rely on the env var |\n| `--socket=~\u002Fpath` dials literal `~` and fails | `~` only expands at word start or after `:` in `PATH`-like assignments; `--flag=~\u002F...` is not an assignment | Always use `$HOME` or `$MINIONS_HOME` in flags and wrappers (`--socket=\"$HOME\u002F.minions\u002F...\"`), and use `${MINIONS_HOME:-$HOME\u002F.minions}` in wrappers so CI\u002F`ssh`\u002F`sudo` resolve correctly |\n| `var=$(failing_cmd); rc=$?` error handler never runs and log stays empty | `set -e` aborts at a failing command substitution before the next line, so the `[FAIL] + tail` branch is dead code | Wrap the capture: `set +e; var=$(cmd 2>&1); rc=$?; set -e` so the handler can print diagnostics |\n| `mkdir` fails \"Read-only filesystem\" or \"Permission denied\" on a path that is a symlink | Dir symlink `lib\u002Follama -> \u002Ftmp\u002Fcache:ro` makes `mkdir -p lib\u002Follama` walk through the ro mount; or the base image pre-creates `$HOME` owned by root and a conditional `[ -d $home ] or chown` skipped the fix | Mount cache outside the tree (`\u002Ftmp\u002Fcache:ro`) and symlink files not dirs (`lib\u002Follama\u002Follama -> \u002Ftmp\u002Fcache\u002Follama`); always `mkdir -p $home && chown` unconditionally |\n| Cache miss causes re-download then a different downstream failure | Version string carries `v` prefix (`v0.40.0` in deps.yaml) but cache dir is `ollama-0.40.0-...` — un-stripped key leaves `OLLAMA_HOST_CACHE` empty so the container mounts nothing | Normalize before use: `version=${version#v}` and map arch aliases before building `OLLAMA_HOST_CACHE` |\n| `bash install.sh \\| tee log` reports success while install OOMed | Pipe exit is `tee`'s 0 when `pipefail` is off, masking left-side failure — downstream keep-config then sees missing dot dirs | Prefix the pipe: `set -o pipefail; cmd \\| tee log` so failure propagates |\n| `npm install` OOM `JavaScript heap out of memory` in constrained container | `NODE_OPTIONS=--max-old-space-size=512` too small for large package (e.g. `omniroute@3.8.x`) under DTS\u002FCI limits | Raise heap in shared `install_npm_package` wrapper (512 → 2048) |\n| `.tar.zst` extract fails `stdin: not in gzip format` or `zstd required` | DTS container lacks `zstd` and `tar -xzf` handles gzip only — `.tar.zst` needs `zstd -dc \\| tar -xf` or `tar --zstd -xf` | Install `zstd` in DTS prereqs and fix `lib\u002Fdownload.sh` `.tar.zst` branch to use `tar --zstd -xf` fallback; add `\u002Ftmp\u002Follama-cache\u002Follama` host-cache copy fast-path in `lib\u002Follama.sh` to skip re-download on reinstall |\n| `Dependency drift (advisory)` fails but overall CI still green | Drift is `continue-on-error: true` — the job fails to surface `OUTDATED` but does not block merge; easy to ignore until 4-file sync invariant breaks `version-sync` | Run `check-updates.sh --json` to identify the outdated pin, then `bump.sh DEP=VERSION --open-pr` + `sync-readme-badges.sh` to fix all 4 files (see `version-single-source` skill) |\n| `self-check.sh --ci` exit 1 treated as failure after reinstall | `--ci` exits 1 for warnings (1 warning, 0 critical) and 2 for critical — the pre-reinstall health check tolerates warnings; a new strict `or fail` after reinstall breaks on advisory warnings | Fail only on critical: `set +e; bash self-check.sh --ci; SC=$?; set -e; [ \"$SC\" -ge 2 ] && fail` — tolerate `SC=1` |\n| `DTS Integration` cancelled at `timeout-minutes: 60` mid `stop.sh` (all 8 tests PASS) | Single DTS run ~38min; 3-retry loop needs ~115min headroom worst case — wall timeout kills mid-test and reports `cancelled`, not a code defect | Bump `dts-integration.timeout-minutes` to 75 (or 90 for 3 retries); verify via `gh api ...\u002Fruns\u002F\u003Cid>\u002Fjobs` duration |\n| `Real Install` `npm 404 pi-ai-1.1.0.tgz` while PI pinned `1.0.4` | `pi-coding-agent@1.0.4` deps `^1.0.4` float to `pi-ai@1.1.0`; registry propagation can 404 transiently | Bump PI via `bump.sh PI=1.1.0` (+ badges) so install pins `1.1.0` directly; avoid fixing via rerun alone |\n| `verify-knowledge.sh` F5 reports `Unknown function: F5` after printing `WARN: .devcontainer references found` | `echo \"$hits\" \\| head -20` always SIGPIPEs (exit 141) when hits exceed 20 lines; with `set -euo pipefail` the pipeline fails, and `type -t \"$fn\" && \"$fn\" \\|\\| { echo Unknown; exit 1; }` misattributes any function failure as a missing function | Append `\\|\\| true` after `head -20` and split dispatch into `if ! type -t \"$fn\"; then echo Unknown; exit 1; fi; \"$fn\"` so real failures surface with their own output |\n| `boot.sh` aborts silently right after `Starting Tailscale (kernel...)` with no error; second run passes | `set -e` + `$(stat -c %u \"$SOCK\" 2>\u002Fdev\u002Fnull \\|\\| stat -f %u \"$SOCK\" 2>\u002Fdev\u002Fnull)` — both `stat` branches fail (exit 1) on missing socket, so the substitution exits 1 and `set -e` kills the script. Second run hits `pgrep` guard, skips Tailscale entirely and masks the bug | Append `\\|\\| true` inside the substitution: `$(stat -c %u \"$SOCK\" 2>\u002Fdev\u002Fnull \\|\\| stat -f %u \"$SOCK\" 2>\u002Fdev\u002Fnull \\|\\| true)` and test emptiness; audit the class (`grep -rn 'stat -c'`), fix wrapper templates and `self-check.sh` too |\n| First boot fails, second boot passes with `already running -- skipping` | Warm-start guard (`pgrep` \u002F `already installed`) skips the failing cold-start path — bug only reproduces on cold start | Repro must force cold state (kill daemon, remove socket\u002Fpid) and run without `set -x` masking; diff the two boot logs to find which step was skipped |\n\n## Anti-patterns\n\n- **Weakening the test to make it pass** — changes the contract, hides the bug.\n- **Silencing output with `>\u002Fdev\u002Fnull` without understanding why** — every\n  silenced probe should have a log capture path (see `ci-log-capture` skill).\n- **Skipping the health-check step** — if the stack isn't fully up, the check\n  that \"works\" is meaningless.\n- **\"It passed locally\"** — always verify in the actual CI environment.\n\n## Real-world example (from this repo)\n\nThe mnemon PATH issue appeared as \"mnemon not found\" in `self-check.sh` even\nthough install claimed success. Root cause was not the install at all: the\nPATH snippet was added only to `~\u002F.bashrc` (not `~\u002F.profile`), and non-\ninteractive login shells (like `dts exec`) never source `.bashrc`. Comparing\npassing (codespace) vs failing (CI) environments revealed the PATH diff.\n\n**Lesson:** when a fix appears to work but the symptom persists, diff the\n*environments*, not just the code.","# CI Debugging — Find the Root Cause\n\nDebug CI failures by comparing passing vs failing commits, never by weakening the test.\n\n## The core rule\n\n**Fix the root cause, never weaken the test.**\n\nWhen a fix seems to work, verify it actually fixed the right thing. If a test\nis flaky or \"fails sometimes\", the failure is telling you something about the\ncode — not about the test.\n\n## The 5-step procedure\n\n1. **Get the failing commit's full log.**\n   ```bash\n   gh run view \u003Crun-id> --log | grep -E \"FAIL|Error|✗\" | head -20\n   # identify the exact step and line\n   ```\n\n2. **Compare passing vs failing commits.**\n   ```bash\n   git log --oneline \u003Cpassing>..\u003Cfailing>\n   git diff \u003Cpassing>..\u003Cfailing> -- \u003Csuspect-file>\n   ```\n   A \"silent failure\" (process killed before writing output) gives you zero\n   error info — the only signal is which commit changed the behavior.\n\n3. **Reproduce locally in isolation** (not by guessing).\n   Use the repo's test harness (`dts` \u002F `scripts\u002Fdts.sh`) to run the exact\n   failing command inside a throwaway container, so the host environment\n   can't mask the bug.\n\n4. **Fix the root cause, then re-run the original test.**\n   If your fix \"works\" but the test had to be weakened, you fixed the wrong\n   thing.\n\n5. **Verify with a clean test run** — not just the one case that failed.\n   Run the full suite; a root-cause fix should not require relaxing other\n   tests.\n\n## Step 0: Ensure logs are captured *before* debugging\n\nIf probes redirect to `\u002Fdev\u002Fnull`, there is nothing to debug with.\nBefore any debugging, verify the test suite captures failure output\nto files that get uploaded as CI artifacts.\n\nSee the `ci-log-capture` skill for the pattern: replace\n`>\u002Fdev\u002Fnull 2>&1` with file captures to `${CI_LOGS_DIR}` so that:\n- On **success**: clean `[PASS]` in CI log, full output sits quietly in artifact\n- On **failure**: `[FAIL]` + the full captured body is `cat`-ed to stderr\n  (visible in CI log immediately) AND persisted to artifact for later analysis\n\n### The `run_and_log` helper pattern\n\n```sh\n# Add after CI_LOGS_DIR setup:\nrun_and_log() {\n    local test_name=\"$1\"; shift\n    local log_file=\"${CI_LOGS_DIR}\u002F${test_name}.log\"\n    if \"$@\" >\"${log_file}\" 2>&1; then\n        log_info \"${test_name} — OK\"\n        return 0\n    else\n        log_error \"${test_name} — FAILED (see ${log_file})\"\n        cat \"${log_file}\"   # print to CI log for immediate visibility\n        return 1\n    fi\n}\n```\n\n### Which probes to capture\n\n| Category | Examples | Why |\n|----------|----------|-----|\n| **Version checks** | `hermes --version`, `9router --version` | Confirms binary installed correctly |\n| **Health\u002Fmodel checks** | `\u002Fv1\u002Fmodels` curl probes | Shows HTTP status, model count |\n| **Chat completion** | Chat completion endpoints | Shows **400\u002F401\u002F403 body** from upstream OC bug |\n| **Preconfig** | 9Router\u002FOmniRoute login\u002Fsetup | Shows which step failed in the chain |\n\nAll captured files must match the artifact upload path (`path: ${{ env.CI_LOGS_DIR }}\u002F`)\nso they're downloaded on failure. See `ci-log-capture` skill.\n\n## Silent failure patterns (hardest to debug)\n\n| Symptom | Likely cause | How to find it |\n|---------|--------------|----------------|\n| Process exits 0 but no output written | Process killed early (OOM, timeout) before flush | Compare exit code + check for killed processes in CI log |\n| Test \"passes\" but artifact is empty | `if-no-files-found: warn` masks empty upload | Check artifact size\u002Fcontent, not just job status |\n| Fail only in CI, never locally | NGINX\u002FCI PATH differs, env var unset, runner image change | Compare `env` output between local and CI |\n| Fail only sometimes (flaky) | Race condition, retry logic, provider rate-limit | Look for \"retry\" loops and timeouts; bump tolerance only if truly ephemeral |\n| `rm -rf` reports success but `verify CLEAN` fails with leftover mount | Docker volume mount (`-v host:container\u002Fpath`) blocks removal — `rm -rf` cannot delete a mountpoint | Check `mount` output for `$HOME` in failure logs; unmount with `umount -l` plus `sudo umount -l` fallback (container runs as uid 1000) before retrying removal |\n| `--keep-config` test fails \"removed X (should preserve)\" | Install never created X because its setup is disabled by default | Ensure install creates empty placeholder dirs (`mkdir -p ~\u002F.mnemon ~\u002F.ollama ~\u002F.9router`) even when disabled so the uninstall contract has something to preserve |\n| Uninstall test deletes real `~\u002F.mnemon`\u002F`~\u002F.hermes` on host\u002Fcodespace | Test ran with real `$HOME` instead of isolated temp dir — codespace mnemon is the memory layer, deleting it kills the session's model proxy | Always run uninstall\u002Fpurge tests with `HOME=$(mktemp -d)` and clean up; never `rm -rf` with real `$HOME` in tests or scripts |\n| CLI env var silently ignored (e.g. `TAILSCALE_SOCKET` ignored by v1.104.1) `fails to connect` on custom socket | `bin --socket` works but `ENV=... bin status` dials default path | Probe with both forms live: `ENV=... bin status` vs `bin --socket=... status`; if only the flag succeeds, fix all probes\u002Fwrappers to use `--socket` and ship a `bin\u002Fts` wrapper that always passes it — never rely on the env var |\n| `--socket=~\u002Fpath` dials literal `~` and fails | `~` only expands at word start or after `:` in `PATH`-like assignments; `--flag=~\u002F...` is not an assignment | Always use `$HOME` or `$MINIONS_HOME` in flags and wrappers (`--socket=\"$HOME\u002F.minions\u002F...\"`), and use `${MINIONS_HOME:-$HOME\u002F.minions}` in wrappers so CI\u002F`ssh`\u002F`sudo` resolve correctly |\n| `var=$(failing_cmd); rc=$?` error handler never runs and log stays empty | `set -e` aborts at a failing command substitution before the next line, so the `[FAIL] + tail` branch is dead code | Wrap the capture: `set +e; var=$(cmd 2>&1); rc=$?; set -e` so the handler can print diagnostics |\n| `mkdir` fails \"Read-only filesystem\" or \"Permission denied\" on a path that is a symlink | Dir symlink `lib\u002Follama -> \u002Ftmp\u002Fcache:ro` makes `mkdir -p lib\u002Follama` walk through the ro mount; or the base image pre-creates `$HOME` owned by root and a conditional `[ -d $home ] or chown` skipped the fix | Mount cache outside the tree (`\u002Ftmp\u002Fcache:ro`) and symlink files not dirs (`lib\u002Follama\u002Follama -> \u002Ftmp\u002Fcache\u002Follama`); always `mkdir -p $home && chown` unconditionally |\n| Cache miss causes re-download then a different downstream failure | Version string carries `v` prefix (`v0.40.0` in deps.yaml) but cache dir is `ollama-0.40.0-...` — un-stripped key leaves `OLLAMA_HOST_CACHE` empty so the container mounts nothing | Normalize before use: `version=${version#v}` and map arch aliases before building `OLLAMA_HOST_CACHE` |\n| `bash install.sh \\| tee log` reports success while install OOMed | Pipe exit is `tee`'s 0 when `pipefail` is off, masking left-side failure — downstream keep-config then sees missing dot dirs | Prefix the pipe: `set -o pipefail; cmd \\| tee log` so failure propagates |\n| `npm install` OOM `JavaScript heap out of memory` in constrained container | `NODE_OPTIONS=--max-old-space-size=512` too small for large package (e.g. `omniroute@3.8.x`) under DTS\u002FCI limits | Raise heap in shared `install_npm_package` wrapper (512 → 2048) |\n| `.tar.zst` extract fails `stdin: not in gzip format` or `zstd required` | DTS container lacks `zstd` and `tar -xzf` handles gzip only — `.tar.zst` needs `zstd -dc \\| tar -xf` or `tar --zstd -xf` | Install `zstd` in DTS prereqs and fix `lib\u002Fdownload.sh` `.tar.zst` branch to use `tar --zstd -xf` fallback; add `\u002Ftmp\u002Follama-cache\u002Follama` host-cache copy fast-path in `lib\u002Follama.sh` to skip re-download on reinstall |\n| `Dependency drift (advisory)` fails but overall CI still green | Drift is `continue-on-error: true` — the job fails to surface `OUTDATED` but does not block merge; easy to ignore until 4-file sync invariant breaks `version-sync` | Run `check-updates.sh --json` to identify the outdated pin, then `bump.sh DEP=VERSION --open-pr` + `sync-readme-badges.sh` to fix all 4 files (see `version-single-source` skill) |\n| `self-check.sh --ci` exit 1 treated as failure after reinstall | `--ci` exits 1 for warnings (1 warning, 0 critical) and 2 for critical — the pre-reinstall health check tolerates warnings; a new strict `or fail` after reinstall breaks on advisory warnings | Fail only on critical: `set +e; bash self-check.sh --ci; SC=$?; set -e; [ \"$SC\" -ge 2 ] && fail` — tolerate `SC=1` |\n| `DTS Integration` cancelled at `timeout-minutes: 60` mid `stop.sh` (all 8 tests PASS) | Single DTS run ~38min; 3-retry loop needs ~115min headroom worst case — wall timeout kills mid-test and reports `cancelled`, not a code defect | Bump `dts-integration.timeout-minutes` to 75 (or 90 for 3 retries); verify via `gh api ...\u002Fruns\u002F\u003Cid>\u002Fjobs` duration |\n| `Real Install` `npm 404 pi-ai-1.1.0.tgz` while PI pinned `1.0.4` | `pi-coding-agent@1.0.4` deps `^1.0.4` float to `pi-ai@1.1.0`; registry propagation can 404 transiently | Bump PI via `bump.sh PI=1.1.0` (+ badges) so install pins `1.1.0` directly; avoid fixing via rerun alone |\n| `verify-knowledge.sh` F5 reports `Unknown function: F5` after printing `WARN: .devcontainer references found` | `echo \"$hits\" \\| head -20` always SIGPIPEs (exit 141) when hits exceed 20 lines; with `set -euo pipefail` the pipeline fails, and `type -t \"$fn\" && \"$fn\" \\|\\| { echo Unknown; exit 1; }` misattributes any function failure as a missing function | Append `\\|\\| true` after `head -20` and split dispatch into `if ! type -t \"$fn\"; then echo Unknown; exit 1; fi; \"$fn\"` so real failures surface with their own output |\n| `boot.sh` aborts silently right after `Starting Tailscale (kernel...)` with no error; second run passes | `set -e` + `$(stat -c %u \"$SOCK\" 2>\u002Fdev\u002Fnull \\|\\| stat -f %u \"$SOCK\" 2>\u002Fdev\u002Fnull)` — both `stat` branches fail (exit 1) on missing socket, so the substitution exits 1 and `set -e` kills the script. Second run hits `pgrep` guard, skips Tailscale entirely and masks the bug | Append `\\|\\| true` inside the substitution: `$(stat -c %u \"$SOCK\" 2>\u002Fdev\u002Fnull \\|\\| stat -f %u \"$SOCK\" 2>\u002Fdev\u002Fnull \\|\\| true)` and test emptiness; audit the class (`grep -rn 'stat -c'`), fix wrapper templates and `self-check.sh` too |\n| First boot fails, second boot passes with `already running -- skipping` | Warm-start guard (`pgrep` \u002F `already installed`) skips the failing cold-start path — bug only reproduces on cold start | Repro must force cold state (kill daemon, remove socket\u002Fpid) and run without `set -x` masking; diff the two boot logs to find which step was skipped |\n\n## Anti-patterns\n\n- **Weakening the test to make it pass** — changes the contract, hides the bug.\n- **Silencing output with `>\u002Fdev\u002Fnull` without understanding why** — every\n  silenced probe should have a log capture path (see `ci-log-capture` skill).\n- **Skipping the health-check step** — if the stack isn't fully up, the check\n  that \"works\" is meaningless.\n- **\"It passed locally\"** — always verify in the actual CI environment.\n\n## Real-world example (from this repo)\n\nThe mnemon PATH issue appeared as \"mnemon not found\" in `self-check.sh` even\nthough install claimed success. Root cause was not the install at all: the\nPATH snippet was added only to `~\u002F.bashrc` (not `~\u002F.profile`), and non-\ninteractive login shells (like `dts exec`) never source `.bashrc`. Comparing\npassing (codespace) vs failing (CI) environments revealed the PATH diff.\n\n**Lesson:** when a fix appears to work but the symptom persists, diff the\n*environments*, not just the code.",{"name":5,"author":23,"license":14,"version":24,"metadata":25,"platforms":37,"description":7},"Hermes Agent","1.0.0",{"hermes":26},{"tags":27,"related_skills":33},[28,29,30,31,32],"ci","debugging","root-cause","tests","github-actions",[34,35,36],"github-codespace","karpathy-coding-guidelines","tdd",[38],"linux","main","e95aec76487a22bbbc23b63234724bc7178d4692","skills\u002Fci-debugging","https:\u002F\u002Fgithub.com\u002Fgitricko\u002F.minions","https:\u002F\u002Fgithub.com\u002Fgitricko","2026-11-13 16:00:00",{"source":8,"skillName":5,"commands":46,"prompt":56,"zipAvailable":57},[47,50,53],{"tool":48,"command":49},"npx","npx skills add gitricko\u002F.minions --skill ci-debugging",{"tool":51,"command":52},"bunx","bunx skills add gitricko\u002F.minions --skill ci-debugging",{"tool":54,"command":55},"pnpm","pnpm dlx skills add gitricko\u002F.minions --skill ci-debugging","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fgitricko\u002F.minions\u002Fblob\u002Fmain\u002Fskills\u002Fci-debugging\u002FSKILL.md",true,[],[60,67,73,80,87,94],{"skillNo":61,"name":62,"nameZh":6,"description":63,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":64,"categories":65,"keywords":66,"publishedAt":18},"01M7B0VJ3Q0YMCTJY69FEJ4100","version-single-source","Use when adding a new component dependency or versioning a binary in install\u002Fboot scripts.","https:\u002F\u002Fgithub.com\u002Fgitricko\u002F.minions\u002Fblob\u002Fmain\u002Fskills\u002Fversion-single-source\u002FSKILL.md",[],[],{"skillNo":68,"name":36,"nameZh":6,"description":69,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":70,"categories":71,"keywords":72,"publishedAt":18},"01M7B0V888DV8J3JY5W9Z6WCF3","Test-driven development. Use when the user wants to build features or fix bugs test-first, mentions \"red-green-refactor\", or wants integration tests.","https:\u002F\u002Fgithub.com\u002Fgitricko\u002F.minions\u002Fblob\u002Fmain\u002Fskills\u002Ftdd\u002FSKILL.md",[],[],{"skillNo":74,"name":75,"nameZh":6,"description":76,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":77,"categories":78,"keywords":79,"publishedAt":18},"01M7B0TVFM38JH3PBA2K219Z2X","shell-path-login","Use when a binary or script works in one shell context but not another (interactive vs non-interactive login vs non-login shell).","https:\u002F\u002Fgithub.com\u002Fgitricko\u002F.minions\u002Fblob\u002Fmain\u002Fskills\u002Fshell-path-login\u002FSKILL.md",[],[],{"skillNo":81,"name":82,"nameZh":6,"description":83,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":84,"categories":85,"keywords":86,"publishedAt":18},"01M7B0TKFCTVQBTSNF9WF1MXJ6","self-update-fix","Fix a failing dependency update PR. Use when a self-update PR's CI fails, or when invoked by scripts\u002Fself-update-fix.sh.","https:\u002F\u002Fgithub.com\u002Fgitricko\u002F.minions\u002Fblob\u002Fmain\u002Fskills\u002Fself-update-fix\u002FSKILL.md",[],[],{"skillNo":88,"name":89,"nameZh":6,"description":90,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":91,"categories":92,"keywords":93,"publishedAt":18},"01M7B0TB57WEW0E6E9874K3C72","pr-change-log","Use when a multi-commit PR needs its description written\u002Fupdated, or when closing out a long branch with many fix commits.","https:\u002F\u002Fgithub.com\u002Fgitricko\u002F.minions\u002Fblob\u002Fmain\u002Fskills\u002Fpr-change-log\u002FSKILL.md",[],[],{"skillNo":95,"name":96,"nameZh":6,"description":97,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":98,"categories":99,"keywords":100,"publishedAt":18},"01M7B0T2FGSV0ZFRZ46PG4BBY2","persistent-knowledge","Persistent skills\u002Fknowledge in Codespace via symlinks.","https:\u002F\u002Fgithub.com\u002Fgitricko\u002F.minions\u002Fblob\u002Fmain\u002Fskills\u002Fpersistent-knowledge\u002FSKILL.md",[],[]]