[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6CASH7VWTH0GDZ74AB5PG3Z":3,"skill-same:01M6CASH7VWTH0GDZ74AB5PG3Z":79},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":22,"publishedAt":37,"linkOnly":38,"markdownContent":39,"markdownBody":40,"frontMatter":41,"refName":42,"commitSha":43,"skillDirectory":44,"forksCount":45,"repositoryHtmlUrl":46,"repositoryHomepageUrl":6,"ownerHtmlUrl":47,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":48,"install":49,"resources":62},"01M6CASH7VWTH0GDZ74AB5PG3Z","spend-alert-bridge",null,"Diagnose why a Spend Alert (GitHub Actions MTD\u002F7-day breach, MCP Agent Mail relay,","jleechanorg\u002Fjleechan-skills","jleechanorg","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F221404966?v=4","01M6C1VJ99XG950X1DWPEBCBGX","01M6C1VJDK5YVDNT3362RT5WSB",6,"MIT","https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002Fhermes\u002Fskills\u002Fspend-alert-bridge\u002FSKILL.md",[17],{"categoryNo":18,"categoryName":19,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",[23,24,25,26,27,28,29,30,31,32,33,34,35,36],"bridge","alert","spend alert bridge","MCP","github","diagnose","actions","github actions","why","mcp agent mail","mail","relay","agent mail","breach","2026-11-10 10:12:30",false,"---\nname: spend-alert-bridge\ndescription: Diagnose why a Spend Alert (GitHub Actions MTD\u002F7-day breach, MCP Agent Mail relay, #worldai-alerts C0BCVG4F560) was missed. Use when user asks \"why didn't you reply to the spend alert?\", the alert didn't reach Slack, or MCP Agent Mail surfaces a spend alert in session context without a corresponding reply.\n---\n\n# Spend Alert Reply Bridge\n\n## Failure pattern\n\nThe `spend-alert-daily.sh` launchd script posts a Slack alert to `#worldai-alerts` (C0BCVG4F560) when GitHub Actions daily Δ or 7-day rolling exceeds its threshold. The user sees this alert re-routed to their session context via **MCP Agent Mail** (a separate relay process, `com.mcp.agent.mail`, PID ~1151), but **nothing in the dropped-thread-watcher pipeline watches Agent Mail**. Combined result: the alert appears in human-facing chat but there is no hook that pings me to reply in-thread.\n\n**Three distinct failure modes, all recurring:**\n\n### Mode A — bridge delivery is fine, the SPEND ITSELF is the runaway (most common, 2026-07-23+)\n\nThe script posts correctly, the alert lands in #worldai-alerts, MCP Agent Mail relays it. But the user still gets a \"why is spend $X\" message because the alert numbers are real and trace to a runaway workflow. The bridge isn't broken — the cost is real.\n\n**Important nuance:** the script's \"MTD delta\" is NOT a true daily-spend metric. GH's billing API returns monthly buckets only (`date == \"2026-07-01T00:00:00Z\"` for all of July). The script subtracts two snapshots taken 24h apart to estimate daily burn — this conflates real-time burn with billing-API lag. When the API finalizes a batch overnight, the next morning's delta will spike without any real new spend.\n\n### Mode A.1 — comment-router-style runaway (workflow-level noise, low minutes)\n\nThe first instinct: look at which workflow fired thousands of times. The 2026-07-23\nincident was initially diagnosed as a `comment-router.yml` runaway (5,566 issue_comment\nruns\u002F32h, all `conclusion=skipped`). The comment-router's `if:` correctly skipped\neach one, but the workflow_run event still registered.\n\nThis is real but **usually a SYMPTOM, not the cause.** The actual 5,566 comments\nwere being posted by a **different process** — see Mode A.2.\n\n### Mode A.2 — upstream spam loop posting to issue_comment (most common runaway source, 2026-07-23+)\n\n**The recurring shape that has actually been driving the spend spikes:**\n\n- An **upstream automation** (typically a local daemon posting with the operator's PAT) posts the same comment to 1-3 PRs every ~1h as part of a feedback loop\n- Each `issue_comment` event fires `comment-router.yml` once, which correctly skips at the job level\n- The workflow_run count in the GH Actions `runs` API balloons to thousands, polluting metrics and (in the daemon's case) triggering more downstream side effects\n- The spend delta is mostly billing-API lag from finalized batches, not real new minutes\n\n**2026-07-23 incident (bead $USER-rouf):**\n- 3 PRs getting hammered: `your-project.com#8428` (1,791 posts), `#8420` (1,773), `#8421` (1,770)\n- Total: 5,334 same-prefix daemon comments in 32h on these 3 PRs alone (vs ~6,000\u002F48h for ALL of `comment-router` traffic)\n- Root cause: the daemon posted the comment BEFORE its dedup check ran, AND the dedup key used a per-tick id (a fresh bead id every tick) instead of the stable branch\n- Fix: re-key dedup on the branch name AND move the dedup check before the comment write\n- Full detail: `references\u002Fcomment-router-runaway-2026-07-23.md`\n\n**Diagnostic recipe when `comment-router` is the noise source:**\n\n```bash\n# 1. Pull the actual comment bodies (not the workflow runs — the comments are upstream)\nREPO=\"$GITHUB_REPOSITORY\"\n# NOTE: the `since` query param is ignored for `repos\u002F\u003Crepo>\u002Fissues\u002Fcomments` —\n# paginate with `sort=created&direction=desc` and filter the timestamp in-process.\ngh api \"repos\u002F$REPO\u002Fissues\u002Fcomments?per_page=100&direction=desc&sort=updated\" \\\n  | jq -r '.[] | \"\\(.user.login)\\t\\(.author_association)\\t\\((.body \u002F\u002F \"\")[0:80])\"'\n\n# 2. Group by author_association + author login + body prefix\n# 99% of \"noisy\" comments will share a body prefix from one automation\ngh api \"repos\u002F$REPO\u002Fissues\u002Fcomments?per_page=100&direction=desc&sort=updated\" \\\n  | jq 'group_by(.author_association) | map({aa: .[0].author_association, n: length})'\n\n# 3. Find the source — likely a local daemon, a GitHub Actions bot, or a cron\n```\n\n**Once the source automation is identified, fix it at its own repo — BUT FIRST verify any claimed fix actually landed:**\n\n```bash\ncd \u003Csource-repo>\nFIX_SHA=\u003Cfix-commit>\ngit merge-base --is-ancestor \"$FIX_SHA\" origin\u002Fmain && echo \"merged\" || echo \"NOT MERGED\"\n# If NOT merged, the daemon is still running buggy code: merge the fix, rebuild,\n# and restart its launchd job. `launchctl bootout` is a stopgap, not a fix.\n# ALWAYS verify the fix is on main BEFORE telling the user \"fix is shipped\".\n```\n\n**Why this matters:** fixing `comment-router.yml` (the downstream consumer) does NOT\nstop the spend alert. The fix lives at the source — the daemon that posts the comments\nin the first place. The comment-router's job-level `if:` is already correct; it just\ngets hit thousands of times because the upstream daemon is in a loop.\n\n### Mode B — `set -u` crash in `spend-alert-daily.sh` (2026-07-16, RESOLVED)\n\n`log_warn \"$1\"` aborts on unbound variable when called with no arg during `slack_post`'s stderr pipe under `set -euo pipefail`. The script terminates before `slack_post` ever fires. Fix landed — `log_warn \"${1:-}\"` (and same for `log`, `log_error`) per PR in `jleechanorg\u002Fjleechanclaw`. If you see this in `tail -40` of the log, the fix has regressed.\n\n## Three independent root causes (one per mode)\n\n| # | Mode | Cause | Fix |\n|---|------|-------|-----|\n| 1 | A.1 | `comment-router.yml` (PR #8354–#8355) was supposed to consolidate 4 issue_comment workflows, but its `on: issue_comment` trigger fires on EVERY comment, and the `if:` guard skips 99%+ of them. The fix only consolidated workflows; it didn't reduce trigger volume. | Add `if: github.event.comment.author_association == 'OWNER'` to the `on: issue_comment` trigger; cap `concurrency` (N=5). **BUT** verify this isn't a downstream symptom of an upstream spam loop first (see A.2). |\n| 2 | A.2 | **Upstream daemon loop** posts the same comment to 1-3 PRs every ~1h. Its dedup key was a per-tick id (fresh bead every tick → never matches) AND the comment posted BEFORE the dedup check ran. | (a) Move the dedup check BEFORE the comment write AND re-key dedup on the branch name (stable across ticks). (b) **VERIFY the fix commit is on main HEAD before claiming the fix is shipped** — `git merge-base --is-ancestor $FIX_SHA HEAD`. |\n| 3 | B | `set -u` in `spend-alert-daily.sh` aborts on `log_warn \"$1\"` when called with no arg | `log_warn \"${1:-}\"` everywhere, OR remove `-u` from the script's `set` line. |\n\nThe bot identity **is** a member of `C0BCVG4F560` (verified 2026-07-23 via `conversations.info` returning `is_member=true` for the bot token fetched from `~\u002F.bashrc HERMES_SLACK_BOT_TOKEN`). The standalone bot token from `HERMES_SLACK_BOT_TOKEN` posts reliably; the scoped MCP identity in this session may report `is_member=false` because it's a different bot identity — don't conflate the two.\n\n## Recovery action when user reports a missed Spend Alert\n\n1. `cat $HOME\u002F.hermes\u002Flogs\u002Fspend-alert-daily.log | tail -40` — confirm whether the last run completed the Slack post or crashed on the `set -u` line. **If it completed, the alert is in the channel — you're in Mode A (real cost spike), not Mode B (bridge broken).**\n2. `jq . $HOME\u002F.hermes\u002Fstate\u002Fspend-alert-state.json` — check the rolling deltas match what the user reported. If `gh_delta` jumped 3x+ over baseline (e.g. $61 vs typical $15), the runway is the cause, not a billing error.\n3. `curl -fsS -H \"Authorization: Bearer $(bash -c 'source ~\u002F.bashrc; echo $HERMES_SLACK_BOT_TOKEN')\" \"https:\u002F\u002Fslack.com\u002Fapi\u002Fconversations.history?channel=C0BCVG4F560&limit=5\"` — confirm whether the alert landed in #worldai-alerts. **This is the right verification** — `mcp__slack__conversations_replies` may falsely report `not_in_channel` for the scoped identity.\n4. **If alert NOT posted (crash \u002F silent failure):** post a manual ack to #worldai-alerts using the bot token (which IS a member) — reply with the same breach numbers so the channel has the missing record.\n5. **If alert IS posted (Mode A — the common case):** reply in-thread with the diagnosis. The user wants to know WHY the spend jumped, not whether the bridge worked. Skip directly to the runaway audit below.\n6. **Runaway audit (Mode A only):** identify the repo and workflow driving the billable minutes. The pattern in 2026-07-23 was a `comment-router.yml` symptom + an upstream daemon cause — see step 7.\n7. **Upstream spam check (Mode A.2 — the most common real cause as of 2026-07-23):** before concluding the comment-router is the runaway, check who is actually posting the comments. If 99% of the `issue_comment` events have one of a small set of body prefixes (e.g. one daemon's escalation prefix), the comment-router is the victim, not the cause. The fix lives in the source that posts the comments. See Mode A.2 in the failure pattern above for the recipe.\n8. **Verify the fix is actually live before claiming resolution.** If the alleged fix is a commit on a branch, run `git -C \u003Crepo> merge-base --is-ancestor \u003Cfix-sha> HEAD`. If the result is \"NOT MERGED\", the daemon is still running buggy code. Either merge the branch or stop claiming the fix is shipped. (On 2026-07-24 a fix left unmerged on a branch produced a second spend spike identical to the first.)\n\n## Verification commands\n\n```bash\n# Step 1: script log + state\ntail -40 $HOME\u002F.hermes\u002Flogs\u002Fspend-alert-daily.log\njq . $HOME\u002F.hermes\u002Fstate\u002Fspend-alert-state.json\n\n# Step 2: channel membership & recent posts\nTOK=$(bash -c 'source ~\u002F.bashrc; echo $HERMES_SLACK_BOT_TOKEN')\ncurl -fsS -H \"Authorization: Bearer $TOK\" \\\n     \"https:\u002F\u002Fslack.com\u002Fapi\u002Fconversations.list?types=public_channel,private_channel&limit=200\" \\\n  | jq '.channels[] | select(.id==\"C0BCVG4F560\")'\n\n# Step 3: identify the cost-driver repo\ngh api \"orgs\u002Fjleechanorg\u002Fsettings\u002Fbilling\u002Fusage\" --paginate \\\n  | jq '[.usageItems[]? | select(.product == \"actions\") | select(.date == \"2026-07-01T00:00:00Z\")] | sort_by(-.netAmount)'\n\n# Step 4: identify the runaway workflow in that repo (last 32h)\n# NOTE: `created=>` query param is treated as equality, not greater-than.\n# Use `sort=created&direction=desc` and walk pages manually until the\n# oldest run's `created_at` \u003C SINCE.\nREPO=\"jleechanorg\u002F\u003Crepo>\"\npython3 .\u002Fscripts\u002Fgh-cost-audit.py   # see scripts\u002F dir\n```\n\n## Prevention — durable fix shape\n\n- **Mode B (script crash):** `scripts\u002Fspend-alert-daily.sh`: change `log_warn \"$1\"` to `log_warn \"${1:-}\"` (and same for `log`, `log_error`), OR remove `-u` from the script's `set` line. Pair with `scripts\u002Ftests\u002Ftest_spend_alert_idempotent.py` (or similar) so the crash mode is regression-tested. **Status 2026-07-23: fix is in place; no crash observed today.**\n- **Mode A (runaway workflow):** the recurring shape is `comment-router.yml` patterns — workflows that listen to high-volume `issue_comment` events and use a job-level `if:` guard. The job guard short-circuits before billable minutes, but the workflow still registers an event. Mitigations:\n  - Add `if:` at the **workflow\u002Fjob level** that filters by `author_association`, comment body prefix, and `issue.pull_request` BEFORE the runner is allocated\n  - Set `concurrency` group to a tight key (`comment-router-${{ github.event.comment.id }}` is already present in this file — verify it)\n  - Audit who posts comments at 174\u002Fhr — likely a bot loop or external webhook source\n  - For the long term, consider an org-wide CODEOWNERS + workflow-level `if: contains(fromJSON('[\"OWNER\",\"MEMBER\",\"COLLABORATOR\"]'), github.event.comment.author_association)` guard\n- **Mode A.2 (upstream daemon runaway):** the recurring shape is a dedup key that uses a fresh-per-tick value (e.g. `bead_id`) instead of a stable identifier (e.g. `head_ref_name`). Mitigations:\n  - All dedup keys must be drawn from STABLE identifiers (branch name, PR number, commit SHA) — never from objects that get re-created every tick\n  - The dedup check must run BEFORE the SCM write (`comment_external`, `gh api repos\u002F...\u002Fissues\u002F...\u002Fcomments`) — otherwise the write is never suppressed even if the dedup would have matched\n  - Add a `cfg.escalation_refire_secs` cooldown (default 3600s) so re-emission is bounded even if the dedup logic has a bug\n  - **Every fix commit must be verified as on the deployment branch before declaring \"fix shipped\"** — `git merge-base --is-ancestor \u003Cfix-sha> \u003Cdeployment-branch>`. A commit on a fix branch is not a fix until that branch is merged into main AND the daemon is rebuilt AND the running launchd job is restarted.\n\n## Pitfalls\n\n- **The bridge is usually NOT broken** when a Spend Alert appears in session context. MCP Agent Mail relays everything from #worldai-alerts into the session — that's working as designed. Spend the first 30 seconds confirming the Slack post landed, then pivot to the cost driver. Don't waste a turn fixing a non-broken bridge.\n- **\"Fix is in PR #N\" ≠ \"fix is shipped\".** The skill recipes (and older incidents) often cite a PR number or branch name as if the fix is live. **ALWAYS verify with `git merge-base --is-ancestor \u003Cfix-sha> HEAD`** before claiming the fix is merged. The 2026-07-24 incident was caused by trusting a previous skill's stated fix location without re-verifying.\n- **The `created=>` query param on `gh api repos\u002F\u003Crepo>\u002Factions\u002Fruns` is treated as EQUALITY, not greater-than.** It returns 0 hits even when plenty of runs exist after the date. Use `sort=created&direction=desc` + manual pagination instead.\n- **`\u002Factions\u002Fruns\u002F{id}\u002Ftiming` returns `billable: {}` for completed runs.** The billable field is reserved for in-progress \u002F queued runs. To approximate cost, use `updated_at - created_at` to compute wall-clock minutes and multiply by the runner type's $\u002Fmin rate. The `gh-actions-cost-monitor.sh` script does this with `COST_PER_MINUTE=0.002` (Linux self-hosted rate).\n- **`gh-api` `repos\u002F\u003Crepo>\u002Fissues\u002Fcomments` ignores the `since` query parameter.** The `since` field on the comment-schema response is also undefined. To filter comments to a time window, paginate (sort=created + direction=desc) and filter by `created_at` in-process.\n- **The `runs` API sees `run_duration_ms` as 0\u002Fnull for skipped runs** even if the workflow allocated a runner. Use `runs.total_count` to gauge trigger volume, not minutes.\n- **Don't trust `run_duration_ms` field** — it returns `null` or `0` for `skipped` runs, even if the workflow was allocated a runner. Use `updated_at - created_at` to get the real wall-clock time per run.\n- Don't assume `is_member=False` from `mcp__slack__conversations_replies` is global to the bot — it's per-identity. The `HERMES_SLACK_BOT_TOKEN` env may resolve to a different bot identity than the one bound to the live Slack MCP server. Use `conversations.history` with the bot token to verify, not `conversations_replies` from the scoped MCP.\n- **Don't trust a `bead_id` as a dedup key.** Per-tick fresh objects (new bead IDs, new session IDs) make the ledger never match. Use stable identifiers (branch name, PR number, commit SHA) for dedup keys.\n- Don't post the user's reply via `mcp__slack__conversations_add_message` from this session context — it's the wrong identity. Use `chat.postMessage` with the bot token from `~\u002F.bashrc`.\n- **GH billing API returns monthly buckets only** — `date == \"2026-07-01T00:00:00Z\"` is the entire month, not a day. Don't try to slice it by day; you'll see empty results. The script's `gh_delta` is a 24h-difference between two MTD snapshots, NOT a true daily-spend figure.\n- The Agent Mail relay is **in addition to**, not a replacement for, the Slack post. Don't \"fix\" by silencing the cron — fix the crash and keep both channels.\n- **A \"skipped\" workflow run still counts.** `conclusion=skipped` means 0 billable minutes, but the workflow_run event is registered. If a workflow fires 5,000+ times in 32h, the `runs` API is polluted even if no minutes are billed. Use the runs API for diagnosing trigger volume, not the billing API.\n- **Comment-router noise is usually a SYMPTOM, not the cause.** When `comment-router.yml` fires 5,000+ times\u002F32h on `issue_comment`, the first instinct is to harden the workflow (add author_association guard, lower concurrency). That fixes the symptom but misses the cause: an upstream automation (typically a PAT-driven daemon) is in a feedback loop posting the same comment to 1-3 PRs every ~1h. Pull the actual comment bodies via `repos\u002F\u003Crepo>\u002Fissues\u002Fcomments?per_page=100&direction=desc&sort=updated` and group by `user.login` + `body` prefix BEFORE proposing a workflow fix. If 99% of comments come from one automation with one body prefix, the fix lives at that automation, not at the comment-router. See Mode A.2 for the fix.\n\n## Companion skills \u002F files\n\n- `references\u002Fcomment-router-runaway-2026-07-23.md` — original 2026-07-23 incident detail\n- `scripts\u002Fgh-cost-audit.py` — runs\u002Fworkflow volume audit helper (32h window, per-workflow breakdown with skip counts)\n- `scripts\u002Fcomment-audit.py` — issue_comments audit helper (per-author, per-prefix, per-hour breakdown for time-windowed analysis)\n","# Spend Alert Reply Bridge\n\n## Failure pattern\n\nThe `spend-alert-daily.sh` launchd script posts a Slack alert to `#worldai-alerts` (C0BCVG4F560) when GitHub Actions daily Δ or 7-day rolling exceeds its threshold. The user sees this alert re-routed to their session context via **MCP Agent Mail** (a separate relay process, `com.mcp.agent.mail`, PID ~1151), but **nothing in the dropped-thread-watcher pipeline watches Agent Mail**. Combined result: the alert appears in human-facing chat but there is no hook that pings me to reply in-thread.\n\n**Three distinct failure modes, all recurring:**\n\n### Mode A — bridge delivery is fine, the SPEND ITSELF is the runaway (most common, 2026-07-23+)\n\nThe script posts correctly, the alert lands in #worldai-alerts, MCP Agent Mail relays it. But the user still gets a \"why is spend $X\" message because the alert numbers are real and trace to a runaway workflow. The bridge isn't broken — the cost is real.\n\n**Important nuance:** the script's \"MTD delta\" is NOT a true daily-spend metric. GH's billing API returns monthly buckets only (`date == \"2026-07-01T00:00:00Z\"` for all of July). The script subtracts two snapshots taken 24h apart to estimate daily burn — this conflates real-time burn with billing-API lag. When the API finalizes a batch overnight, the next morning's delta will spike without any real new spend.\n\n### Mode A.1 — comment-router-style runaway (workflow-level noise, low minutes)\n\nThe first instinct: look at which workflow fired thousands of times. The 2026-07-23\nincident was initially diagnosed as a `comment-router.yml` runaway (5,566 issue_comment\nruns\u002F32h, all `conclusion=skipped`). The comment-router's `if:` correctly skipped\neach one, but the workflow_run event still registered.\n\nThis is real but **usually a SYMPTOM, not the cause.** The actual 5,566 comments\nwere being posted by a **different process** — see Mode A.2.\n\n### Mode A.2 — upstream spam loop posting to issue_comment (most common runaway source, 2026-07-23+)\n\n**The recurring shape that has actually been driving the spend spikes:**\n\n- An **upstream automation** (typically a local daemon posting with the operator's PAT) posts the same comment to 1-3 PRs every ~1h as part of a feedback loop\n- Each `issue_comment` event fires `comment-router.yml` once, which correctly skips at the job level\n- The workflow_run count in the GH Actions `runs` API balloons to thousands, polluting metrics and (in the daemon's case) triggering more downstream side effects\n- The spend delta is mostly billing-API lag from finalized batches, not real new minutes\n\n**2026-07-23 incident (bead $USER-rouf):**\n- 3 PRs getting hammered: `your-project.com#8428` (1,791 posts), `#8420` (1,773), `#8421` (1,770)\n- Total: 5,334 same-prefix daemon comments in 32h on these 3 PRs alone (vs ~6,000\u002F48h for ALL of `comment-router` traffic)\n- Root cause: the daemon posted the comment BEFORE its dedup check ran, AND the dedup key used a per-tick id (a fresh bead id every tick) instead of the stable branch\n- Fix: re-key dedup on the branch name AND move the dedup check before the comment write\n- Full detail: `references\u002Fcomment-router-runaway-2026-07-23.md`\n\n**Diagnostic recipe when `comment-router` is the noise source:**\n\n```bash\n# 1. Pull the actual comment bodies (not the workflow runs — the comments are upstream)\nREPO=\"$GITHUB_REPOSITORY\"\n# NOTE: the `since` query param is ignored for `repos\u002F\u003Crepo>\u002Fissues\u002Fcomments` —\n# paginate with `sort=created&direction=desc` and filter the timestamp in-process.\ngh api \"repos\u002F$REPO\u002Fissues\u002Fcomments?per_page=100&direction=desc&sort=updated\" \\\n  | jq -r '.[] | \"\\(.user.login)\\t\\(.author_association)\\t\\((.body \u002F\u002F \"\")[0:80])\"'\n\n# 2. Group by author_association + author login + body prefix\n# 99% of \"noisy\" comments will share a body prefix from one automation\ngh api \"repos\u002F$REPO\u002Fissues\u002Fcomments?per_page=100&direction=desc&sort=updated\" \\\n  | jq 'group_by(.author_association) | map({aa: .[0].author_association, n: length})'\n\n# 3. Find the source — likely a local daemon, a GitHub Actions bot, or a cron\n```\n\n**Once the source automation is identified, fix it at its own repo — BUT FIRST verify any claimed fix actually landed:**\n\n```bash\ncd \u003Csource-repo>\nFIX_SHA=\u003Cfix-commit>\ngit merge-base --is-ancestor \"$FIX_SHA\" origin\u002Fmain && echo \"merged\" || echo \"NOT MERGED\"\n# If NOT merged, the daemon is still running buggy code: merge the fix, rebuild,\n# and restart its launchd job. `launchctl bootout` is a stopgap, not a fix.\n# ALWAYS verify the fix is on main BEFORE telling the user \"fix is shipped\".\n```\n\n**Why this matters:** fixing `comment-router.yml` (the downstream consumer) does NOT\nstop the spend alert. The fix lives at the source — the daemon that posts the comments\nin the first place. The comment-router's job-level `if:` is already correct; it just\ngets hit thousands of times because the upstream daemon is in a loop.\n\n### Mode B — `set -u` crash in `spend-alert-daily.sh` (2026-07-16, RESOLVED)\n\n`log_warn \"$1\"` aborts on unbound variable when called with no arg during `slack_post`'s stderr pipe under `set -euo pipefail`. The script terminates before `slack_post` ever fires. Fix landed — `log_warn \"${1:-}\"` (and same for `log`, `log_error`) per PR in `jleechanorg\u002Fjleechanclaw`. If you see this in `tail -40` of the log, the fix has regressed.\n\n## Three independent root causes (one per mode)\n\n| # | Mode | Cause | Fix |\n|---|------|-------|-----|\n| 1 | A.1 | `comment-router.yml` (PR #8354–#8355) was supposed to consolidate 4 issue_comment workflows, but its `on: issue_comment` trigger fires on EVERY comment, and the `if:` guard skips 99%+ of them. The fix only consolidated workflows; it didn't reduce trigger volume. | Add `if: github.event.comment.author_association == 'OWNER'` to the `on: issue_comment` trigger; cap `concurrency` (N=5). **BUT** verify this isn't a downstream symptom of an upstream spam loop first (see A.2). |\n| 2 | A.2 | **Upstream daemon loop** posts the same comment to 1-3 PRs every ~1h. Its dedup key was a per-tick id (fresh bead every tick → never matches) AND the comment posted BEFORE the dedup check ran. | (a) Move the dedup check BEFORE the comment write AND re-key dedup on the branch name (stable across ticks). (b) **VERIFY the fix commit is on main HEAD before claiming the fix is shipped** — `git merge-base --is-ancestor $FIX_SHA HEAD`. |\n| 3 | B | `set -u` in `spend-alert-daily.sh` aborts on `log_warn \"$1\"` when called with no arg | `log_warn \"${1:-}\"` everywhere, OR remove `-u` from the script's `set` line. |\n\nThe bot identity **is** a member of `C0BCVG4F560` (verified 2026-07-23 via `conversations.info` returning `is_member=true` for the bot token fetched from `~\u002F.bashrc HERMES_SLACK_BOT_TOKEN`). The standalone bot token from `HERMES_SLACK_BOT_TOKEN` posts reliably; the scoped MCP identity in this session may report `is_member=false` because it's a different bot identity — don't conflate the two.\n\n## Recovery action when user reports a missed Spend Alert\n\n1. `cat $HOME\u002F.hermes\u002Flogs\u002Fspend-alert-daily.log | tail -40` — confirm whether the last run completed the Slack post or crashed on the `set -u` line. **If it completed, the alert is in the channel — you're in Mode A (real cost spike), not Mode B (bridge broken).**\n2. `jq . $HOME\u002F.hermes\u002Fstate\u002Fspend-alert-state.json` — check the rolling deltas match what the user reported. If `gh_delta` jumped 3x+ over baseline (e.g. $61 vs typical $15), the runway is the cause, not a billing error.\n3. `curl -fsS -H \"Authorization: Bearer $(bash -c 'source ~\u002F.bashrc; echo $HERMES_SLACK_BOT_TOKEN')\" \"https:\u002F\u002Fslack.com\u002Fapi\u002Fconversations.history?channel=C0BCVG4F560&limit=5\"` — confirm whether the alert landed in #worldai-alerts. **This is the right verification** — `mcp__slack__conversations_replies` may falsely report `not_in_channel` for the scoped identity.\n4. **If alert NOT posted (crash \u002F silent failure):** post a manual ack to #worldai-alerts using the bot token (which IS a member) — reply with the same breach numbers so the channel has the missing record.\n5. **If alert IS posted (Mode A — the common case):** reply in-thread with the diagnosis. The user wants to know WHY the spend jumped, not whether the bridge worked. Skip directly to the runaway audit below.\n6. **Runaway audit (Mode A only):** identify the repo and workflow driving the billable minutes. The pattern in 2026-07-23 was a `comment-router.yml` symptom + an upstream daemon cause — see step 7.\n7. **Upstream spam check (Mode A.2 — the most common real cause as of 2026-07-23):** before concluding the comment-router is the runaway, check who is actually posting the comments. If 99% of the `issue_comment` events have one of a small set of body prefixes (e.g. one daemon's escalation prefix), the comment-router is the victim, not the cause. The fix lives in the source that posts the comments. See Mode A.2 in the failure pattern above for the recipe.\n8. **Verify the fix is actually live before claiming resolution.** If the alleged fix is a commit on a branch, run `git -C \u003Crepo> merge-base --is-ancestor \u003Cfix-sha> HEAD`. If the result is \"NOT MERGED\", the daemon is still running buggy code. Either merge the branch or stop claiming the fix is shipped. (On 2026-07-24 a fix left unmerged on a branch produced a second spend spike identical to the first.)\n\n## Verification commands\n\n```bash\n# Step 1: script log + state\ntail -40 $HOME\u002F.hermes\u002Flogs\u002Fspend-alert-daily.log\njq . $HOME\u002F.hermes\u002Fstate\u002Fspend-alert-state.json\n\n# Step 2: channel membership & recent posts\nTOK=$(bash -c 'source ~\u002F.bashrc; echo $HERMES_SLACK_BOT_TOKEN')\ncurl -fsS -H \"Authorization: Bearer $TOK\" \\\n     \"https:\u002F\u002Fslack.com\u002Fapi\u002Fconversations.list?types=public_channel,private_channel&limit=200\" \\\n  | jq '.channels[] | select(.id==\"C0BCVG4F560\")'\n\n# Step 3: identify the cost-driver repo\ngh api \"orgs\u002Fjleechanorg\u002Fsettings\u002Fbilling\u002Fusage\" --paginate \\\n  | jq '[.usageItems[]? | select(.product == \"actions\") | select(.date == \"2026-07-01T00:00:00Z\")] | sort_by(-.netAmount)'\n\n# Step 4: identify the runaway workflow in that repo (last 32h)\n# NOTE: `created=>` query param is treated as equality, not greater-than.\n# Use `sort=created&direction=desc` and walk pages manually until the\n# oldest run's `created_at` \u003C SINCE.\nREPO=\"jleechanorg\u002F\u003Crepo>\"\npython3 .\u002Fscripts\u002Fgh-cost-audit.py   # see scripts\u002F dir\n```\n\n## Prevention — durable fix shape\n\n- **Mode B (script crash):** `scripts\u002Fspend-alert-daily.sh`: change `log_warn \"$1\"` to `log_warn \"${1:-}\"` (and same for `log`, `log_error`), OR remove `-u` from the script's `set` line. Pair with `scripts\u002Ftests\u002Ftest_spend_alert_idempotent.py` (or similar) so the crash mode is regression-tested. **Status 2026-07-23: fix is in place; no crash observed today.**\n- **Mode A (runaway workflow):** the recurring shape is `comment-router.yml` patterns — workflows that listen to high-volume `issue_comment` events and use a job-level `if:` guard. The job guard short-circuits before billable minutes, but the workflow still registers an event. Mitigations:\n  - Add `if:` at the **workflow\u002Fjob level** that filters by `author_association`, comment body prefix, and `issue.pull_request` BEFORE the runner is allocated\n  - Set `concurrency` group to a tight key (`comment-router-${{ github.event.comment.id }}` is already present in this file — verify it)\n  - Audit who posts comments at 174\u002Fhr — likely a bot loop or external webhook source\n  - For the long term, consider an org-wide CODEOWNERS + workflow-level `if: contains(fromJSON('[\"OWNER\",\"MEMBER\",\"COLLABORATOR\"]'), github.event.comment.author_association)` guard\n- **Mode A.2 (upstream daemon runaway):** the recurring shape is a dedup key that uses a fresh-per-tick value (e.g. `bead_id`) instead of a stable identifier (e.g. `head_ref_name`). Mitigations:\n  - All dedup keys must be drawn from STABLE identifiers (branch name, PR number, commit SHA) — never from objects that get re-created every tick\n  - The dedup check must run BEFORE the SCM write (`comment_external`, `gh api repos\u002F...\u002Fissues\u002F...\u002Fcomments`) — otherwise the write is never suppressed even if the dedup would have matched\n  - Add a `cfg.escalation_refire_secs` cooldown (default 3600s) so re-emission is bounded even if the dedup logic has a bug\n  - **Every fix commit must be verified as on the deployment branch before declaring \"fix shipped\"** — `git merge-base --is-ancestor \u003Cfix-sha> \u003Cdeployment-branch>`. A commit on a fix branch is not a fix until that branch is merged into main AND the daemon is rebuilt AND the running launchd job is restarted.\n\n## Pitfalls\n\n- **The bridge is usually NOT broken** when a Spend Alert appears in session context. MCP Agent Mail relays everything from #worldai-alerts into the session — that's working as designed. Spend the first 30 seconds confirming the Slack post landed, then pivot to the cost driver. Don't waste a turn fixing a non-broken bridge.\n- **\"Fix is in PR #N\" ≠ \"fix is shipped\".** The skill recipes (and older incidents) often cite a PR number or branch name as if the fix is live. **ALWAYS verify with `git merge-base --is-ancestor \u003Cfix-sha> HEAD`** before claiming the fix is merged. The 2026-07-24 incident was caused by trusting a previous skill's stated fix location without re-verifying.\n- **The `created=>` query param on `gh api repos\u002F\u003Crepo>\u002Factions\u002Fruns` is treated as EQUALITY, not greater-than.** It returns 0 hits even when plenty of runs exist after the date. Use `sort=created&direction=desc` + manual pagination instead.\n- **`\u002Factions\u002Fruns\u002F{id}\u002Ftiming` returns `billable: {}` for completed runs.** The billable field is reserved for in-progress \u002F queued runs. To approximate cost, use `updated_at - created_at` to compute wall-clock minutes and multiply by the runner type's $\u002Fmin rate. The `gh-actions-cost-monitor.sh` script does this with `COST_PER_MINUTE=0.002` (Linux self-hosted rate).\n- **`gh-api` `repos\u002F\u003Crepo>\u002Fissues\u002Fcomments` ignores the `since` query parameter.** The `since` field on the comment-schema response is also undefined. To filter comments to a time window, paginate (sort=created + direction=desc) and filter by `created_at` in-process.\n- **The `runs` API sees `run_duration_ms` as 0\u002Fnull for skipped runs** even if the workflow allocated a runner. Use `runs.total_count` to gauge trigger volume, not minutes.\n- **Don't trust `run_duration_ms` field** — it returns `null` or `0` for `skipped` runs, even if the workflow was allocated a runner. Use `updated_at - created_at` to get the real wall-clock time per run.\n- Don't assume `is_member=False` from `mcp__slack__conversations_replies` is global to the bot — it's per-identity. The `HERMES_SLACK_BOT_TOKEN` env may resolve to a different bot identity than the one bound to the live Slack MCP server. Use `conversations.history` with the bot token to verify, not `conversations_replies` from the scoped MCP.\n- **Don't trust a `bead_id` as a dedup key.** Per-tick fresh objects (new bead IDs, new session IDs) make the ledger never match. Use stable identifiers (branch name, PR number, commit SHA) for dedup keys.\n- Don't post the user's reply via `mcp__slack__conversations_add_message` from this session context — it's the wrong identity. Use `chat.postMessage` with the bot token from `~\u002F.bashrc`.\n- **GH billing API returns monthly buckets only** — `date == \"2026-07-01T00:00:00Z\"` is the entire month, not a day. Don't try to slice it by day; you'll see empty results. The script's `gh_delta` is a 24h-difference between two MTD snapshots, NOT a true daily-spend figure.\n- The Agent Mail relay is **in addition to**, not a replacement for, the Slack post. Don't \"fix\" by silencing the cron — fix the crash and keep both channels.\n- **A \"skipped\" workflow run still counts.** `conclusion=skipped` means 0 billable minutes, but the workflow_run event is registered. If a workflow fires 5,000+ times in 32h, the `runs` API is polluted even if no minutes are billed. Use the runs API for diagnosing trigger volume, not the billing API.\n- **Comment-router noise is usually a SYMPTOM, not the cause.** When `comment-router.yml` fires 5,000+ times\u002F32h on `issue_comment`, the first instinct is to harden the workflow (add author_association guard, lower concurrency). That fixes the symptom but misses the cause: an upstream automation (typically a PAT-driven daemon) is in a feedback loop posting the same comment to 1-3 PRs every ~1h. Pull the actual comment bodies via `repos\u002F\u003Crepo>\u002Fissues\u002Fcomments?per_page=100&direction=desc&sort=updated` and group by `user.login` + `body` prefix BEFORE proposing a workflow fix. If 99% of comments come from one automation with one body prefix, the fix lives at that automation, not at the comment-router. See Mode A.2 for the fix.\n\n## Companion skills \u002F files\n\n- `references\u002Fcomment-router-runaway-2026-07-23.md` — original 2026-07-23 incident detail\n- `scripts\u002Fgh-cost-audit.py` — runs\u002Fworkflow volume audit helper (32h window, per-workflow breakdown with skip counts)\n- `scripts\u002Fcomment-audit.py` — issue_comments audit helper (per-author, per-prefix, per-hour breakdown for time-windowed analysis)\n",{"name":5,"description":7},"main","8144f00ebafc4306264c63239e4b9fa42e48a90e","hermes\u002Fskills\u002Fspend-alert-bridge",0,"https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills","https:\u002F\u002Fgithub.com\u002Fjleechanorg","2026-11-11 22:19:36",{"source":8,"skillName":5,"commands":50,"prompt":60,"zipAvailable":61},[51,54,57],{"tool":52,"command":53},"npx","npx skills add jleechanorg\u002Fjleechan-skills --skill spend-alert-bridge",{"tool":55,"command":56},"bunx","bunx skills add jleechanorg\u002Fjleechan-skills --skill spend-alert-bridge",{"tool":58,"command":59},"pnpm","pnpm dlx skills add jleechanorg\u002Fjleechan-skills --skill spend-alert-bridge","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002Fhermes\u002Fskills\u002Fspend-alert-bridge\u002FSKILL.md",true,[63,69,75],{"path":64,"resourceType":65,"resourceTypeDesc":66,"sizeBytes":67,"sourceHtmlUrl":68},"references\u002Fcomment-router-runaway-2026-07-23.md","REFERENCE","参考文档",10103,"https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002Fhermes\u002Fskills\u002Fspend-alert-bridge\u002Freferences\u002Fcomment-router-runaway-2026-07-23.md",{"path":70,"resourceType":71,"resourceTypeDesc":72,"sizeBytes":73,"sourceHtmlUrl":74},"scripts\u002Fcomment-audit.py","SCRIPT","脚本",3906,"https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002Fhermes\u002Fskills\u002Fspend-alert-bridge\u002Fscripts\u002Fcomment-audit.py",{"path":76,"resourceType":71,"resourceTypeDesc":72,"sizeBytes":77,"sourceHtmlUrl":78},"scripts\u002Fgh-cost-audit.py",3422,"https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002Fhermes\u002Fskills\u002Fspend-alert-bridge\u002Fscripts\u002Fgh-cost-audit.py",[80,98,120,134,148,164],{"skillNo":81,"name":82,"nameZh":6,"description":83,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":84,"categories":85,"keywords":86,"publishedAt":37},"01M716Y5MTJ6GYJZ2ATXCJK8G4","instinct","Use when the user invokes \u002Finstinct or \u002Fin, or asks to \"message Instinct\", \"ask Instinct\", \"tell Instinct\", \"check Instinct\", or read replies from the configured peer assistant. iMessage first, WhatsApp Web as fallback.","https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Finstinct\u002FSKILL.md",[],[82,87,88,89,90,91,92,93,94,95,96,97],"user","when","replies","check","use","identity","whatsapp","gmail","read","message","ask",{"skillNo":99,"name":100,"nameZh":6,"description":101,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":102,"categories":103,"keywords":109,"publishedAt":37},"01M716Y4HF7FVVMJSA2CCVN5VF","hr","Compatibility alias for history-resume; produce the same fresh-state, provenance-aware resume briefing.","https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fhr\u002FSKILL.md",[104],{"categoryNo":105,"categoryName":106,"taxonomyType":107,"taxonomyTypeDesc":108},"01JBSEEDCAT000000000000038","人力资源","PROFESSION","适用职业",[100,110,111,112,113,114,115,116,117,118,119],"resume","compatibility","provenance","for","history","produce","alias","briefing","state","fresh",{"skillNo":121,"name":122,"nameZh":6,"description":123,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":124,"categories":125,"keywords":127,"publishedAt":37},"01M716Y1FFNFKFJYYKF8HQA6T8","history-resume","Resume briefing for the current repository branch or worktree using fresh state and admitted history sources.","https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fhistory-resume\u002FSKILL.md",[126],{"categoryNo":105,"categoryName":106,"taxonomyType":107,"taxonomyTypeDesc":108},[114,110,128,129,87,130,117,131,113,132,133,118,119],"history resume","branch","analysis","repository","worktree","sources",{"skillNo":135,"name":136,"nameZh":6,"description":137,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":138,"categories":139,"keywords":140,"publishedAt":37},"01M716XWXJ8EPHK4RH9T14TR85","grok","Alias for the canonical grokbot browser conversation workflow; \u002Fgrok talks to the Grok bot the same way \u002Fdot talks to Dot.","https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fgrok\u002FSKILL.md",[],[136,141,142,143,113,144,116,145,146,147],"dot","bot","workflow","canonical","conversation","browser","grokbot",{"skillNo":149,"name":150,"nameZh":6,"description":151,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":152,"categories":153,"keywords":154,"publishedAt":37},"01M70N9Q07DTE0EW5KFXMMMSJ0","codexa-bench","Compare `codexa` and normal `codex` on a fair split of comparable work by default; use `--strict` for duplicate runs of the exact same task.","https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Fcodexa-bench\u002FSKILL.md",[],[155,156,157,91,158,113,159,160,161,162,163],"codexa bench","work","split","default","duplicate","task","compare","normal","codex",{"skillNo":165,"name":166,"nameZh":6,"description":167,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":168,"licenseFileUrl":169,"sourceHtmlUrl":170,"categories":171,"keywords":181,"publishedAt":37},"01M6C24S2X60SP8BBFGAEKPEGW","frontend-design","Create distinctive, production-grade frontend interfaces with high design quality. Use this skill when the user asks to build web components, pages, artifacts, posters, or applications (examples include websites, landing pages, dashboards, React components, HTML\u002FCSS layouts, or when styling\u002Fbeautifying any web UI). Generates creative, polished code and UI design that avoids generic AI aesthetics.","Apache-2.0","https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Ffrontend-design\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Fjleechanorg\u002Fjleechan-skills\u002Fblob\u002Fmain\u002F.claude\u002Fskills\u002Ffrontend-design\u002FSKILL.md",[172,175,178],{"categoryNo":173,"categoryName":174,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000009","前端开发",{"categoryNo":176,"categoryName":177,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000015","视觉设计",{"categoryNo":179,"categoryName":180,"taxonomyType":107,"taxonomyTypeDesc":108},"01JBSEEDCAT000000000000032","设计师",[182,183,184,91,185,186,187,87,188,189,190,191,192,193,194,195,196,197,198,199,200,201,202,203,204,205,88,206],"frontend","design","frontend design","generic","ui design","react","html","components","code","artifacts","ui","aesthetics","Production","web components","landing","applications","quality","web","css","create","examples","with","react components","web ui","pages"]