incident
Production incident response. Walks docs/runbooks/incident.md step by step, proposes the rollback (or kill switch) FIRST when a release lines up with the start, then traces the failure from error_id / request_id to the line of code, ships the fix, and drafts the postmortem with the guard it added. Use when the app or API is down, broken, crashing or slow for users, when a Sentry alert fires, when /health says degraded, or when someone says "something is wrong in production".
- License 许可
- MIT
- 最近更新
- 2026-10-06
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- incident
- description
- Production incident response. Walks docs/runbooks/incident.md step by step, proposes the rollback (or kill switch) FIRST when a release lines up with the start, then traces the failure from error_id / request_id to the line of code, ships the fix, and drafts the postmortem with the guard it added. Use when the app or API is down, broken, crashing or slow for users, when a Sentry alert fires, when /health says degraded, or when someone says "something is wrong in production".
- argument-hint
- [what users see, an error code, or a Sentry link]
03 SKILL.md 正文
Incident
Stop the bleeding first, understand it second. Most incidents start with a release,
and rolling that release back takes a minute while a diagnosis takes an hour. So this
skill finds what changed and proposes undoing it before it reads a single stack
trace. The runbook is docs/runbooks/incident.md; this walks it in order and keeps a
timeline as it goes (times in UTC, for the postmortem).
1. Is it live, and how big? (2 minutes)
curl -s "$API_URL/health?deep=1" # status, db, features_unavailable (a missing env var shows by name)
Then the size: Sentry issue (users affected, trend, which release), PostHog
api_failed by path / status over the last hour, and the crash-free rate of the
newest release. Set a severity from the runbook's table (SEV1 data exposure or
everyone broken, SEV2 a core flow broken for some, SEV3 degraded with a workaround).
2. What changed? Propose the rollback first
List every change in the window before the first signal, newest first:
| Layer | How to see it |
|---|---|
| OTA update | cd mobile && eas update:list --branch production --limit 5 (a staged rollout shows its %) |
| API deploy | railway deployment list, or /health version against git log --oneline -5 origin/main |
| Migration | ls -t supabase/migrations/ | head -3 and its merge time |
| Flags | PostHog feature-flag history; FLAG_* variables on Railway |
If one lines up with the start, propose undoing it now, before any diagnosis.
Name the exact command from docs/runbooks/rollback.md:
- OTA:
scripts/rollback-ota.sh(it plans first; it reverts a staged rollout instead of publishing over it), thenscripts/rollback-ota.sh --yes -m "<reason>". - API:
railway redeploy --deployment <last good id>, then confirm/healthversion. - A feature behind a kill switch (
kill-*inmobile/lib/flags.ts/backend/flags.py): turn that flag ON in PostHog (both app and API read it), or setFLAG_KILL_<NAME>=1on Railway if PostHog itself is the problem. - A migration: usually roll the API back and leave the schema (migrations are additive); the runbook covers the rare real undo.
Ask with a structured question: the rollback (Recommended), fix forward, or keep
investigating. Run it only on a yes. Afterwards confirm it worked: the Sentry issue
stops getting events for the new release, /health is ok, the funnel recovers.
If nothing lines up (no change in the window), say so and go to step 3: a rollback with no suspect is a guess.
3. Find the line of code
- Start from an id: the
error_idthe user read out (the error screen shows it), a Sentry issue, or a time. - Sentry →
error_id:<id>→ stack trace, release,request_idtag. Events are scrubbed by design (no bodies, user data or exception messages). - Railway logs carry
[<request_id>]on every line (@request_id:<id>withLOG_FORMAT=json). The request id is also the trace id: the app'straceparentuses the request id without its dashes, so a sampled request (5% by default) has a Sentry trace with its spans under that id. - Reproduce it in a test that fails before the fix (the filter-honouring fakes in
tests/test_prod_fakes.py, a jest test for app code).
4. Fix forward
File it with the backlog skill (a fix ticket, p0 for SEV1/SEV2), branch, fix
with the failing test now passing, and land it. JS-only fixes go out as an OTA
(ship skill); API fixes deploy from main. If you rolled back in step 2, the fix
re-ships what the rollback removed.
5. Postmortem, with its guard
Draft docs/postmortems/YYYY-MM-DD-<slug>.md from docs/postmortems/TEMPLATE.md,
filled from the timeline you kept. Blameless: the gap in the system, never a person.
The Guard added section links the test or check that now fails on this exact
failure, proven against a planted violation; tests/test_postmortems.py fails CI
without one. Put the postmortem in the fix's PR, or a follow-up PR when the fix
couldn't wait. Write a memory note (reflect) only if the lesson is non-obvious and
will still be true in 30 days.
Reply in under 8 lines: severity, what changed, what you rolled back (or why not), the cause, the fix PR, and the guard.
Ask the owner
Follow .agents/rules/product-judgement.md: ask with a structured question (recommended option first), never decide these silently. In this skill that means: whether to roll back, flip a kill switch or fix forward (rollback recommended when a release lines up); what users are told and where (in-app, status page, email) for a SEV1 or SEV2; and whether a data exposure needs notifying. Never roll back, flip a flag or message users without a yes.
04 同仓库更多 Skills
ship
Release checklist for getting a build to testers or the stores. Runs the risk gate first (refuses while a risk item is open) and walks the owner through the compliance checklist. Covers the version bump, EAS build and submit, OTA updates and when they're safe, changelog, store listing, privacy nutrition label / data safety form, the Accessibility Nutrition Labels, and the account-deletion requirement. Guidance-level; it never handles signing secrets. Use when asked to ship, release, submit to TestFlight / App Store / Play, or push an OTA update.
routines
Put the project's recurring rituals on a schedule, with the owner's consent. Weekly north-star report, monthly market watch, dependency triage, Supabase advisor sweep, reflect and harness-optimize. Uses Claude Code Routines (cloud cron) where they fit, and documents the Codex / local cron equivalent. Use when asked to automate, schedule or "keep the project moving on its own", or after setup finishes.
reflect
Consolidate what recent sessions learned into the project's long-term memory (.agents/memory). Use at the end of a substantial session, after a correction from the user, after a painful debugging session, when the session-start healthcheck says reflect is overdue or a pending-reflection seed is waiting, or when MEMORY.md exceeds its budget.
pr-review
Review a pull request on the owner's behalf and fix what's safe to fix. Runs the deterministic gates first, then correctness/security, design-system, accessibility and analytics review passes, verifies each blocking finding, applies safe fixes, re-checks, and ends with a plain-English verdict. Use when a PR is opened, when asked to review a PR, a branch or a pasted diff, or before merging anything.
north-star-report
Weekly product-analytics and reliability readout. Pulls the funnel of the brief's 5 north-star events from PostHog, finds the biggest drop-offs and errors, checks each SLO in docs/slo.yaml against its error budget, compares with last week, and proposes (then, with a yes, files) tickets that target them. Use weekly, when the session-start line says it's overdue, when the backlog is empty, or when asked "how is the app doing?".
next
Decide the single next best thing to work on, with two alternates. Reads setup progress, work in progress, open PRs and failing CI, the backlog (GitHub Issues or Linear), overdue harness rituals, product decisions parked until now, and north-star analytics if PostHog is connected. Use when someone asks "what next?", "what should I work on?", at the start of a session, or after a PR merges.