macos-incident-postmortem

macOS Incident Postmortem

When to Use This Skill

Use this skill when...	Use something else when...
GUI froze and you're not sure if the machine rebooted	Live-debugging a hung process — use sample / spindump
Investigating recent kernel panics or watchdog timeouts	Application crashes — open the per-app .crash / .ips directly
Cross-referencing "what was I doing at time T?" against logs	Active CPU diagnosis — see launchservices-health
Auditing whether the system is "due for a reboot"	Pre-incident hardening — wrong skill, this is forensics

Platform Guard

This skill is macOS-only. /Library/Logs/DiagnosticReports/, kern.boottime, pmset -g log, and the log command are Darwin-specific. Refuse to act if uname -s is not Darwin.

test "$(uname -s)" = "Darwin" || { echo "macos-plugin: not Darwin, refusing"; exit 1; }

Core Expertise

A macOS "incident" can mean any of:

• A kernel panic (system reset by the kernel)
• A WindowServer userspace watchdog timeout (the GUI froze and the user power-cycled)
• A LaunchServices / coreaudiod / mds-stores XPC stall (one daemon dragged the GUI down)
• A jetsam memory-pressure event (the kernel killed apps to reclaim RAM)
• A thermal throttle (CPU clamped to base frequency for minutes)
• A user-initiated force-reboot after a hang (no panic; just lost state)

The first job in a postmortem is distinguishing between actual reboots and GUI hangs. The 2026-04-22 incident that motivated this plugin looked like a crash to the user but last reboot showed no reboot — the kernel was fine, only the GUI stack hung.

The second job is timeline reconstruction: cross-reference Diagnostic Reports, kern.boottime, last reboot, last shutdown, and shell history to answer "what happened around time T?".

Gather the deterministic signals

Run the bundled script to gather every mechanical signal in one pass — reboot-vs-hang detection, per-category DiagnosticReports counts, the CPU-event offender histogram, the jetsam victim list, and the fixed reboot/hang classifier:

bash "${CLAUDE_SKILL_DIR}/scripts/macos-incident-postmortem.sh" --home-dir "$HOME"

Parse STATUS= and ISSUES: from the output. The script emits CLASSIFICATION= (REBOOT_PANIC / REBOOT_CLEAN / HANG_UI / HANG_CPU / HANG_JETSAM / HANG_OR_POWERCYCLE / UNKNOWN) with a CLASSIFICATION_REASON=, plus PANIC_COUNT= / HANG_COUNT= / JETSAM_COUNT= / CPU_RESOURCE_COUNT= etc., LATEST_PANIC= / LATEST_HANG= paths, and CPU_OFFENDERS: / JETSAM_VICTIMS: histograms.

Pass the incident time T as an epoch to enable the classifier (MACOS_PM_INCIDENT_EPOCH=<epoch> bash …), or supply an alternate report source with --reports-dir <path>. With no incident time the classifier reports UNKNOWN rather than guessing — feed it T once you know it.

The reboot/hang decision tree the script encodes:

Pattern	Interpretation
Boot at/after T with a .panic report	True reboot from a kernel panic
Boot at/after T, no panic	Clean restart (user-initiated or watchdog)
Boot before T with a .hang/.spindump.txt	GUI hang — machine did not reboot
Boot before T with a .cpu_resource.diag	Daemon CPU storm
Boot before T with a JetsamEvent-*	Memory-pressure kill
Boot before T, none of the above	Power loss or hard power-cycle

When a report filename or category is unfamiliar, or last reboot comes back empty because wtmp rotated past T, open references/diagnostic-reports.md for the suffix → category → severity table and the unified-log boot/shutdown query.

Timeline Reconstruction (judgment)

With the deterministic classification and counts in hand, the remaining steps are the agent's job: read the panic backtrace, name suspect kexts, correlate with logs and shell activity, and write the narrative.

Step 1: Inspect the unified log around T

# Adjust the time window to bracket T
log show --start "2026-04-22 08:15:00" --end "2026-04-22 08:25:00" \
  --predicate 'subsystem == "com.apple.WindowServer" OR process == "launchservicesd" OR process == "coreaudiod"' \
  --style syslog \
  | head -500

Common signatures to grep for:

Signature	Meaning
WindowServer: watchdog	UI froze long enough to trip the watchdog
posix_spawn failures	Fork/exec storm — usually shell loops or runaway scripts
Jetsam Killing	Kernel killing processes for memory
_dispatch_*_timeout	Daemon stuck on a synchronous IPC call
Thermal pressure	CPU thermal-throttled

Step 2: Inspect the panic / hang report

The script's LATEST_PANIC= / LATEST_HANG= lines give you the file to read. Key fields in a panic report:

Field	Meaning
panic(cpu N caller ...)	The instruction that panicked; first line says why
Backtrace (CPU N)	Call stack at the time of panic
Mac OS version, Kernel version	OS state
System uptime in nanoseconds	Uptime at the moment of panic
last loaded kext / loaded kexts	Likely third-party suspect

Hang reports are spindump-style: one column per thread of the hung process (typically WindowServer or the offender daemon), with stack traces at sample intervals.

Step 3: Correlate with shell activity

# Most recent zsh history entries (assumes default zsh)
fc -l -t '%Y-%m-%d %H:%M:%S' -100

# Or directly:
tail -50 ~/.zsh_history

The zsh EXTENDED_HISTORY format : <epoch>:<elapsed>;<cmd> lets you grep for commands run within the incident window.

Step 4: Synthesize

Write a one-paragraph timeline including:

• Type of incident (panic / hang / jetsam / power-cycle)
• Time T and time-since-boot at T
• Top suspects (loaded kexts, busy daemons, low-memory processes)
• What recovery looked like (clean reboot / hard power-cycle / GUI returned on its own)

Common Patterns

The reboot-vs-hang one-liner, per-category report counts, the CPU-event offender histogram, and the jetsam victim list are all produced in one pass by the bundled script (scripts/macos-incident-postmortem.sh — see "Gather the deterministic signals" above). Read CLASSIFICATION=, the *_COUNT= keys, CPU_OFFENDERS:, and JETSAM_VICTIMS: from its output rather than re-running the equivalent shell pipelines. A sudden rise in any category count is a leading indicator before a major hang or panic; offenders/victims that recur are running near their CPU/memory budget.

Correlate with sleep/wake history

pmset -g log | grep -E 'Sleep|Wake|DarkWake' | tail -50

A spurt of DarkWake entries followed by a hang report often means a peripheral or sleep-assertion-holder is the trigger.

Skip List (Common False Suspects)

Suspect	Why it's usually NOT the cause
mds, mds_stores (Spotlight)	Heavy I/O is normal after large file changes; rarely panics
cloudd	High network use is normal during iCloud sync
bird (CloudKit)	Same as cloudd
Time Machine	Throttles itself; almost never the proximate cause
kernel_task at 100%	This is thermal management running, not a bug

If one of these is the only thing visible in your timeline, look harder — the real cause is usually a daemon that blocked on one of these, not the daemon itself.

Agentic Optimizations

Context	Command
All deterministic signals + classifier	bash "${CLAUDE_SKILL_DIR}/scripts/macos-incident-postmortem.sh" --home-dir "$HOME"
Classify against a known T	MACOS_PM_INCIDENT_EPOCH=<epoch> bash "${CLAUDE_SKILL_DIR}/scripts/macos-incident-postmortem.sh"
Last 5 boots	last reboot | head -5
Last 5 shutdowns	last shutdown | head -5
WindowServer log slice	log show --predicate 'subsystem == "com.apple.WindowServer"' --last 1h --style syslog | head -200

Quick Reference

Key paths, useful log show predicates, and time selectors for building the timeline queries: references/quick-reference.md.

Decision Flow

Did `last reboot` advance near time T?
├─ YES → Kernel-level event
│   ├─ *.panic file present? → Panic; read backtrace
│   └─ No panic file → Clean restart (user-initiated or watchdog)
└─ NO → Userspace event
    ├─ *.hang or *.spindump.txt near T? → UI thread hang
    ├─ *.cpu_resource.diag spike near T? → Daemon CPU storm
    ├─ JetsamEvent-* near T? → Memory-pressure kill
    └─ None of the above → Power loss or hard power-cycle

Error Handling

When a command fails or the reports do not reach back to T (permission denied, empty last reboot, slow log show, rotated reports), open references/troubleshooting.md.

Related Skills

• launchservices-health — when the timeline points at launchservicesd, dig deeper there
• kitty-plugin:kitty-session-persistence — recover terminal state lost during the incident