bug-triage

Bug Triage Skill

Triage bugs into well-structured GitHub issues on the correct upstream repo.

1. Pre-flight

• Pull latest code: git pull origin main. Stale code = bad triage.
• Target repo: Always file on the current repo (Untrivial-ai/agent-orchestrator), not forks.
• Record source: chat URL, reporter name, attachments.

2. Gather Context

2a. Extract the report

Source	How to gather
Discord/Slack thread	Read full thread. Extract: reporter name, original description (the thread starter, not whoever tagged you), screenshots, follow-ups
GitHub issue	gh issue view <number> --repo <repo> --json body,comments
Live observation	Pull live state via observability tools

2b. Minimum viable report gate

Before tracing code, verify the report has enough substance:

Required (ALL): what happened, where (page/command/feature), when (after upgrade? first time?)

Required (2 of 4): OS/shell/runtime, AO version (ao --version), reproducibility (consistent vs intermittent), reproduction steps

If insufficient, ask:

"I'd like to triage this but need more info: (1) What happened? (error/behavior), (2) Where? (page/command), (3) When did it start?, (4) How to reproduce?"

2c. Local diagnostics (if bug is on same machine)

Gather everything yourself before asking the reporter:

# Environment
ao --version && node --version && echo $SHELL && uname -a
cat agent-orchestrator.yaml
cat ~/.agent-orchestrator/running.json

# Process health
pm2 status
tmux list-sessions
lsof -i :3000

# AO event log — structured timeline
ao events list --limit 50                          # recent events
ao events list --session ao-5 --limit 100          # filter by session
ao events list --log-level error --since 1h        # errors only
ao events search "spawn failed"                    # full-text search
ao events stats                                    # counts by kind/source

# Session state files
cat ~/.agent-orchestrator/projects/*/sessions/*.json | python3 -m json.tool

Event kinds: session.spawned, session.spawn_failed, session.killed, lifecycle.transition, ci.failing, review.pending, runtime.probe_failed, agent.process_probe_failed, reaction.escalated, lifecycle.poll_failed. Sources: lifecycle, session-manager, api, runtime, agent, reaction.

Try the reproduction steps. Running the actual command is worth 100 lines of code tracing.

3. Investigate

3a. Trace the code path

Always trace the actual code — don't surface-level diagnose. #1129 looked like a simple ao stop issue but was actually a session lineage/cascade problem.

git fetch origin main && git log --oneline origin/main -5   # current HEAD
# Record the commit hash you're analyzing against

Git archaeology — find which commits introduced/removed specific code:

git log --oneline -S 'exact-string' -- <file>
git show <sha> -- <file> | grep -B 5 -A 10 'pattern'

Example: #1391 traced a mobile layout break to a display: flex → display: grid change.

Research upstream dependencies (xterm, node-pty, React, etc.) — check installed vs latest version, search their GitHub issues, check changelogs. Root cause is often upstream.

3b. Cross-platform check

AO runs on Windows, macOS, Linux as first-class targets. If env info indicates Windows (or is unknown), check for these patterns:

• Path separators — hardcoded / or \ breaks cross-platform
• Shell syntax — PowerShell lacks &&, $VAR, $(cat ...), /dev/null, here-docs
• process.platform === "win32" inline — must use isWindows() from @aoagents/ao-core
• process.kill(-pid) — POSIX-only; use killProcessTree()
• Named pipes vs Unix sockets — Windows uses \\.\pipe\ao-pty-<id>
• localhost — Windows resolves to ::1 first, causing ~21s stalls on IPv4-only servers
• NTFS case-insensitivity — use pathsEqual(), not ===
• ConPTY orphans — can trigger WER dialogs if pty-host not shut down cooperatively
• .cmd shim resolution — needs shell: true for PATHEXT lookup

Key files: packages/core/src/platform.ts, docs/CROSS_PLATFORM.md, packages/plugins/runtime-process/, packages/cli/src/lib/path-equality.ts

3c. Stop-and-ask triggers

Stop and ask for more info if:

• 3 failed hypotheses — traced 3 code paths, none explain it
• Root cause is upstream — file with upstream reference, don't guess a local fix
• UI-only bug and you can't screenshot — ask reporter to describe
• Can't reproduce — ask for different config/sequence

4. Search for Duplicates

Search with multiple strategies, always using --state all (closed bugs regress):

gh issue list --repo <repo> --state all --search "<symptom>"
gh issue list --repo <repo> --state all --search "<component-name>"
gh issue list --repo <repo> --state all --search "<error-message>"
gh pr list --repo <repo> --state all --search "<keywords>"

Duplicate found → comment on existing issue

gh issue comment <number> --repo <repo> --body "$(cat <<'EOF'
## New Report
**Reported by:** @<reporter> in [chat](<url>)
**Date:** <YYYY-MM-DD> | **Checkout:** `<commit-hash>`
<context, differences from original, screenshots>
EOF
)"

No duplicate → file new issue (next section)

5. File New Issue

5a. Pre-submission checklist

• [ ] Reporter attribution correct (original reporter, not who tagged you)
• [ ] Commit hash recorded
• [ ] AO version recorded
• [ ] Root cause confidence scored (see 5c)
• [ ] Related issues cross-linked
• [ ] Reproduction steps are concrete
• [ ] Screenshots uploaded with real URLs (see 5b)

5b. Upload screenshots

⛔ NEVER use placeholder URLs. Upload BEFORE creating the issue. (#1151 RCA on this pattern.)

SLUG="descriptive-slug"
# Create asset branch
gh api -X POST repos/<repo>/git/refs \
  -f ref="refs/heads/issue-assets-${SLUG}" \
  -f sha=$(git rev-parse origin/main)

# Upload (portable base64)
IMG_B64=$(base64 < /path/to/screenshot.png | tr -d '\n')
gh api -X PUT "repos/<repo>/contents/.issue-assets/${SLUG}/name.png" \
  -f message="chore: upload screenshot" \
  -f content="$IMG_B64" \
  -f branch="issue-assets-${SLUG}"
# Use: ![screenshot](https://raw.githubusercontent.com/<repo>/issue-assets-<slug>/.issue-assets/<file>)

5c. Create the issue

gh issue create --repo <repo> --title "<title>" --body "$(cat <<'EOF'
## Bug
<summary>

**Source:** <url> | **Reported by:** @<reporter> | **Analyzed against:** `<hash>`
**Confidence:** High/Medium/Low

## Reproduction
1. <step>

## Root Cause
<file paths, line numbers, explanation>

## Fix
<suggested approach>

## Impact
- <effects>
EOF
)"

5d. Label and prioritize

gh issue edit <number> --repo <repo> --add-label "bug"

Priority label	Criteria
priority: critical	Data loss, security, system down
priority: high	Core feature broken, no workaround
priority: medium	Feature degraded, workaround exists
priority: low	Cosmetic, edge case

Confidence scoring (include in issue body):

Level	Meaning	Extra labels
High	Traced exact code path, specific lines, mechanism explained	bug only
Medium	Strong hypothesis but unconfirmed	bug, to-explore
Low	Can't trace, multiple conflicting theories	bug, to-reproduce

Example: PR #1608 was diagnosed High as xterm v6 issue — real cause was a = prefix on tmux set-option. Should have been Medium.

All available labels: priority: critical/high/medium/low, bug, enhancement, good-first-issue, to-reproduce, to-explore. No others (no p0, p1, etc.).

5e. Cross-link related issues

Search by subsystem and add a ## Related section to the issue body:

## Related
- [#1020](url) — stale session blocking ao start (same subsystem)
- [#1035](url) — same race condition

5f. Push a fix PR (always attempt)

• Trivial fix: Push immediately.
• Complex fix: Note in issue, suggest spawning an agent.
• Unclear fix: Don't push a guess. Document and flag.

OLD_STRING='<old>' NEW_STRING='<new>' \
python3 skills/bug-triage/scripts/push_fix_to_github.py \
  <owner/repo> fix/slug path/to/file.tsx \
  "fix(scope): commit msg" "fix(scope): PR title" \
  "Fixes #<n>

## Summary
<what changed>

## Test
<how to verify>"

The script reads from GitHub API, applies one replacement, pushes, opens PR — no local checkout needed. Verify OLD_STRING matches GitHub first: gh api repos/<repo>/contents/<path>?ref=main -q '.content' | python3 -c "import base64,sys; sys.stdout.buffer.write(base64.b64decode(sys.stdin.read()))"

Multiple edits to same file: The push script does one replacement per run. For multiple changes, use a Python script to read from the branch (gh api + base64.b64decode), apply all replacements, then push via gh api -X PUT with the updated SHA.

5g. Report back

Issue URL, PR URL (if created), labels, root cause summary, whether fix agent was suggested.

---

Appendix

A. Subsystem Quick Reference

Subsystem	Collect	Key files
CLI (ao start/stop/spawn)	Config YAML, install method, version, OS	packages/cli/src/commands/
Web UI	Screenshot, browser, viewport	packages/web/src/components/, globals.css
Terminal	Runtime type, tmux version, shell	DirectTerminal.tsx, useXtermTerminal.ts
Lifecycle	State transitions, session IDs	core/src/lifecycle-manager.ts, core/src/lifecycle-state.ts
Sessions	Session ID, spawn config, runtime	core/src/session-manager.ts
Plugins	Plugin name, agent version	packages/plugins/<agent>/
Config	YAML contents, project path	packages/core/src/config.ts

Misrouting patterns:

• Terminal bugs → tmux (runtime-tmux) vs xterm (web) vs PTY (runtime-process/Windows). Trace where bytes flow.
• "Session stuck" → lifecycle state machine vs agent process vs runtime connection.
• "Config not saving" → config loading (c12) vs project registration (running-state.ts) vs YAML write (permissions).

B. Remote Code Inspection (no local clone)

gh api repos/{owner}/{repo}/git/trees/main?recursive=1 --jq '.tree[].path'    # list files
gh api repos/{owner}/{repo}/contents/{path} --jq '.content' | python3 -c "import base64,sys; sys.stdout.buffer.write(base64.b64decode(sys.stdin.read()))"  # read file
gh search code "term" --repo {owner}/{repo} --json path --jq '.[].path'        # search code
gh api "repos/{owner}/{repo}/commits?path={path}&per_page=10" --jq '.[] | "\(.sha[0:8]) \(.commit.message | split("\n")[0])"'  # file history

C. NPM Package Regression Diffing

Diff published packages (not local builds) when regression follows an upgrade:

mkdir -p /tmp/ao-diff/{v1,v2}
curl -sL https://registry.npmjs.org/@scope/pkg/-/pkg-OLD.tgz | tar xz -C /tmp/ao-diff/v1
curl -sL https://registry.npmjs.org/@scope/pkg/-/pkg-NEW.tgz | tar xz -C /tmp/ao-diff/v2
diff -rq /tmp/ao-diff/v1/package/ /tmp/ao-diff/v2/package/

Example: PR #1608 — source analysis led to wrong theories, npm diff showed the only change was a = prefix on tmux set-option.

Formatting Rules

• Linkify all issue/PR refs: [#123](https://github.com/Untrivial-ai/agent-orchestrator/issues/123), [PR #456](url). Never bare #123.

Pitfalls

• Reporter ≠ person who tagged you. Always attribute to the original reporter.
• Record the commit hash you analyzed — code changes fast.
• GitHub issue is mandatory — every triaged bug gets one, even if fix is trivial.
• gh api --jq .content truncates large files (>~100KB). Use local git instead.
• Push script arg limits — long commit messages hit OSError: Argument list too long. Use a Python script with JSON payloads instead.
• OLD_STRING must match GitHub byte-for-byte — local code may differ from origin/main.
• New fields on shared TS interfaces MUST be optional (field?: Type). Downstream Partial<X> spreads break on required fields. Example: PR #1523.