[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6993WZ28KEWZMXDNA1PF28W":3,"skill-same:01M6993WZ28KEWZMXDNA1PF28W":47},{"skillNo":4,"name":5,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":17,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":24,"commitSha":25,"skillDirectory":26,"forksCount":27,"repositoryHtmlUrl":28,"repositoryHomepageUrl":6,"ownerHtmlUrl":29,"ownerBlogUrl":30,"ownerDescription":31,"updatedAt":32,"install":33,"resources":46},"01M6993WZ28KEWZMXDNA1PF28W","dev-debugging",null,"google\u002Fcapsem","google","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F1342004?v=4","01M698VYEFDM0VJBZDYYJ2T83S","Google","01M69939YZJTP726NQK4NZNB48",186,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fcapsem\u002Fblob\u002Fmain\u002Fskills\u002Fdev-debugging\u002FSKILL.md",[],[18],"dev debugging","2026-11-10 10:12:30",false,"---\nname: dev-debugging\ndescription: Debugging methodology: reproduce first, find the root cause, fix the whole pattern. Use for bugs, test failures, or CI\u002Flocal parity failures.\n---\n\n# Debugging\n\n## The rule\n\nNever fix code before you understand why it broke. The temptation to \"just make the test pass\" or \"just patch the symptom\" leads to fragile fixes that hide deeper problems. Follow the three-step workflow below every time.\n\n## Choose the cheapest truthful feedback loop\n\nKeep implementation moving. Use a minimal reproducer and focused TDD checks\nduring coding, then return to the next implementation item. Broad qualification\nruns belong at the end of a major phase; a small fix does not justify another\nwhole contract\u002Fworkspace\u002FVM\u002Finstall run. The bounded wrapper retains timeout\nand cleanup without a routine machine lease.\n\nCapsem now has two distinct execution surfaces:\n\n- Just exposes the small public product interface.\n- Python under `build_system\u002Fbuilder\u002Fgate\u002F` owns build\u002Ftest\u002Frelease plans, graph edges,\n  locking, resources, evidence, and cleanup.\n\nFor a gate failure, debug the Python step and action that failed; do not infer\nthe implementation from the Just recipe or recreate it with shell commands.\nStart with:\n\n```bash\nuv run --project build_system --frozen capsem-gate runs last --failed\nuv run --project build_system --frozen capsem-gate \u003Ccommand> --dry-run\nuv run --project build_system --frozen capsem-gate \u003Ccommand> --graph\n```\n\nUse the smallest focused pytest, cargo, pnpm, or script command for red\u002Fgreen\nwork. Use `just focus-test functional` for focused integration feedback, and\n`just fast-test` for the fast gate itself. Use `just test` when the forward fix\nis ready for the complete local verification a release requires.\n\nDirect diagnostics which may block, build, launch children, or wait on input\nmust use the portable bounded-process wrapper rather than a bare shell command:\n\n```bash\npython3 build_system\u002Fscripts\u002Fci\u002Frun-bounded-command.py --timeout-seconds 1800 -- \u003Ccommand> \u003Cargs...>\n```\n\nThe wrapper closes stdin, creates a dedicated process group, and terminates the\nwhole group on timeout or interruption. This prevents a PTY-backed `docker\nbuild -f -`, compiler, test runner, or child helper from surviving its owning\ndiagnostic. Pick a finite timeout appropriate to the focused operation; do not\nwrap `just test` or a release command, whose config-owned step timeouts,\njournal, teardown, and resumable graph are the authority.\n\n### Diagnostic continuation for a late gate failure\n\nWhen a clean candidate fails after expensive predecessors have succeeded, use\n**diagnostic continuation** to reuse that retained state and reach the frontier\nquickly. The current CLI retains legacy flag names:\n\n```bash\nuv run --project build_system --frozen capsem-gate runs last --failed\nuv run --project build_system --frozen capsem-gate candidate --prefix \u003Cretained-prefix> --from \u003Cfailed-step>\n```\n\nThe `--from` step runs; graph predecessors are reported as `carried`. Before\nusing it, match the prefix and failed step to the origin run, confirm the\nrequired predecessor really completed there, and record the origin run ID in\nthe diagnosis. Prefer a fresh focused test when the edit invalidates the\nexpensive producer itself.\n\nCarried steps with external runtime products are revalidated before resumed\nwork. If one was reclaimed, use the exact owning `--from` label in the refusal;\ndo not guess an earlier frontier or recreate the product by hand.\n\nCall the result `diagnostic-passed` or `diagnostic-failed`, even if the current\nimplementation still summarizes a zero exit as `ok`. It combines earlier\noutputs with current source and therefore answers only whether the new segment\ncan proceed. It does not prove the complete current tree.\n\nNever use diagnostic continuation with `release-binaries` or\n`release-assets`, and never let it authorize publication. After the fix, use\nthe smallest owning `focus-test` group. Before release, run `just test \u003Ccommit>`\nto success: the public release command requires its complete local journal for\nthat commit or an identical Git tree by default. An explicitly authorized\nrelease `force` permits a focused test-only correction without repeating the\ncomplete local run; source guards and full hosted qualification remain.\nThe unattended nightly scheduler is also exempt (`[release].unattended_channels` in\n[`config\u002Fgate.toml`](..\u002F..\u002Fconfig\u002Fgate.toml); see\n[`RELEASE.md`](..\u002F..\u002FRELEASE.md) section 8). The hosted lane then independently\nqualifies the artifact pairing it publishes.\n\n## Step 1: Reproduce with a test\n\nBefore touching any implementation code, write a test that captures the bug. This test must:\n- Fail right now, demonstrating the broken behavior\n- Be specific enough to distinguish the bug from correct behavior\n- Live in the right test location (see dev-testing for where tests go)\n\nIf you can't reproduce it in a test, you don't understand it well enough to fix it. For VM-level issues, use capsem-doctor or write a targeted diagnostic command:\n```bash\njust exec \"\u003Ccommand that triggers the bug>\"\n```\n\nFor telemetry issues, inspect a live session through the typed timeline,\nhistory, statistics, and security routes. Use `check_session.py --db` only on\na coherent, quiesced evidence copy after its ledger worker has stopped; see\n`\u002Fdev-session-debug`.\n\n## Step 2: Diagnose the root cause\n\nWith a failing test in hand, investigate. Do not skip this step. Common diagnostic approaches:\n\n**MCP triage trio (run FIRST when an investigation is open-ended):**\n\n```\ncapsem_panics { since: \"1h\" }       # any Rust panic in any host log? -> rank highest\ncapsem_triage { id: \"vm-1\" }        # ranked recent ipc-warns + 4xx\u002F5xx + slow_ops + session.db errors\ncapsem_timeline { id: \"vm-1\", trace_id: \"\u003CX>\" }   # follow ONE logical operation across exec\u002Ftool\u002Fnet\u002Ffs\u002Fmodel\n```\n\nThese read post-W2 JSON logs (`~\u002F.capsem\u002Frun\u002F{service,mcp,gateway,tray}.log` + capsem-app's latest jsonl) and post-W6 session.db tables. The W4 `target=fs op=fsync duration_ms=...` markers feed `capsem_triage`'s slow-op rank; the W3 schema_hash check appears in `capsem_panics` output as `IPC handshake failed; refusing connection` events. Always start with `capsem_panics` -- a single panic outranks a hundred warns.\n\n**Cross-version mix?** The `service.start` log line emits `protocol_version=N, schema_hash=\u003Chex>` per binary. If the support bundle (`capsem support-bundle`) shows two different schema_hash values across binaries, you're hitting the W3 cross-version-mix detection -- rebuild + restart the lagging binary.\n\n\n\n**Integration-test failures: read the preserved service log.** When any integration test fails, the test fixture (`tests\u002Fhelpers\u002Fservice.py::ServiceInstance`, the e2e `RealService`, and the MCP `_start_capsem_service`) archives its tmp_dir to `cache\u002Ftarget\u002Ftests\u002Fevidence\u002F\u003Ctimestamp>-\u003Cworker>-\u003Cnodeid>\u002F\u003Ctmp-basename>\u002F` **before** the usual rmtree. The failing test's stderr has the exact path: look for a line `ARTIFACT: preserved \u002Fvar\u002Ffolders\u002F... -> cache\u002Ftarget\u002Ftests\u002Fevidence\u002F...`. Inside that directory:\n\n```\nservice.log                     host-side capsem-service debug log (RUST_LOG=debug)\nlogs\u002Fgateway.log                gateway stdout\u002Fstderr\nlogs\u002Ftray.log                   tray stdout\u002Fstderr (if spawned)\nsessions\u002F\u003Cvm-id>\u002Fprocess.log    per-VM capsem-process log (vsock bridge, IPC, spawn chain)\nsessions\u002F\u003Cvm-id>\u002Fserial.log     VM serial console (kernel boot, capsem-init, agent startup)\nsessions\u002F\u003Cvm-id>\u002Fsession.db     SQLite telemetry DB (net_events, model_calls, ...)\npersistent\u002F\u003Cname>\u002F...           persistent-VM state (checkpoint.vzsave, workspace)\n```\n\n`cache\u002Ftarget\u002Ftests\u002Fevidence\u002F` is gitignored. Multiple failures sharing a session-scoped service land in different subdirs but the latest run's name tags them by the most recent failing nodeid. First place to look for \"VM didn't become exec-ready\" style failures: `sessions\u002F\u003Cid>\u002Fserial.log` (did the VM boot?) and `sessions\u002F\u003Cid>\u002Fprocess.log` (did the agent come up + IPC handshake?). For \"provision hung\" or service-side contention: `service.log`, grep for the VM id.\n\n**Rust code**: Read the code path the test exercises. Trace the data flow. Add `tracing` instrumentation if needed (`RUST_LOG=capsem=debug`). Check if the issue is in capsem-core, capsem-app, or capsem-agent.\n\n**Guest VM issues**: Boot with targeted commands and inspect behavior:\n```bash\njust exec \"capsem-doctor -k \u003Ccategory>\"   # Run specific diagnostic category\njust exec \"\u003Cmanual investigation command>\"\n```\nCheck boot logs for daemon startup failures, vsock connection issues, or timing problems.\n\n**Network\u002Fsecurity issues**: Check the network intercept path -- SNI parsing,\nHTTP\u002FDNS\u002Fmodel normalization, cert minting, `SecurityEvent` construction,\nsecurity rule evaluation, plugin execution, runtime materialization, and ledger\nmaterialization. Do not debug by adding credential handling to formatters,\nroutes, DB readers, frontend transforms, or harnesses. Inspect the typed\ntimeline and security routes to see what the live ledger recorded. For page or\narchive integrity, preserve a quiesced evidence copy and follow\n`\u002Fdev-session-debug`.\n\n**Frontend issues**: Run `just dev ui`, open Chrome DevTools, check console errors, use `take_screenshot` to capture state. See dev-testing-frontend for the full visual verification workflow.\n\n**Build pipeline issues**: Check `cache\u002Fcontainers\u002Flogs\u002Fbuild.log` -- all build infrastructure (runner, code signing, generation scripts) logs here. The runner (`build_system\u002Fpackaging\u002Fmacos\u002Frun_signed.sh`) and `_generate-settings` recipe both append to this file. Never write diagnostics to stdout from build scripts (it contaminates binary output like `mcp-export`).\n\nFor a Python gate or release failure, inspect the recorded run first. Step\nlabels, argv, timing, captured output, resource events, and the first failed\naction live under `cache\u002Ftarget\u002Fgate-runs\u002F`; `capsem-gate runs last --failed` is the\nsupported reader. `cache\u002Fcontainers\u002Flogs\u002Fbuild.log` is supporting build evidence, not the\ngate's execution ledger.\n\nDo not retry a release through its script, `capsem-admin release`, or a GitHub\nworkflow. The public command first accepts the exact qualification journal,\nthen runs prechecks, source publication, and dispatch as one graph. A direct\nscript or workflow skips the edges that make release safe.\n\n**Telemetry pipeline issues**: The canonical session ledgers (net_events, model_calls, tool_calls, tool_responses, fs_events, dns_events, security_rule_events) each have their own boundary. If a table is empty or has wrong data:\n- Check if the guest daemon started (boot logs)\n- Check if the vsock connection was accepted (host logs)\n- Check timing -- did the VM shut down before the DB-owned buffer flushed? Use\n  the DB flush barrier or shutdown\u002Freopen proof; do not add a sleep or poll.\n\nFor route latency or stale stats, do not add service-owned logged-data\nprojections. Logged-data hot state belongs inside the logger DB object as\ntable-level `mem`\u002Fdisk ownership with DB-layer tests and benchmarks. Service\nroutes may describe the query they need, but production service code must not\nopen rusqlite connections or `DbReader` directly.\n\nDo not \"fix\" route latency by hardcoding route-specific query helpers in\n`DbWriter`, by adding service caches, or by swallowing missing tables\u002Fcolumns as\nempty data. The correct diagnosis target is the DB object: connection\u002Fthread\nownership, `mem`\u002Fdisk layout, batching, flush, and query execution.\nIf the schema is missing, surface the broken ledger contract.\n\nWrite down what you find. The diagnosis should explain *why* the bug exists, not just *where* the symptom appears.\n\n## Concurrency flakes are product bugs, not test-tuning problems\n\n`just test` runs the python suite under `pytest -n 4 --dist=loadfile`. Four real VMs boot in parallel; this is dogfooding. Capsem ships as a multi-VM sandbox for AI agents -- if the test suite cannot safely run 4 concurrent VMs, real users running an agent farm will hit the same bug. When a test flakes only under concurrency, the diagnosis target is **Capsem's product code**, not the test:\n\n- \"Suspend timed out\" appearing only at `-n 4` -> `handle_suspend` IPC race; investigate the `with_quiescence` path and the `Suspend` round-trip, not the test timeout\n- \"Session did not become ready\" only with multiple parallel provisions -> Apple VZ resource contention, VirtioFS lock, or service handle_provision serialization gap\n- Two tests collide on the same VM\u002Fsession name -> `validate_vm_name` \u002F persistent registry has a TOCTOU; UUID prefix in the test is not the bug\n- \"Connection refused\" on a per-VM UDS only at `-n 4` -> service spawned the process but didn't wait for the socket to be bound; race in the spawn path\n- A test passes serial but hangs at n=4 -> a global lock somewhere (state mutex held across an await, blocking Tokio worker; or a sync `std::Mutex` on a hot path)\n\nAnti-patterns to avoid:\n- Adding `time.sleep` in the test \"to let things settle\"\n- Bumping a per-test timeout from 30s to 120s \"because it's flaky\"\n- Marking the test `serial` -- defeats the dogfooding signal\n- Adding retries with backoff in the client\n\nRight pattern: capture a service log of the failing run (set `RUST_LOG=capsem=trace`), find the operation that took unexpectedly long or returned an error, fix the underlying race in capsem-service \u002F capsem-process \u002F capsem-core. Then re-run at `-n 4` to confirm.\n\n## Step 2.5: Fix the pattern, not the instance\n\nFor long HTTP operations, trace the complete deadline chain: service work,\ngateway response headers, generated operation, SDK facade and UI caller. Image\npulls use the API-owned whole-operation budget and OpenAPI's\n`x-capsem-request-timeout-ms` default across all SDKs. An operation default\nextends a short transport default; an explicit per-call timeout still wins.\nTest ordinary calls, explicit deadlines, cancellation and absence of replay.\nDo not fix a gateway timeout by adding a longer timeout to one UI caller.\n\nWhen diagnosis reveals a **systemic pattern** (the same mistake repeated across the codebase), the fix must cover every instance -- not just the one that was reported.\n\n- **Audit the entire codebase for the same pattern.** If blocking I\u002FO in async context caused one hang, grep for every other site that does the same thing. A bug is a symptom -- the pattern is the disease.\n- **Never simplify a fix to the minimum diff.** A \"quick fix\" that patches one call site while 6 others have the identical problem is not a fix -- it's deferred breakage.\n- **Document the pattern in the relevant skill** (e.g., dev-rust-patterns) so it's never reintroduced.\n- **Add tests that would catch the pattern** if it recurs (e.g., a contract test between the frontend and backend response format).\n\nExample: an MCP tool hang was caused by blocking I\u002FO (clonefile, walkdir, blake3) on tokio worker threads. The same anti-pattern existed in 7 sibling tool handlers, a background timer, and asset hash verification. Fixing only the reported call would have left 9 other sites broken.\n\n## Step 3: Fix with a comprehensive solution\n\nNow that you understand the root cause, write the fix. The fix should:\n- Make your reproducing test pass\n- Not break any existing tests (`just test`)\n- Address the root cause, not just the symptom\n- Include the test from Step 1 in the same commit\n\nAfter the fix, run the full validation:\n1. `just test` -- unit + cross-compile + frontend\n2. `just exec \"capsem-doctor\"` -- VM smoke test\n3. If the bug touched telemetry: inspect the real session through typed routes;\n   run `check_session.py --db` only on a preserved, quiesced evidence copy\n\nA diagnostic continuation may shorten investigation before this validation;\nit never replaces any item in the final proof.\n\n## Local\u002FCI execution parity\n\nWhen a bug appears only in CI, first identify the exact production entrypoint,\nrunner dependency, architecture, environment variable, permission, device, or\nservice-manager difference that local testing skipped. Reproduce every portable\nLinux difference in Docker and execute the same production entrypoint or shared\npredicate that CI executes. A hand-written approximation is not a regression\ntest.\n\nKeep an executable parity test after the fix and audit sibling workflows for\nthe same one-sided assumption. If reproduction crosses an unavoidable platform\nboundary, document the boundary and preserve the nearest local contract plus\nthe required owning release-job or physical-machine proof. Do not relabel an\nunreproduced CI failure as transient.\n\n## What NOT to do\n\n- **Do not \"fix\" a failing test by changing the test assertion.** The test is telling you something. Listen to it. If the test is genuinely wrong, explain why in detail before changing it.\n- **Do not dismiss failures as \"pre-existing\" or \"unrelated.\"** Investigate every failure. If it truly is pre-existing, file it and fix it -- don't leave broken windows.\n- **Do not guess-and-check.** Random changes hoping something sticks waste time and often introduce new bugs. Understand first, then act.\n- **Do not patch symptoms.** If requests fail because gzip content-encoding isn't handled, don't strip the Accept-Encoding header -- implement proper decompression. Fix the system, not the surface.\n- **Do not apply narrow fixes to systemic problems.** If the same anti-pattern exists in 7 places and you fix 1, you haven't fixed the bug -- you've hidden 6 more. Audit first, then fix all instances in a single pass.\n","# Debugging\n\n## The rule\n\nNever fix code before you understand why it broke. The temptation to \"just make the test pass\" or \"just patch the symptom\" leads to fragile fixes that hide deeper problems. Follow the three-step workflow below every time.\n\n## Choose the cheapest truthful feedback loop\n\nKeep implementation moving. Use a minimal reproducer and focused TDD checks\nduring coding, then return to the next implementation item. Broad qualification\nruns belong at the end of a major phase; a small fix does not justify another\nwhole contract\u002Fworkspace\u002FVM\u002Finstall run. The bounded wrapper retains timeout\nand cleanup without a routine machine lease.\n\nCapsem now has two distinct execution surfaces:\n\n- Just exposes the small public product interface.\n- Python under `build_system\u002Fbuilder\u002Fgate\u002F` owns build\u002Ftest\u002Frelease plans, graph edges,\n  locking, resources, evidence, and cleanup.\n\nFor a gate failure, debug the Python step and action that failed; do not infer\nthe implementation from the Just recipe or recreate it with shell commands.\nStart with:\n\n```bash\nuv run --project build_system --frozen capsem-gate runs last --failed\nuv run --project build_system --frozen capsem-gate \u003Ccommand> --dry-run\nuv run --project build_system --frozen capsem-gate \u003Ccommand> --graph\n```\n\nUse the smallest focused pytest, cargo, pnpm, or script command for red\u002Fgreen\nwork. Use `just focus-test functional` for focused integration feedback, and\n`just fast-test` for the fast gate itself. Use `just test` when the forward fix\nis ready for the complete local verification a release requires.\n\nDirect diagnostics which may block, build, launch children, or wait on input\nmust use the portable bounded-process wrapper rather than a bare shell command:\n\n```bash\npython3 build_system\u002Fscripts\u002Fci\u002Frun-bounded-command.py --timeout-seconds 1800 -- \u003Ccommand> \u003Cargs...>\n```\n\nThe wrapper closes stdin, creates a dedicated process group, and terminates the\nwhole group on timeout or interruption. This prevents a PTY-backed `docker\nbuild -f -`, compiler, test runner, or child helper from surviving its owning\ndiagnostic. Pick a finite timeout appropriate to the focused operation; do not\nwrap `just test` or a release command, whose config-owned step timeouts,\njournal, teardown, and resumable graph are the authority.\n\n### Diagnostic continuation for a late gate failure\n\nWhen a clean candidate fails after expensive predecessors have succeeded, use\n**diagnostic continuation** to reuse that retained state and reach the frontier\nquickly. The current CLI retains legacy flag names:\n\n```bash\nuv run --project build_system --frozen capsem-gate runs last --failed\nuv run --project build_system --frozen capsem-gate candidate --prefix \u003Cretained-prefix> --from \u003Cfailed-step>\n```\n\nThe `--from` step runs; graph predecessors are reported as `carried`. Before\nusing it, match the prefix and failed step to the origin run, confirm the\nrequired predecessor really completed there, and record the origin run ID in\nthe diagnosis. Prefer a fresh focused test when the edit invalidates the\nexpensive producer itself.\n\nCarried steps with external runtime products are revalidated before resumed\nwork. If one was reclaimed, use the exact owning `--from` label in the refusal;\ndo not guess an earlier frontier or recreate the product by hand.\n\nCall the result `diagnostic-passed` or `diagnostic-failed`, even if the current\nimplementation still summarizes a zero exit as `ok`. It combines earlier\noutputs with current source and therefore answers only whether the new segment\ncan proceed. It does not prove the complete current tree.\n\nNever use diagnostic continuation with `release-binaries` or\n`release-assets`, and never let it authorize publication. After the fix, use\nthe smallest owning `focus-test` group. Before release, run `just test \u003Ccommit>`\nto success: the public release command requires its complete local journal for\nthat commit or an identical Git tree by default. An explicitly authorized\nrelease `force` permits a focused test-only correction without repeating the\ncomplete local run; source guards and full hosted qualification remain.\nThe unattended nightly scheduler is also exempt (`[release].unattended_channels` in\n[`config\u002Fgate.toml`](..\u002F..\u002Fconfig\u002Fgate.toml); see\n[`RELEASE.md`](..\u002F..\u002FRELEASE.md) section 8). The hosted lane then independently\nqualifies the artifact pairing it publishes.\n\n## Step 1: Reproduce with a test\n\nBefore touching any implementation code, write a test that captures the bug. This test must:\n- Fail right now, demonstrating the broken behavior\n- Be specific enough to distinguish the bug from correct behavior\n- Live in the right test location (see dev-testing for where tests go)\n\nIf you can't reproduce it in a test, you don't understand it well enough to fix it. For VM-level issues, use capsem-doctor or write a targeted diagnostic command:\n```bash\njust exec \"\u003Ccommand that triggers the bug>\"\n```\n\nFor telemetry issues, inspect a live session through the typed timeline,\nhistory, statistics, and security routes. Use `check_session.py --db` only on\na coherent, quiesced evidence copy after its ledger worker has stopped; see\n`\u002Fdev-session-debug`.\n\n## Step 2: Diagnose the root cause\n\nWith a failing test in hand, investigate. Do not skip this step. Common diagnostic approaches:\n\n**MCP triage trio (run FIRST when an investigation is open-ended):**\n\n```\ncapsem_panics { since: \"1h\" }       # any Rust panic in any host log? -> rank highest\ncapsem_triage { id: \"vm-1\" }        # ranked recent ipc-warns + 4xx\u002F5xx + slow_ops + session.db errors\ncapsem_timeline { id: \"vm-1\", trace_id: \"\u003CX>\" }   # follow ONE logical operation across exec\u002Ftool\u002Fnet\u002Ffs\u002Fmodel\n```\n\nThese read post-W2 JSON logs (`~\u002F.capsem\u002Frun\u002F{service,mcp,gateway,tray}.log` + capsem-app's latest jsonl) and post-W6 session.db tables. The W4 `target=fs op=fsync duration_ms=...` markers feed `capsem_triage`'s slow-op rank; the W3 schema_hash check appears in `capsem_panics` output as `IPC handshake failed; refusing connection` events. Always start with `capsem_panics` -- a single panic outranks a hundred warns.\n\n**Cross-version mix?** The `service.start` log line emits `protocol_version=N, schema_hash=\u003Chex>` per binary. If the support bundle (`capsem support-bundle`) shows two different schema_hash values across binaries, you're hitting the W3 cross-version-mix detection -- rebuild + restart the lagging binary.\n\n\n\n**Integration-test failures: read the preserved service log.** When any integration test fails, the test fixture (`tests\u002Fhelpers\u002Fservice.py::ServiceInstance`, the e2e `RealService`, and the MCP `_start_capsem_service`) archives its tmp_dir to `cache\u002Ftarget\u002Ftests\u002Fevidence\u002F\u003Ctimestamp>-\u003Cworker>-\u003Cnodeid>\u002F\u003Ctmp-basename>\u002F` **before** the usual rmtree. The failing test's stderr has the exact path: look for a line `ARTIFACT: preserved \u002Fvar\u002Ffolders\u002F... -> cache\u002Ftarget\u002Ftests\u002Fevidence\u002F...`. Inside that directory:\n\n```\nservice.log                     host-side capsem-service debug log (RUST_LOG=debug)\nlogs\u002Fgateway.log                gateway stdout\u002Fstderr\nlogs\u002Ftray.log                   tray stdout\u002Fstderr (if spawned)\nsessions\u002F\u003Cvm-id>\u002Fprocess.log    per-VM capsem-process log (vsock bridge, IPC, spawn chain)\nsessions\u002F\u003Cvm-id>\u002Fserial.log     VM serial console (kernel boot, capsem-init, agent startup)\nsessions\u002F\u003Cvm-id>\u002Fsession.db     SQLite telemetry DB (net_events, model_calls, ...)\npersistent\u002F\u003Cname>\u002F...           persistent-VM state (checkpoint.vzsave, workspace)\n```\n\n`cache\u002Ftarget\u002Ftests\u002Fevidence\u002F` is gitignored. Multiple failures sharing a session-scoped service land in different subdirs but the latest run's name tags them by the most recent failing nodeid. First place to look for \"VM didn't become exec-ready\" style failures: `sessions\u002F\u003Cid>\u002Fserial.log` (did the VM boot?) and `sessions\u002F\u003Cid>\u002Fprocess.log` (did the agent come up + IPC handshake?). For \"provision hung\" or service-side contention: `service.log`, grep for the VM id.\n\n**Rust code**: Read the code path the test exercises. Trace the data flow. Add `tracing` instrumentation if needed (`RUST_LOG=capsem=debug`). Check if the issue is in capsem-core, capsem-app, or capsem-agent.\n\n**Guest VM issues**: Boot with targeted commands and inspect behavior:\n```bash\njust exec \"capsem-doctor -k \u003Ccategory>\"   # Run specific diagnostic category\njust exec \"\u003Cmanual investigation command>\"\n```\nCheck boot logs for daemon startup failures, vsock connection issues, or timing problems.\n\n**Network\u002Fsecurity issues**: Check the network intercept path -- SNI parsing,\nHTTP\u002FDNS\u002Fmodel normalization, cert minting, `SecurityEvent` construction,\nsecurity rule evaluation, plugin execution, runtime materialization, and ledger\nmaterialization. Do not debug by adding credential handling to formatters,\nroutes, DB readers, frontend transforms, or harnesses. Inspect the typed\ntimeline and security routes to see what the live ledger recorded. For page or\narchive integrity, preserve a quiesced evidence copy and follow\n`\u002Fdev-session-debug`.\n\n**Frontend issues**: Run `just dev ui`, open Chrome DevTools, check console errors, use `take_screenshot` to capture state. See dev-testing-frontend for the full visual verification workflow.\n\n**Build pipeline issues**: Check `cache\u002Fcontainers\u002Flogs\u002Fbuild.log` -- all build infrastructure (runner, code signing, generation scripts) logs here. The runner (`build_system\u002Fpackaging\u002Fmacos\u002Frun_signed.sh`) and `_generate-settings` recipe both append to this file. Never write diagnostics to stdout from build scripts (it contaminates binary output like `mcp-export`).\n\nFor a Python gate or release failure, inspect the recorded run first. Step\nlabels, argv, timing, captured output, resource events, and the first failed\naction live under `cache\u002Ftarget\u002Fgate-runs\u002F`; `capsem-gate runs last --failed` is the\nsupported reader. `cache\u002Fcontainers\u002Flogs\u002Fbuild.log` is supporting build evidence, not the\ngate's execution ledger.\n\nDo not retry a release through its script, `capsem-admin release`, or a GitHub\nworkflow. The public command first accepts the exact qualification journal,\nthen runs prechecks, source publication, and dispatch as one graph. A direct\nscript or workflow skips the edges that make release safe.\n\n**Telemetry pipeline issues**: The canonical session ledgers (net_events, model_calls, tool_calls, tool_responses, fs_events, dns_events, security_rule_events) each have their own boundary. If a table is empty or has wrong data:\n- Check if the guest daemon started (boot logs)\n- Check if the vsock connection was accepted (host logs)\n- Check timing -- did the VM shut down before the DB-owned buffer flushed? Use\n  the DB flush barrier or shutdown\u002Freopen proof; do not add a sleep or poll.\n\nFor route latency or stale stats, do not add service-owned logged-data\nprojections. Logged-data hot state belongs inside the logger DB object as\ntable-level `mem`\u002Fdisk ownership with DB-layer tests and benchmarks. Service\nroutes may describe the query they need, but production service code must not\nopen rusqlite connections or `DbReader` directly.\n\nDo not \"fix\" route latency by hardcoding route-specific query helpers in\n`DbWriter`, by adding service caches, or by swallowing missing tables\u002Fcolumns as\nempty data. The correct diagnosis target is the DB object: connection\u002Fthread\nownership, `mem`\u002Fdisk layout, batching, flush, and query execution.\nIf the schema is missing, surface the broken ledger contract.\n\nWrite down what you find. The diagnosis should explain *why* the bug exists, not just *where* the symptom appears.\n\n## Concurrency flakes are product bugs, not test-tuning problems\n\n`just test` runs the python suite under `pytest -n 4 --dist=loadfile`. Four real VMs boot in parallel; this is dogfooding. Capsem ships as a multi-VM sandbox for AI agents -- if the test suite cannot safely run 4 concurrent VMs, real users running an agent farm will hit the same bug. When a test flakes only under concurrency, the diagnosis target is **Capsem's product code**, not the test:\n\n- \"Suspend timed out\" appearing only at `-n 4` -> `handle_suspend` IPC race; investigate the `with_quiescence` path and the `Suspend` round-trip, not the test timeout\n- \"Session did not become ready\" only with multiple parallel provisions -> Apple VZ resource contention, VirtioFS lock, or service handle_provision serialization gap\n- Two tests collide on the same VM\u002Fsession name -> `validate_vm_name` \u002F persistent registry has a TOCTOU; UUID prefix in the test is not the bug\n- \"Connection refused\" on a per-VM UDS only at `-n 4` -> service spawned the process but didn't wait for the socket to be bound; race in the spawn path\n- A test passes serial but hangs at n=4 -> a global lock somewhere (state mutex held across an await, blocking Tokio worker; or a sync `std::Mutex` on a hot path)\n\nAnti-patterns to avoid:\n- Adding `time.sleep` in the test \"to let things settle\"\n- Bumping a per-test timeout from 30s to 120s \"because it's flaky\"\n- Marking the test `serial` -- defeats the dogfooding signal\n- Adding retries with backoff in the client\n\nRight pattern: capture a service log of the failing run (set `RUST_LOG=capsem=trace`), find the operation that took unexpectedly long or returned an error, fix the underlying race in capsem-service \u002F capsem-process \u002F capsem-core. Then re-run at `-n 4` to confirm.\n\n## Step 2.5: Fix the pattern, not the instance\n\nFor long HTTP operations, trace the complete deadline chain: service work,\ngateway response headers, generated operation, SDK facade and UI caller. Image\npulls use the API-owned whole-operation budget and OpenAPI's\n`x-capsem-request-timeout-ms` default across all SDKs. An operation default\nextends a short transport default; an explicit per-call timeout still wins.\nTest ordinary calls, explicit deadlines, cancellation and absence of replay.\nDo not fix a gateway timeout by adding a longer timeout to one UI caller.\n\nWhen diagnosis reveals a **systemic pattern** (the same mistake repeated across the codebase), the fix must cover every instance -- not just the one that was reported.\n\n- **Audit the entire codebase for the same pattern.** If blocking I\u002FO in async context caused one hang, grep for every other site that does the same thing. A bug is a symptom -- the pattern is the disease.\n- **Never simplify a fix to the minimum diff.** A \"quick fix\" that patches one call site while 6 others have the identical problem is not a fix -- it's deferred breakage.\n- **Document the pattern in the relevant skill** (e.g., dev-rust-patterns) so it's never reintroduced.\n- **Add tests that would catch the pattern** if it recurs (e.g., a contract test between the frontend and backend response format).\n\nExample: an MCP tool hang was caused by blocking I\u002FO (clonefile, walkdir, blake3) on tokio worker threads. The same anti-pattern existed in 7 sibling tool handlers, a background timer, and asset hash verification. Fixing only the reported call would have left 9 other sites broken.\n\n## Step 3: Fix with a comprehensive solution\n\nNow that you understand the root cause, write the fix. The fix should:\n- Make your reproducing test pass\n- Not break any existing tests (`just test`)\n- Address the root cause, not just the symptom\n- Include the test from Step 1 in the same commit\n\nAfter the fix, run the full validation:\n1. `just test` -- unit + cross-compile + frontend\n2. `just exec \"capsem-doctor\"` -- VM smoke test\n3. If the bug touched telemetry: inspect the real session through typed routes;\n   run `check_session.py --db` only on a preserved, quiesced evidence copy\n\nA diagnostic continuation may shorten investigation before this validation;\nit never replaces any item in the final proof.\n\n## Local\u002FCI execution parity\n\nWhen a bug appears only in CI, first identify the exact production entrypoint,\nrunner dependency, architecture, environment variable, permission, device, or\nservice-manager difference that local testing skipped. Reproduce every portable\nLinux difference in Docker and execute the same production entrypoint or shared\npredicate that CI executes. A hand-written approximation is not a regression\ntest.\n\nKeep an executable parity test after the fix and audit sibling workflows for\nthe same one-sided assumption. If reproduction crosses an unavoidable platform\nboundary, document the boundary and preserve the nearest local contract plus\nthe required owning release-job or physical-machine proof. Do not relabel an\nunreproduced CI failure as transient.\n\n## What NOT to do\n\n- **Do not \"fix\" a failing test by changing the test assertion.** The test is telling you something. Listen to it. If the test is genuinely wrong, explain why in detail before changing it.\n- **Do not dismiss failures as \"pre-existing\" or \"unrelated.\"** Investigate every failure. If it truly is pre-existing, file it and fix it -- don't leave broken windows.\n- **Do not guess-and-check.** Random changes hoping something sticks waste time and often introduce new bugs. Understand first, then act.\n- **Do not patch symptoms.** If requests fail because gzip content-encoding isn't handled, don't strip the Accept-Encoding header -- implement proper decompression. Fix the system, not the surface.\n- **Do not apply narrow fixes to systemic problems.** If the same anti-pattern exists in 7 places and you fix 1, you haven't fixed the bug -- you've hidden 6 more. Audit first, then fix all instances in a single pass.\n",{},"main","d357170bb677125c80cebed720e62bf47bf083ad","skills\u002Fdev-debugging",26,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fcapsem","https:\u002F\u002Fgithub.com\u002Fgoogle","https:\u002F\u002Fopensource.google\u002F","Google ❤️ Open Source","2026-11-04 04:36:10",{"source":7,"skillName":5,"commands":34,"prompt":44,"zipAvailable":45},[35,38,41],{"tool":36,"command":37},"npx","npx skills add google\u002Fcapsem --skill dev-debugging",{"tool":39,"command":40},"bunx","bunx skills add google\u002Fcapsem --skill dev-debugging",{"tool":42,"command":43},"pnpm","pnpm dlx skills add google\u002Fcapsem --skill dev-debugging","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fcapsem\u002Fblob\u002Fmain\u002Fskills\u002Fdev-debugging\u002FSKILL.md",true,[],[48,62,78,87,94,101],{"skillNo":49,"name":50,"nameZh":6,"description":51,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":52,"categories":53,"keywords":59,"publishedAt":19},"01M699558T2T4DJMBN08ZCQ79K","use-capsem","Run tasks in Capsem VMs through MCP. Use when an agent needs isolated execution, file analysis, or reports returned to the host.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fcapsem\u002Fblob\u002Fmain\u002Fskills\u002Fuse-capsem\u002FSKILL.md",[54],{"categoryNo":55,"categoryName":56,"taxonomyType":57,"taxonomyTypeDesc":58},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",[60,61],"use capsem","MCP",{"skillNo":63,"name":64,"nameZh":6,"description":65,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":66,"categories":67,"keywords":76,"publishedAt":19},"01M69953Z1CSVQ4N07QX529CZY","site-marketing","The marketing website, capsem.org. Use when editing copy, adding sections, or changing the site theme.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fcapsem\u002Fblob\u002Fmain\u002Fskills\u002Fsite-marketing\u002FSKILL.md",[68,71],{"categoryNo":69,"categoryName":70,"taxonomyType":57,"taxonomyTypeDesc":58},"01JBSEEDCAT000000000000019","营销推广",{"categoryNo":72,"categoryName":73,"taxonomyType":74,"taxonomyTypeDesc":75},"01JBSEEDCAT000000000000035","市场营销","PROFESSION","适用职业",[77],"site marketing",{"skillNo":79,"name":80,"nameZh":6,"description":81,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":82,"categories":83,"keywords":84,"publishedAt":19},"01M69952B33KBBV37N8QTC4191","site-infra","The documentation site, docs.capsem.org. Use for user-directed Diátaxis documentation and Astro Starlight changes.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fcapsem\u002Fblob\u002Fmain\u002Fskills\u002Fsite-infra\u002FSKILL.md",[],[85,86],"site infra","docs",{"skillNo":88,"name":89,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":90,"categories":91,"keywords":92,"publishedAt":19},"01M69950YKCZKCDB98JDQNBBRJ","site-architecture","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fcapsem\u002Fblob\u002Fmain\u002Fskills\u002Fsite-architecture\u002FSKILL.md",[],[93],"site architecture",{"skillNo":95,"name":96,"nameZh":6,"description":6,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":97,"categories":98,"keywords":99,"publishedAt":19},"01M6994Z61JWZ0G2YRJTWFH933","release-process","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fcapsem\u002Fblob\u002Fmain\u002Fskills\u002Frelease-process\u002FSKILL.md",[],[100],"release process",{"skillNo":102,"name":103,"nameZh":6,"description":104,"descriptionZh":6,"repositoryFullName":7,"ownerLogin":8,"ownerAvatarUrl":9,"ownerAccountNo":10,"ownerDisplayName":11,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":105,"categories":106,"keywords":114,"publishedAt":19},"01M6994XKHADM6FBTYTAPG7FYC","meta-skill-creation","Create, improve, and evaluate skills. Use when writing a new skill, editing an existing one, or tuning a description for better triggering.","https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fcapsem\u002Fblob\u002Fmain\u002Fskills\u002Fmeta-skill-creation\u002FSKILL.md",[107,110,111],{"categoryNo":108,"categoryName":109,"taxonomyType":57,"taxonomyTypeDesc":58},"01JBSEEDCAT000000000000018","写作与文案",{"categoryNo":55,"categoryName":56,"taxonomyType":57,"taxonomyTypeDesc":58},{"categoryNo":112,"categoryName":113,"taxonomyType":74,"taxonomyTypeDesc":75},"01JBSEEDCAT000000000000036","内容创作者",[115,116],"meta skill creation","writing"]