[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M69A76WHY669EFTAKP9STKQ2":3,"skill-same:01M69A76WHY669EFTAKP9STKQ2":76},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":23,"publishedAt":38,"linkOnly":39,"markdownContent":40,"markdownBody":41,"frontMatter":42,"refName":43,"commitSha":44,"skillDirectory":45,"forksCount":46,"repositoryHtmlUrl":47,"repositoryHomepageUrl":6,"ownerHtmlUrl":48,"ownerBlogUrl":49,"ownerDescription":50,"updatedAt":51,"install":52,"resources":65},"01M69A76WHY669EFTAKP9STKQ2","debug-orchestration",null,"Diagnose and fix nondeterminism errors in the duroxide orchestration. Covers the replay model, common violations, and how to audit orchestration code for determinism.","microsoft\u002FPilotSwarm","microsoft","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F6154722?v=4","01M698VYDXZ6JW5WH3CRV650R8","Microsoft","01M69A6RNK492A5JSNBTXANKX5",15,"MIT","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FPilotSwarm\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Fdebug-orchestration\u002FSKILL.md",[18],{"categoryNo":19,"categoryName":20,"taxonomyType":21,"taxonomyTypeDesc":22},"01JBSEEDCAT000000000000030","开发工程师","PROFESSION","适用职业",[24,25,26,27,28,29,30,31,32,33,34,35,36,37],"orchestration","debug","debug orchestration","diagnose","code","audit","how","model","common","fix","replay","how to","errors","for","2026-11-10 10:12:30",false,"---\nname: debug-orchestration\ndescription: Diagnose and fix nondeterminism errors in the duroxide orchestration. Covers the replay model, common violations, and how to audit orchestration code for determinism.\n---\n\n# Debug Orchestration Nondeterminism\n\nIf the issue is about live worker routing, session-affined activity placement, hydration\u002Fdehydration traces, or cross-pod runtime evidence rather than replay-safe code shape, use the sibling skill [`.github\u002Fskills\u002Finvestigate-duroxide-runtime\u002FSKILL.md`](..\u002Finvestigate-duroxide-runtime\u002FSKILL.md).\n\nDuroxide replays the orchestration generator from the beginning on every new event. The generator must produce the **exact same sequence of yielded actions** during replay as during original execution. A mismatch causes:\n\n```\nnondeterministic: custom status mismatch: action=CallActivity { ... } vs event=CustomStatusUpdated { ... }\n```\n\n## How replay works\n\n1. Duroxide stores every action (yield) result in a history log.\n2. On replay, it feeds stored results back into the generator at each `yield`.\n3. It compares each yielded action descriptor with the stored history event.\n4. If they don't match → **nondeterminism error** → orchestration fails.\n\n`setCustomStatus()` is **also recorded** in history as `CustomStatusUpdated`. It's fire-and-forget (no yield), but its position relative to yields matters.\n\n## Diagnosis steps\n\n1. **Read the error message carefully.** It tells you:\n   - What the orchestration **tried to do** (`action=CallActivity { name: \"X\" }`)\n   - What the history **expected** (`event=CustomStatusUpdated { ... }`)\n\n2. **Find the divergence point.** The orchestration took a different code path during replay. Look for:\n   - `Date.now()` used in a condition before a yield or `setCustomStatus()`\n   - `Math.random()` or `crypto.randomUUID()` in control flow\n   - Any I\u002FO (fetch, fs, database) done directly in the generator\n   - `setCustomStatus()` that moved relative to a yield\n\n3. **Trace the yield sequence.** Number every `yield` and `setCustomStatus()` call in `orchestration.ts`. During replay, the Nth yield must match the Nth history event.\n\n## Common bugs and fixes\n\n| Bug | Fix |\n|-----|-----|\n| `Date.now()` in condition before yield | Use `yield ctx.utcNow()` — deterministic, replay-safe |\n| `Math.random()` for IDs | Use `yield ctx.newGuid()` |\n| Direct I\u002FO in generator | Wrap in an activity (`ctx.scheduleActivity()`) |\n| `setTimeout` \u002F `setInterval` | Use `yield ctx.scheduleTimer(ms)` |\n| `setCustomStatus()` order changed | Ensure it appears at the same position relative to yields |\n| Redeployed with changed yields | Freeze the released handler and register a new orchestration version. Deploys never reset the database. |\n\n## Deterministic alternatives\n\n| Non-deterministic | Deterministic equivalent |\n|-------------------|--------------------------|\n| `Date.now()` | `yield ctx.utcNow()` |\n| `new Date()` | `yield ctx.utcNow()` (returns ms, wrap with `new Date(ms)` if needed) |\n| `Math.random()` | `yield ctx.newGuid()` |\n| `crypto.randomUUID()` | `yield ctx.newGuid()` |\n| `setTimeout(fn, ms)` | `yield ctx.scheduleTimer(ms)` |\n| `fetch()` \u002F any I\u002FO | `yield ctx.scheduleActivity(\"name\", input)` |\n\n## The `setCustomStatus` trap\n\n`setCustomStatus()` doesn't require a yield, but duroxide records it as a `CustomStatusUpdated` event in the history. This means:\n\n```typescript\n\u002F\u002F Original execution: condition is false → no activity yield → setCustomStatus\nif (Date.now() > threshold) {           \u002F\u002F ← non-deterministic!\n    yield someActivity();               \u002F\u002F recorded as CallActivity\n}\nsetStatus(\"idle\");                      \u002F\u002F recorded as CustomStatusUpdated\n\n\u002F\u002F Replay: condition is NOW true → activity yield → mismatch!\n\u002F\u002F Replay expects CustomStatusUpdated but gets CallActivity\n```\n\nThe fix: use `yield ctx.utcNow()` for any time-based branching.\n\n## Runaway \u002F leaky deployments\n\nA very common cause of nondeterminism is **old workers from a previous deployment still running** in the same cluster, connected to the same database. This happens when:\n\n- A previous deployment used a different Kubernetes namespace (e.g. `copilot-sdk` vs `copilot-runtime`) and was never cleaned up.\n- A rolling update left old ReplicaSet pods running alongside new ones during the transition.\n- Manual `kubectl run` or port-forward sessions left orphaned pods polling the same database.\n\nOld workers run **old orchestration code** but process orchestrations that were started (or replayed) by the **new code**. The yield sequences differ → nondeterminism error.\n\n### Diagnosis\n\n```bash\n# Check ALL namespaces for worker pods — not just the expected one\nkubectl get pods --all-namespaces -l app.kubernetes.io\u002Fcomponent=worker --no-headers\n```\n\nIf you see pods in an unexpected namespace or from a different ReplicaSet, that's the culprit.\n\n### Fix\n\n```bash\n# Delete the old deployment and namespace\nkubectl delete deployment copilot-runtime-worker -n \u003Cold-namespace>\nkubectl delete namespace \u003Cold-namespace> --wait=false\n\n# Reset the database to clear any tainted history\nNODE_TLS_REJECT_UNAUTHORIZED=0 node --env-file=.env.remote scripts\u002Fdb-reset.js --yes\n\n# Restart current workers\nkubectl rollout restart deployment\u002Fcopilot-runtime-worker -n copilot-runtime\n```\n\n## After fixing\n\n1. **Reset the database** — existing orchestrations have stale history:\n   ```bash\n   NODE_TLS_REJECT_UNAUTHORIZED=0 node --env-file=.env.remote scripts\u002Fdb-reset.js --yes\n   ```\n2. **Redeploy** — or use the full deploy script:\n   ```bash\n   .\u002Fscripts\u002Fdeploy-aks.sh\n   ```\n\n## Key files\n- [src\u002Forchestration.ts](..\u002F..\u002F..\u002Fsrc\u002Forchestration.ts) — the orchestration generator (audit this for determinism)\n- [src\u002Fsession-proxy.ts](..\u002F..\u002F..\u002Fsrc\u002Fsession-proxy.ts) — activity definitions (safe — activities run outside replay)\n- [scripts\u002Fdeploy-aks.sh](..\u002F..\u002F..\u002Fscripts\u002Fdeploy-aks.sh) — deploy with DB reset\n- [scripts\u002Fdb-reset.js](..\u002F..\u002F..\u002Fscripts\u002Fdb-reset.js) — standalone DB reset\n","# Debug Orchestration Nondeterminism\n\nIf the issue is about live worker routing, session-affined activity placement, hydration\u002Fdehydration traces, or cross-pod runtime evidence rather than replay-safe code shape, use the sibling skill [`.github\u002Fskills\u002Finvestigate-duroxide-runtime\u002FSKILL.md`](..\u002Finvestigate-duroxide-runtime\u002FSKILL.md).\n\nDuroxide replays the orchestration generator from the beginning on every new event. The generator must produce the **exact same sequence of yielded actions** during replay as during original execution. A mismatch causes:\n\n```\nnondeterministic: custom status mismatch: action=CallActivity { ... } vs event=CustomStatusUpdated { ... }\n```\n\n## How replay works\n\n1. Duroxide stores every action (yield) result in a history log.\n2. On replay, it feeds stored results back into the generator at each `yield`.\n3. It compares each yielded action descriptor with the stored history event.\n4. If they don't match → **nondeterminism error** → orchestration fails.\n\n`setCustomStatus()` is **also recorded** in history as `CustomStatusUpdated`. It's fire-and-forget (no yield), but its position relative to yields matters.\n\n## Diagnosis steps\n\n1. **Read the error message carefully.** It tells you:\n   - What the orchestration **tried to do** (`action=CallActivity { name: \"X\" }`)\n   - What the history **expected** (`event=CustomStatusUpdated { ... }`)\n\n2. **Find the divergence point.** The orchestration took a different code path during replay. Look for:\n   - `Date.now()` used in a condition before a yield or `setCustomStatus()`\n   - `Math.random()` or `crypto.randomUUID()` in control flow\n   - Any I\u002FO (fetch, fs, database) done directly in the generator\n   - `setCustomStatus()` that moved relative to a yield\n\n3. **Trace the yield sequence.** Number every `yield` and `setCustomStatus()` call in `orchestration.ts`. During replay, the Nth yield must match the Nth history event.\n\n## Common bugs and fixes\n\n| Bug | Fix |\n|-----|-----|\n| `Date.now()` in condition before yield | Use `yield ctx.utcNow()` — deterministic, replay-safe |\n| `Math.random()` for IDs | Use `yield ctx.newGuid()` |\n| Direct I\u002FO in generator | Wrap in an activity (`ctx.scheduleActivity()`) |\n| `setTimeout` \u002F `setInterval` | Use `yield ctx.scheduleTimer(ms)` |\n| `setCustomStatus()` order changed | Ensure it appears at the same position relative to yields |\n| Redeployed with changed yields | Freeze the released handler and register a new orchestration version. Deploys never reset the database. |\n\n## Deterministic alternatives\n\n| Non-deterministic | Deterministic equivalent |\n|-------------------|--------------------------|\n| `Date.now()` | `yield ctx.utcNow()` |\n| `new Date()` | `yield ctx.utcNow()` (returns ms, wrap with `new Date(ms)` if needed) |\n| `Math.random()` | `yield ctx.newGuid()` |\n| `crypto.randomUUID()` | `yield ctx.newGuid()` |\n| `setTimeout(fn, ms)` | `yield ctx.scheduleTimer(ms)` |\n| `fetch()` \u002F any I\u002FO | `yield ctx.scheduleActivity(\"name\", input)` |\n\n## The `setCustomStatus` trap\n\n`setCustomStatus()` doesn't require a yield, but duroxide records it as a `CustomStatusUpdated` event in the history. This means:\n\n```typescript\n\u002F\u002F Original execution: condition is false → no activity yield → setCustomStatus\nif (Date.now() > threshold) {           \u002F\u002F ← non-deterministic!\n    yield someActivity();               \u002F\u002F recorded as CallActivity\n}\nsetStatus(\"idle\");                      \u002F\u002F recorded as CustomStatusUpdated\n\n\u002F\u002F Replay: condition is NOW true → activity yield → mismatch!\n\u002F\u002F Replay expects CustomStatusUpdated but gets CallActivity\n```\n\nThe fix: use `yield ctx.utcNow()` for any time-based branching.\n\n## Runaway \u002F leaky deployments\n\nA very common cause of nondeterminism is **old workers from a previous deployment still running** in the same cluster, connected to the same database. This happens when:\n\n- A previous deployment used a different Kubernetes namespace (e.g. `copilot-sdk` vs `copilot-runtime`) and was never cleaned up.\n- A rolling update left old ReplicaSet pods running alongside new ones during the transition.\n- Manual `kubectl run` or port-forward sessions left orphaned pods polling the same database.\n\nOld workers run **old orchestration code** but process orchestrations that were started (or replayed) by the **new code**. The yield sequences differ → nondeterminism error.\n\n### Diagnosis\n\n```bash\n# Check ALL namespaces for worker pods — not just the expected one\nkubectl get pods --all-namespaces -l app.kubernetes.io\u002Fcomponent=worker --no-headers\n```\n\nIf you see pods in an unexpected namespace or from a different ReplicaSet, that's the culprit.\n\n### Fix\n\n```bash\n# Delete the old deployment and namespace\nkubectl delete deployment copilot-runtime-worker -n \u003Cold-namespace>\nkubectl delete namespace \u003Cold-namespace> --wait=false\n\n# Reset the database to clear any tainted history\nNODE_TLS_REJECT_UNAUTHORIZED=0 node --env-file=.env.remote scripts\u002Fdb-reset.js --yes\n\n# Restart current workers\nkubectl rollout restart deployment\u002Fcopilot-runtime-worker -n copilot-runtime\n```\n\n## After fixing\n\n1. **Reset the database** — existing orchestrations have stale history:\n   ```bash\n   NODE_TLS_REJECT_UNAUTHORIZED=0 node --env-file=.env.remote scripts\u002Fdb-reset.js --yes\n   ```\n2. **Redeploy** — or use the full deploy script:\n   ```bash\n   .\u002Fscripts\u002Fdeploy-aks.sh\n   ```\n\n## Key files\n- [src\u002Forchestration.ts](..\u002F..\u002F..\u002Fsrc\u002Forchestration.ts) — the orchestration generator (audit this for determinism)\n- [src\u002Fsession-proxy.ts](..\u002F..\u002F..\u002Fsrc\u002Fsession-proxy.ts) — activity definitions (safe — activities run outside replay)\n- [scripts\u002Fdeploy-aks.sh](..\u002F..\u002F..\u002Fscripts\u002Fdeploy-aks.sh) — deploy with DB reset\n- [scripts\u002Fdb-reset.js](..\u002F..\u002F..\u002Fscripts\u002Fdb-reset.js) — standalone DB reset\n",{"name":5,"description":7},"main","cecafe1ca7a9f2778a6166abdd12f6165d99f777",".github\u002Fskills\u002Fdebug-orchestration",9,"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FPilotSwarm","https:\u002F\u002Fgithub.com\u002Fmicrosoft","https:\u002F\u002Fopensource.microsoft.com","Open source projects and samples from Microsoft","2026-11-12 03:13:58",{"source":8,"skillName":5,"commands":53,"prompt":63,"zipAvailable":64},[54,57,60],{"tool":55,"command":56},"npx","npx skills add microsoft\u002FPilotSwarm --skill debug-orchestration",{"tool":58,"command":59},"bunx","bunx skills add microsoft\u002FPilotSwarm --skill debug-orchestration",{"tool":61,"command":62},"pnpm","pnpm dlx skills add microsoft\u002FPilotSwarm --skill debug-orchestration","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FPilotSwarm\u002Fblob\u002Fmain\u002F.github\u002Fskills\u002Fdebug-orchestration\u002FSKILL.md",true,[66,72],{"path":67,"resourceType":68,"resourceTypeDesc":69,"sizeBytes":70,"sourceHtmlUrl":71},"agents\u002Fopenai.yaml","REFERENCE","参考文档",248,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fxrblocks\u002Fblob\u002Fmain\u002Fskills\u002Fxb-debug-app\u002Fagents\u002Fopenai.yaml",{"path":73,"resourceType":68,"resourceTypeDesc":69,"sizeBytes":74,"sourceHtmlUrl":75},"references\u002Fdiagnostic-branches.md",9352,"https:\u002F\u002Fgithub.com\u002Fgoogle\u002Fxrblocks\u002Fblob\u002Fmain\u002Fskills\u002Fxb-debug-app\u002Freferences\u002Fdiagnostic-branches.md",[77,98,122,137,164,187],{"skillNo":78,"name":79,"nameZh":6,"description":80,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":81,"categories":82,"keywords":83,"publishedAt":38},"01M69A9CY8CXHED3AFC8DAT9G0","pilotswarm-three-tier","Reference for adding a Tier 3 worker AKS cluster to a PilotSwarm deployment. Covers provisioning, identity, Dockerfile changes, entrypoint, manifests, and verification.","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FPilotSwarm\u002Fblob\u002Fmain\u002Ftemplates\u002Fbuilder-agents\u002Fskills\u002Fpilotswarm-three-tier\u002FSKILL.md",[],[84,85,86,87,37,88,89,90,91,92,93,94,95,96,97],"pilotswarm three tier","three","entrypoint","manifests","changes","aks","verification","deployment","reference","dockerfile","identity","provisioning","worker","cluster",{"skillNo":99,"name":100,"nameZh":6,"description":101,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":102,"categories":103,"keywords":104,"publishedAt":38},"01M69A9B6R5H88HJFEYC0P4R57","pilotswarm-sdk-builder","Use when creating or updating an SDK-first app on top of PilotSwarm. Covers the client\u002Fworker split, plugin layering, tool registration, tests, and the DevOps sample structure.","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FPilotSwarm\u002Fblob\u002Fmain\u002Ftemplates\u002Fbuilder-agents\u002Fskills\u002Fpilotswarm-sdk-builder\u002FSKILL.md",[],[105,106,107,108,109,110,111,112,113,114,115,116,117,96,118,119,120,121],"sdk","pilotswarm sdk builder","builder","tool","first","plugin","use","client","when","top","split","app","tests","sample","structure","registration","devops",{"skillNo":123,"name":124,"nameZh":6,"description":125,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":126,"categories":127,"keywords":128,"publishedAt":38},"01M69A99A43M97VTYZBCN1JQ1H","pilotswarm-portal-builder","Use when creating or updating a browser portal on top of PilotSwarm. Covers portal branding, plugin.json.portal, named-agent exposure, auth add-ons, and remote deployment wiring.","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FPilotSwarm\u002Fblob\u002Fmain\u002Ftemplates\u002Fbuilder-agents\u002Fskills\u002Fpilotswarm-portal-builder\u002FSKILL.md",[],[129,107,130,113,131,114,132,133,111,134,135,91,110,136],"pilotswarm portal builder","remote","wiring","branding","add","json","browser","auth",{"skillNo":138,"name":139,"nameZh":6,"description":140,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":141,"categories":142,"keywords":148,"publishedAt":38},"01M69A97JZY10V1VE12NF4JR5A","pilotswarm-knowledge-harvester","Use when an SDK or CLI app needs to ingest external sources into durable, searchable knowledge and an open knowledge graph. Covers enabling the EnhancedFactStore + GraphStore providers, authoring a crawler:true agent, the crawl→graph→reader flow, ACL\u002FscopeKey evidence, and tests.","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FPilotSwarm\u002Fblob\u002Fmain\u002Ftemplates\u002Fbuilder-agents\u002Fskills\u002Fpilotswarm-knowledge-harvester\u002FSKILL.md",[143],{"categoryNo":144,"categoryName":145,"taxonomyType":146,"taxonomyTypeDesc":147},"01JBSEEDCAT000000000000021","资料检索与研究","SCENARIO","使用场景",[149,150,111,113,151,152,153,154,116,155,117,156,157,158,159,160,161,162,105,163],"knowledge","pilotswarm knowledge harvester","cli app","true","crawler","sources","authoring","crawl","ingest","evidence","reader","open","flow","knowledge graph","cli",{"skillNo":165,"name":166,"nameZh":6,"description":167,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":168,"categories":169,"keywords":173,"publishedAt":38},"01M69A95Y3EWHSZ1TDFMXWAKX3","pilotswarm-hybrid-datastore","Use when a PilotSwarm app needs stock PostgreSQL for runtime storage plus HorizonDB for enhanced facts, semantic\u002Flexical search, and the knowledge graph. Covers env topology, horizonConfigFromEnv wiring, harvester agents, Azure secret\u002Fresource guidance, and the rule that Docker defaults remain stock Postgres.","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FPilotSwarm\u002Fblob\u002Fmain\u002Ftemplates\u002Fbuilder-agents\u002Fskills\u002Fpilotswarm-hybrid-datastore\u002FSKILL.md",[170],{"categoryNo":171,"categoryName":172,"taxonomyType":146,"taxonomyTypeDesc":147},"01JBSEEDCAT000000000000011","运维与部署",[174,175,176,116,177,178,179,180,111,149,162,181,182,183,184,185,113,131,37,186],"hybrid","pilotswarm hybrid datastore","storage","runtime","stock","postgres","guidance","search","postgresql","azure","env","docker","facts",{"skillNo":188,"name":189,"nameZh":6,"description":190,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":191,"categories":192,"keywords":193,"publishedAt":38},"01M69A94F4WR5CXDZ73N658DV0","pilotswarm-duroxide-versioning","Use when changing PilotSwarm durable orchestration behavior, continue-as-new state, or replay-sensitive control flow.","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002FPilotSwarm\u002Fblob\u002Fmain\u002Ftemplates\u002Fbuilder-agents\u002Fskills\u002Fpilotswarm-duroxide-versioning\u002FSKILL.md",[],[194,195,24,196,113,197,198,34,111,199,200,161,201],"pilotswarm duroxide versioning","versioning","new","as","control","behavior","continue","state"]