debug-orchestration

Debug Orchestration Nondeterminism

If the issue is about live worker routing, session-affined activity placement, hydration/dehydration traces, or cross-pod runtime evidence rather than replay-safe code shape, use the sibling skill .github/skills/investigate-duroxide-runtime/SKILL.md.

Duroxide replays the orchestration generator from the beginning on every new event. The generator must produce the exact same sequence of yielded actions during replay as during original execution. A mismatch causes:

nondeterministic: custom status mismatch: action=CallActivity { ... } vs event=CustomStatusUpdated { ... }

How replay works

1. Duroxide stores every action (yield) result in a history log.
2. On replay, it feeds stored results back into the generator at each yield.
3. It compares each yielded action descriptor with the stored history event.
4. If they don't match → nondeterminism error → orchestration fails.

setCustomStatus() is also recorded in history as CustomStatusUpdated. It's fire-and-forget (no yield), but its position relative to yields matters.

Diagnosis steps

1. Read the error message carefully. It tells you:

   • What the orchestration tried to do (action=CallActivity { name: "X" })
   • What the history expected (event=CustomStatusUpdated { ... })

2. Find the divergence point. The orchestration took a different code path during replay. Look for:

   • Date.now() used in a condition before a yield or setCustomStatus()
   • Math.random() or crypto.randomUUID() in control flow
   • Any I/O (fetch, fs, database) done directly in the generator
   • setCustomStatus() that moved relative to a yield

3. Trace the yield sequence. Number every yield and setCustomStatus() call in orchestration.ts. During replay, the Nth yield must match the Nth history event.

Common bugs and fixes

Bug	Fix
Date.now() in condition before yield	Use yield ctx.utcNow() — deterministic, replay-safe
Math.random() for IDs	Use yield ctx.newGuid()
Direct I/O in generator	Wrap in an activity (ctx.scheduleActivity())
setTimeout / setInterval	Use yield ctx.scheduleTimer(ms)
setCustomStatus() order changed	Ensure it appears at the same position relative to yields
Redeployed with changed yields	Freeze the released handler and register a new orchestration version. Deploys never reset the database.

Deterministic alternatives

Non-deterministic	Deterministic equivalent
Date.now()	yield ctx.utcNow()
new Date()	yield ctx.utcNow() (returns ms, wrap with new Date(ms) if needed)
Math.random()	yield ctx.newGuid()
crypto.randomUUID()	yield ctx.newGuid()
setTimeout(fn, ms)	yield ctx.scheduleTimer(ms)
fetch() / any I/O	yield ctx.scheduleActivity("name", input)

The setCustomStatus trap

setCustomStatus() doesn't require a yield, but duroxide records it as a CustomStatusUpdated event in the history. This means:

// Original execution: condition is false → no activity yield → setCustomStatus
if (Date.now() > threshold) {           // ← non-deterministic!
    yield someActivity();               // recorded as CallActivity
}
setStatus("idle");                      // recorded as CustomStatusUpdated

// Replay: condition is NOW true → activity yield → mismatch!
// Replay expects CustomStatusUpdated but gets CallActivity

The fix: use yield ctx.utcNow() for any time-based branching.

Runaway / leaky deployments

A very common cause of nondeterminism is old workers from a previous deployment still running in the same cluster, connected to the same database. This happens when:

• A previous deployment used a different Kubernetes namespace (e.g. copilot-sdk vs copilot-runtime) and was never cleaned up.
• A rolling update left old ReplicaSet pods running alongside new ones during the transition.
• Manual kubectl run or port-forward sessions left orphaned pods polling the same database.

Old workers run old orchestration code but process orchestrations that were started (or replayed) by the new code. The yield sequences differ → nondeterminism error.

Diagnosis

# Check ALL namespaces for worker pods — not just the expected one
kubectl get pods --all-namespaces -l app.kubernetes.io/component=worker --no-headers

If you see pods in an unexpected namespace or from a different ReplicaSet, that's the culprit.

Fix

# Delete the old deployment and namespace
kubectl delete deployment copilot-runtime-worker -n <old-namespace>
kubectl delete namespace <old-namespace> --wait=false

# Reset the database to clear any tainted history
NODE_TLS_REJECT_UNAUTHORIZED=0 node --env-file=.env.remote scripts/db-reset.js --yes

# Restart current workers
kubectl rollout restart deployment/copilot-runtime-worker -n copilot-runtime

After fixing

1. Reset the database — existing orchestrations have stale history:

   NODE_TLS_REJECT_UNAUTHORIZED=0 node --env-file=.env.remote scripts/db-reset.js --yes
   

2. Redeploy — or use the full deploy script:

   ./scripts/deploy-aks.sh
   

Key files

• src/orchestration.ts — the orchestration generator (audit this for determinism)
• src/session-proxy.ts — activity definitions (safe — activities run outside replay)
• scripts/deploy-aks.sh — deploy with DB reset
• scripts/db-reset.js — standalone DB reset