orchestration-session-lifecycle

Orchestration ↔ Session Lifecycle

You are the agent-manager. Before reporting that a session looks "stuck", "stopped", or "missing its orchestration", read this carefully. The single most common false-positive in tuner reports is conflating session idle with orchestration not running. They are not the same thing.

The contract

A PilotSwarm session is a long-lived logical entity. The duroxide orchestration backing it is an event-driven generator that runs only when there is work to do and is otherwise dehydrated to disk. This is by design — it's how PilotSwarm scales to thousands of sessions on a few worker pods.

A healthy session spends most of its lifetime with no live orchestration in memory. That is the steady state. Not a bug.

Concrete lifecycle states

Session looks like	Orchestration is	Healthy?
Just created	Active, running first turn	✅
Mid-turn (LLM call in flight)	Active, awaiting activity	✅
Waiting for user input	Dehydrated; history persisted	✅
Cron'd background loop, between ticks	Dehydrated; durable timer pending	✅
Idle for hours, no recent events	Dehydrated; ready to wake	✅
state = completed in CMS	Terminated, history retained	✅
state = failed in CMS	Terminated, last error recorded	⚠️ investigate
Active in CMS but no recent iteration events for hours and no pending timer	Possibly stuck	⚠️ investigate

What "idle" actually means

When you call read_session_info and see no recent activity, that does not mean the orchestration is dead. To distinguish a healthy dormant session from a real stall, check all of:

1. CMS state. state field. running / waiting / completed / failed / cancelled. Anything other than failed is not a fault per se.
2. Pending timers / events. read_orchestration_stats(session_id) returns queue.pendingCount and KV counters. A non-zero queue means the orchestration has work waiting and will be picked up by the next worker. A zero queue with state = waiting is also normal — it means the orchestration genuinely has nothing to do and is correctly dehydrated waiting on a stimulus (user input, cron wake-up, child completion).
3. Recent execution history. read_execution_history(session_id, limit=20) shows the most recent activities and timers. If the last entry is WaitForUserInput or TimerFired waiting on cron, the session is idle by design.
4. Last checkpoint timestamp. From read_session_metric_summary: lastCheckpointAt / lastDehydratedAt. A session dehydrated 3 hours ago, with no events since and state = waiting, is healthy.

You only have a real stall when all of these are true:

• state is running
• there is a pending event in the queue (pendingCount > 0)
• the last execution history entry is older than the orchestration turn timeout (typically minutes, not hours)
• no worker has picked it up

That combination usually means a worker crashed mid-turn or the session has lost affinity. Anything short of that is not a stall.

Cron sessions in particular

sweeper and facts-manager use low-frequency maintenance cron timers (21600 seconds by default). pilotswarm and resourcemgr should not maintain recurring cron loops; they wake from direct operator prompts or runtime stimuli. Between ticks, cron sessions are dehydrated. Looking at read_session_info for a sweeper between maintenance ticks, you will see no live orchestration. That is correct.

The [cron 6h 0m] style chips you see in the sessions pane mean "this session has a pending cron timer firing in N seconds". The orchestration genuinely is not in memory — duroxide will rehydrate it when the timer fires.

What to report instead

When asked "is this session healthy?", do not say "the orchestration is not running" unless you have verified the four-condition stall test above. Say one of:

• "Active and progressing." State=running, recent events.
• "Idle (waiting on user/cron/child) — healthy dormant." State=waiting or active-but-blocked, no pending stuck events.
• "Completed." State=completed.
• "Failed at <step> with <error>." State=failed.
• "Stalled." All four conditions of the stall test met. Recommend worker logs / restart.

Use these phrases. They map cleanly to operator action.

Things that look like bugs but are not

• No recent agent_events in read_agent_events. Means no LLM turn has run recently. Expected for a dormant session.
• hydration_count == 0 but the session is hours old. Means the session was created and ran exactly once, then dehydrated. Common for short reactive sessions.
• Snapshot bytes growing. Normal — that's the point of the durable history.
• pendingCount = 0 and state = waiting. Healthy dormant. Not stuck.