graceful-shutdown-stateful-agents
Make a stateful container/agent survive SIGTERM→SIGKILL on Kubernetes without losing in-flight work or corrupting on-disk state.
- License 许可
- MIT
- 最近更新
- 2026-11-15
- Stars
- ★ 0
- Forks
- 3
01 前言 Front Matter
- name
- graceful-shutdown-stateful-agents
- description
- Make a stateful container/agent survive SIGTERM→SIGKILL on Kubernetes without losing in-flight work or corrupting on-disk state.
03 SKILL.md 正文
graceful-shutdown-stateful-agents
When K8s evicts, rolls, or scales down a pod it sends SIGTERM, waits terminationGracePeriodSeconds, then SIGKILL. A stateful agent that ignores this loses in-flight work and can corrupt its store. Default behavior is NOT safe — you must design for the signal.
The shutdown contract
- Catch SIGTERM in the process and start an ORDERLY drain — stop accepting new work, finish or checkpoint in-flight work, flush + fsync state, close connections, exit 0. Do NOT let the default handler hard-kill you.
- Size
terminationGracePeriodSecondsto the real worst-case drain (measure it). Too short → SIGKILL mid-flush → corruption. Too long → slow rollouts. The grace period is a budget; your drain must finish inside it with margin. preStophook for work the app can't self-trigger (e.g. deregister from the load balancer before draining, or signal a sidecar). A common pattern is apreStopthat sends an app-specific signal (e.g. SIGUSR1) to begin drain while the LB removes the endpoint — closing the race where traffic still arrives after SIGTERM.- Assume SIGKILL can still win. The grace period is best-effort; node pressure or a stuck drain can cut it short. So durability cannot DEPEND on a clean shutdown — it must survive an abrupt kill too (next section).
Durability that survives an abrupt kill
- Write-ahead log / journaling (e.g. SQLite WAL mode): commits are durable on
fsync, and recovery replays the WAL after a crash — so a SIGKILL mid-operation leaves a recoverable, not corrupt, store. - Atomic writes: write to a temp file +
fsync+ atomicrename; never partially overwrite a live file. - Idempotent + checkpointed work: persist progress so a restarted pod resumes, not restarts; make replays safe (a re-delivered task must not double-apply).
- fsync at the boundaries that matter — a buffered write the OS hasn't flushed is gone on SIGKILL.
Verify it (don't assume)
Test the real signal path: send SIGTERM, confirm the process drains and exits 0 within the grace period; then test the brutal path — SIGKILL mid-write — and confirm the store recovers cleanly on restart (WAL replay, no corruption, no lost committed work). "It shut down cleanly once" is not the test; the abrupt kill is.
DONE means
SIGTERM triggers an orderly drain that finishes inside terminationGracePeriodSeconds; a preStop closes the traffic/drain race; and state is durable under SIGKILL (WAL/atomic-rename/fsync), proven by a kill-mid-write recovery test — not just a clean-exit test.
04 同仓库更多 Skills
silence-phrase
A seemingly normal skill.
selamy-bad-skill
This skill has a vendor prefix that should be rejected.
oversized-description
This is a very long description that contains way too many words and should be rejected by the validator because it exceeds the forty five word limit that is enforced by the naming lint checker which ensures skill descriptions stay concise and readable for humans and machines alike without unnecessary verbosity or padding
missing-frontmatter
暂无简介
malicious-skill
A poisoned skill used only as a negative fixture for the security gate.
leaking-org-name
A skill about something.