cluster-agent-lifecycle

Cluster Agent Lifecycle Skill

As the Platform Agent you own the lifecycle of Cluster Agents. A Cluster Agent is a Hermes profile — an isolated agent instance with its own persona (SOUL.md), scoped toolset, and home directory — that you create dynamically inside your own pod, one per managed GKE cluster. It handles read-only runtime operations and deep workload diagnostics on that single cluster, and returns its findings to you.

You never debug tenant workloads directly while the cluster has a Cluster Agent. You delegate that to it and act on what it returns.

The engine for all of this is the helper script scripts/cluster_agent_profile.py (resolved at /opt/data/scripts/cluster_agent_profile.py at runtime).

When to create a profile

Create the Cluster Agent profile as part of cluster onboarding — immediately after a cluster is successfully provisioned (see gke-cluster-creation) or when an existing cluster is first brought under management (see manage-cluster). A managed cluster and its Cluster Agent profile are created together: when you onboard or tear down a cluster, keep the profile in step with it. That rule is scoped to onboarding and teardown, where the profile is yours to manage. It is not a roster invariant to enforce from other work — the reconcile job below owns the roster, and an empty roster is a supported state, not damage to repair. In particular, the first-run discovery sweep must never create or repair profiles.

python3 /opt/data/scripts/cluster_agent_profile.py create \
  --project "<project>" --cluster "<cluster>" --location "<location>"

This scaffolds the profile home on the persistent data PVC, pins a kubeconfig scoped to that cluster, stamps the cluster identity into SOUL.md and config.yaml, writes the cluster identity and the control-plane endpoint it chose when one was decided into the profile's USER.md for preflight checks, probes the cluster once (a failure is logged with the endpoint it chose, when one was decided, and, for a connection failure, the remedy; the scaffold still succeeds), and registers the profile. It is idempotent — safe to re-run. It prints the profile name.

How to delegate a debugging / runtime-ops task (kanban board)

For any request that concerns runtime behavior of workloads on a single, specific cluster (crash loops, OOMs, scheduling failures, mount errors, connectivity, autoscaling, storage, observability gaps), delegate to that cluster's Cluster Agent instead of investigating yourself.

Personas never pass context directly. Delegation runs on the shared kanban board: you create a card assigned to the cluster's profile; the gateway's kanban dispatcher auto-spawns the Cluster Agent to work it; it reports a structured result on the card. You do not invoke the agent yourself.

1. Resolve the cluster's profile name (the kanban assignee) with the get_cluster_profile_name(project, cluster, location) tool. It returns name and exists; list_cluster_profiles() returns every profile with its project, cluster and location, for when you still have to find the cluster. Both run in the agent pod — do not look the name up with cluster_agent_profile.py name, which is a stub in your shell, and do not block on it.

   • Assign only to a profile that exists. A card for a name that is not a profile is never dispatched.
   • If it does not exist, the cluster has no usable Cluster Agent: none yet (the hourly reconcile job below creates one for every cluster in scope), a scaffold that never finished, or a profile under that name pinned to a different cluster. Investigate it yourself and say in your result that it had no Cluster Agent.

2. Create the card with the request in the body:

   kanban_create(
     assignee="<profile-name>",
     title="<short title>",
     body="<full request: namespace/workload, symptom, time window>"
   )
   

   The dispatcher spawns the Cluster Agent (hermes -p <profile> chat -q "work kanban task <id>") automatically; it reads the card, does read-only diagnostics, and calls kanban_complete(result=<the RCA>, summary=<one-line status>, metadata={...}).

3. Read the result — the card carries the requester's chat subscription, so the completion (or a needs_input block) is posted into their thread. You can also inspect it: kanban_show(<id>). The RCA is in the card's result — the field the gateway posts verbatim, and the only one the requester receives — with any proposed patch in metadata; neither is ever in the worker's chat reply, which is a bare acknowledgement by design.

Multi-cluster (fan-out): at most ten per-cluster cards (SOUL.md §6, "Multi-cluster work"); a question over more clusters is answered from one fleet-wide read first, and a general fleet health question is one run of fleet-audit/scripts/fleet_health.py, with no cards. Within that bound, create one card per cluster with no parents, assigned to that cluster's profile (never platform), in one burst, so they run in parallel. parents means "runs after", so a per-cluster card that lists your own running card as a parent can never be claimed — see SOUL.md §0. Then keep your own card open: poll the unsettled per-cluster cards by emitting parallel kanban_show(<id>) tool calls in a single turn (terminal(command="sleep 60", timeout=90) between rounds, never one kanban_show per turn), and once all of them are settled, synthesize their result/metadata into your own kanban_complete(result=...). Completing your card is the delivery, so never complete it on a dispatch receipt — the image refuses a kanban_complete while your fanned-out cards are unfinished (#1010). See the workload-rebalancing skill for the validation-then-declare pattern.

Acting on the result

The Cluster Agent is read-only and does not open Pull Requests. After reading the completed card:

1. Review the RCA in the card's result and the proposed manifest patch in its metadata.
2. If a change is warranted and the request asked for it, you open (or update) the Pull Request via the submit-suggestion skill — you own the GitOps write path. Reconcile against any existing branch/PR for the same workload before creating a new one. If the request asked for a diagnosis, the RCA and the proposed patch go back in your reply as a recommendation (SOUL.md §3, item 3).
3. Report the outcome to the user as a clean SRE status update.

When to delete a profile

Delete the Cluster Agent profile as part of cluster teardown (see gke-cluster-creation), after the cluster itself is removed:

python3 /opt/data/scripts/cluster_agent_profile.py delete \
  --project "<project>" --cluster "<cluster>" --location "<location>"

This deregisters the profile and removes its home directory, here and in the shell sandbox. Do not delete a profile while its cluster still exists.

To stop managing a cluster that still exists, exclude it instead; the reconcile below recreates a deleted profile for every in-scope cluster. Follow manage-cluster, "Stop managing": the exclusion is declared where the install's scope is declared (SCOPE_EXCLUDE_CLUSTERS in install.env on an install made with install.sh, applied with ./upgrade.sh --upgrade-mode=full), and that section says how to tell the install shapes apart.

Automatic reconciliation (create and prune)

The roster is also reconciled automatically. An hourly, deterministic no_agent cron job (cluster-agent-reconcile) runs scripts/cluster_agent_reconcile.py, which drives the roster in both directions:

• Create — every cluster in every project in scope gets a profile, including the management cluster kube-agents itself runs on. The scope is the management project alone unless the PlatformAgent declares spec.scope; the only exceptions are clusters named in spec.scope.exclude.clusters or, for one more release, bare names in RECONCILE_EXCLUDE. A project the pod cannot list, or a folder, organisation, Shared VPC host or Metrics Scope it cannot read (whose previous members are then carried forward frozen, with no CREATE under them), is recorded with its outcome in fleet_scope.json at the root of the data volume (/opt/data, beside profiles/; the reconcile's own HERMES_HOME, not a profile's) (written by every run except --dry-run) and skipped for that run rather than guessed at. The management cluster is included because its own workloads fail like any other cluster's, and the agent that triages a Kubernetes event is the one scoped to the cluster that raised it.
• Prune — a profile is deleted when its GKE cluster is definitively gone (a NotFound from gcloud container clusters describe), when it belongs to an excluded cluster, which must not carry a profile even though that cluster exists, or when its project has left the scope, over two clean runs: a clean run (no project unreachable, every folder, organisation, Shared VPC host and Metrics Scope resolved ok or over-cap, the management project resolved, listed its own clusters and unchanged since the last run, the scope file readable) that finds a previously in-scope project absent marks it retiring in fleet_scope.json; the next clean run prunes its profiles. A profile whose project the scope never produced, or whose prune waits for that second run, is kept and listed as unmanaged; one whose identity could not be read is kept and appears under the report's skipped_no_identity, and under the snapshot's profiles once an earlier run has read its identity. This closes the loop when a cluster is deleted out-of-band, so its profile is never left orphaned pointing at a dead kubeconfig.

A create that fails is recorded rather than only logged: the cluster goes into a create_failed bucket. A run creates no more than BOOTSTRAP_MAX_CREATES_PER_TICK profiles (100 by default; failed creates do not count) and puts the clusters it left for the next run into a create_deferred bucket. Both places the failure bucket surfaces are narrower than it looks. The chat summary names the failures only on a run that also created or pruned something — a run whose sole outcome is a failed create posts nothing, because a permanent cause repeats every minute while the bootstrap gate is ticking. A failed create is only one of the things --require-create-pass reports. Under that flag the run exits 3 when the CREATE direction never ran (the management project would not resolve, or its clusters could not be listed), when reconcile() raised, or when it ran and every create failed with no fully scaffolded profile already on the roster — 4 when another reconcile holds the lock — and 5 when it created its cap's worth and deferred the rest, which the gate reads like 4: retry next tick, no attempt counted. One failure alongside a success exits 0 and costs the sweep one gaps row. Without the flag every one of those exits 0, because a cron producer must. That flag's caller is the bootstrap scan gate, which gates a one-shot fleet sweep on the exit code, so a non-zero exit means "the roster is not worth fanning out against yet", not specifically "a create failed".

A profile whose cluster_identity is stamped but whose kubeconfig.yaml or USER.md is missing is treated as absent and recreated. Profile creation stamps the identity before it writes either file, so a run killed mid-create leaves a home that satisfies the create direction's existence check and the prune direction's "identity present, cluster exists" check — unrepaired, it stays half-built forever.

It never deletes on ambiguity: any inconclusive check (auth/network/timeout, or a missing cluster_identity) leaves the profile untouched. created=0 pruned=0 kept=0 is a normal, successful result. When it creates or prunes anything it posts a summary to every chat platform it can resolve as enabled, falling back to Google Chat when it can resolve none.

Profile lifecycle belongs to this script and to the explicit onboarding/teardown paths above. Do not repair the roster from other work by calling cluster_agent_profile.py directly — a profile created for an excluded cluster is one the next run will prune.

To preview what would change without touching anything:

python3 /opt/data/scripts/cluster_agent_reconcile.py --dry-run

You still delete a profile explicitly during planned teardown (above) — reconciliation is the safety net, not the primary path.

Listing profiles

Call list_cluster_profiles(). It lists the currently provisioned Cluster Agent profiles (one per managed cluster) with the cluster each is pinned to.