auto-remediation-fixes
Essential auto-remediation fixes for staging server guardian — timeout patterns, provider drift fixes, and operational hygiene for cron job errors and agent inbox remediation.
- License 许可
- MIT
- 最近更新
- 2026-11-14
- Stars
- ★ 4
- Forks
- 1
01 前言 Front Matter
- name
- auto-remediation-fixes
- version
- 1.0.0
- category
- devops
- description
- Essential auto-remediation fixes for staging server guardian — timeout patterns, provider drift fixes, and operational hygiene for cron job errors and agent inbox remediation.
03 SKILL.md 正文
Auto-Remediation Fixes — Orchestrator Patterns
General Principles
Conservative Editing of Monitoring Lists
When modifying a cron quality watchdog's MONITORED_CRONS list (or any monitoring inventory):
- Only remove entries that don't exist. An existing cron, even if it's a
no_agentscript rather than an LLM-driven cron, still produces output worth monitoring. The watchdog checks apply to all output, not just LLM-generated text. - Add new entries rather than shuffling. If the list is missing crons that should be monitored, add them. Rearranging is cosmetic and risks accidentally removing a valid entry.
- Verify each entry exists. Check
cronjob(action='list')orread_file('~/.hermes/cron/jobs.json')before deciding an entry is stale.
Timeout Patterns
Cron Timeout Fires
When a cron times out ("timed out after Ns"):
- Check the cron's actual runtime —
cronjob action='list'showslast_duration. - If the script legitimately needs more time, raise the timeout — do not "optimize" a script that is correctly slow.
- If the script hangs (waiting on network, lock, or stdin), fix the hang — raising the timeout just delays the failure.
Provider Timeout Drift
When an API provider (OpenRouter, DeepSeek, Ollama) starts timing out:
- Verify the provider's own status page (health-external-verification).
- Check
OLLAMA_KEEP_ALIVE— a 0 value unloads the model and the cold-load exceeds the_embed()timeout (see loop-governance lesson). - Add retry-with-backoff at the call site rather than raising timeouts globally.
Provider Drift Fixes
Model/Config Drift After Update
When an update cycle silently changes provider config:
- Diff the deployed config vs repo source —
config-drift-diagnosticscompares 3 locations. - Fix the repo source first, then deploy — never patch the deployed copy directly (fleet-preflight rule).
- Verify the running service actually reloaded — file-on-disk ≠ change-live (Principle 15).
Cron Prompts Referencing Missing Skills
When a cron prompt references a skill that doesn't exist or was renamed:
- Find the real skill name —
skills_list(). - Update the cron's skill attachment —
cronjob action='update' skills=[...]. - Update the installer source in
install-crons.shso the fix survives redeploys.
Operational Hygiene
Remediation Must Be Idempotent
Every fix applied by the guardian must be safe to re-run on an already-fixed system:
# ✅ Idempotent: check-then-act
if ! grep -q "setting" config; then
echo "setting" >> config
fi
# ❌ Non-idempotent: appends every time
echo "setting" >> config
Escalate After Repeated Failure
A remediation that fires 3+ times for the same root cause is a failed fix, not a healthy loop:
- First occurrence: remediate, log outcome.
- Second occurrence: remediate + check the fixer's own logic (wrong pattern?).
- Third occurrence: escalate — stop auto-fixing, report to the orchestrator with the failure history.
Unattended Destructive Actions Default to No-Op
In unattended/cron mode, destructive operations (prunes, deletions, resets) default to inaction when ambiguous. Disk pressure can be remediated; deleted data cannot.
Related
auto-remediation-ecosystem— full-pipeline deployment + monitoringauto-remediation-setup— setup and verificationcron-quality-gate— the watchdog that monitors cron outputsensor-false-positive-remediation— handling false sensor positivesfix-without-asking— when to auto-fix vs escalate
04 同仓库更多 Skills
macbook-linux-maintenance
Legacy MacBook Linux fixes: GPU wedge, reboot hang, thermal.
governance-regression-triage
Triage a host's governance issues: what changed.
enforcement-immutability
Use when enforcement files lose their immutable flag.
steadfaste-fleet-repo-deploy
Use when committing to the steadfaste-fleet repo.
steadfaste-fleet-maintenance
Maintain an installed Steadfaste Fleet instance — update both the upstream Hermes Agent and the fleet repo layer, merge diverged histories, sync skills safely, and troubleshoot common issues. Covers the dual-update workflow (hermes update + fleet-update.sh), encrypted brain data backup, mycortex import, and recovery techniques.
fleet-preflight
DEPRECATED alias — merged into survey-before-action (2026-08-20). Load survey-before-action instead; its "Repo-Specific Pre-Flight" section covers git search, Hermes boundary, and deployment verification.