AGENT SKILL

anvil-serving-candidate-operations

Prepare, recover, switch, test, and restore an Anvil Serving model candidate through managed recipes and serves. Use for RTX 5090 lane preparation, Windows/WSL GPU-memory discrepancies, HF token handoff, unhealthy recipe-loaded containers, Q4/MTP or other profile switches, live endpoint checks, and bounded candidate tests that must preserve the starting state.

License 许可
MIT
最近更新
2026-11-14
Stars
★ 1
Forks
0

01 前言 Front Matter

name
anvil-serving-candidate-operations
description
Prepare, recover, switch, test, and restore an Anvil Serving model candidate through managed recipes and serves. Use for RTX 5090 lane preparation, Windows/WSL GPU-memory discrepancies, HF token handoff, unhealthy recipe-loaded containers, Q4/MTP or other profile switches, live endpoint checks, and bounded candidate tests that must preserve the starting state.

02 资源文件 1

03 SKILL.md 正文

Anvil Serving Candidate Operations

Run candidate work as one evidence-backed transaction. Container health, endpoint readiness, preflight, benchmark qualification, and route promotion are separate states; never infer one from another.

Start state

  1. Read README.md, CLAUDE.md, the selected recipe, and .agents/skills/anvil-serving-llm-qualification/SKILL.md before mutation.
  2. Record repository revision and dirty state, command host/runtime, topology target, router and serve state, reservations, loaded recipe containers, GPU inventory, candidate recipe revision, and exact state to restore.
  3. Use operation_contracts and the structured controller/MCP tools first. Use the documented CLI commands below only when the wrapper is absent and report that product gap.

Preflight the host boundary

  1. Resolve credentials by variable name only. If the general host-operations:credential-source-diagnostics plugin skill is installed, use it to trace the shell, Windows/WSL, service, Compose, and container boundary. Otherwise check presence without printing, hashing, or copying the value. A .env file is not proof that the candidate process receives HF_TOKEN or the recipe's declared credential variable.
  2. Run anvil-serving host gpus --json for product inventory. On Windows/WSL, use host-operations:windows-gpu-lane-hygiene when installed; otherwise capture Windows and WSL NVIDIA views, exact PIDs, managed workloads, active allocation, and residual WDDM/driver reservation. Do not claim the lane is empty from a utilization percentage or an empty CUDA process table alone.
  3. Stop for a human gate before terminating a process, changing Windows graphics preferences, restarting Docker/WSL, unloading a live serve, or changing a deployment role. Prefer per-app iGPU placement and exact managed workload lifecycle over global graphics changes.

Bound host memory before an unqualified load

GPU fit is not host-RAM fit. Record available RAM, swap, desktop/OS reserve, and the loader peak estimate before a new checkpoint/runtime startup. If the loader peak is unknown, require a managed container RAM and RAM-plus-swap ceiling that preserves the reserve. If the managed lifecycle cannot enforce that ceiling, stop the candidate trial and record the product gap; do not substitute raw Docker or rely on the global OOM killer. Preserve exact known-good restoration separately from new candidate experimentation.

For every authorized candidate load or model swap, retain bounded, timestamped host-memory snapshots in private attempt evidence: before unloading the incumbent, immediately before loading the candidate, after startup, and after restoration. A refusal or startup failure requires an immediate snapshot before cleanup when safe; capture must never delay urgent restoration. Record missing phases and unavailable fields explicitly. A documentation-only change does not require a live capture or model operation.

Each snapshot must include:

  • Total, free and available host RAM; swap total/used; memory pressure; file cache, shared memory and reclaimable/unreclaimable kernel memory. Keep these accounting views separate rather than adding overlapping counters.
  • The largest processes and managed services/containers, with PID/parent, identity, counts and proportional resident memory (PSS) when available. Group desktop/agent child tools so many small workers are not missed. Retain cgroup memory.current, anonymous/file/shared/kernel and swap accounting when available; container totals include cache and are not process PSS.
  • Candidate RAM and RAM-plus-swap ceilings, host reserve, required available RAM, observed available RAM and signed headroom/shortfall. A ceiling is not measured consumption. Bind the recipe, command host, phase and timestamp.

Use supported host/status/monitoring commands first. If Linux attribution or numeric refusal values are unavailable, retain bounded read-only /proc and cgroup evidence and record the product gap. Never read process environments or retain raw command lines, credentials or shared environment files. A nearby snapshot is not the guard's exact sample: label refusal-time values unknown unless the guard retained them. An admission refusal is not an engine OOM. Investigate co-resident owners before proposing a new memory budget; do not silently lower the reserve, evict caches, terminate agent/session processes or retry the same frozen attempt. Apply existing lifecycle authority without requesting duplicate approval.

A host-wide OOM or desktop-session loss blocks retries until containment and loader diagnosis are independently checked. See the September 17 failure and containment ticket.

Select one lifecycle path

Use recipe lifecycle for an isolated candidate:

anvil-serving models recipes show MODEL
anvil-serving models recipes load MODEL --container NAME --gpu-device GPU --dry-run
anvil-serving models recipes load MODEL --container NAME --gpu-device GPU --confirm
anvil-serving models recipes status MODEL --container NAME
anvil-serving models recipes logs MODEL --container NAME --tail 500 --contains ERROR --contains "KV cache"
anvil-serving models recipes unload MODEL --container NAME --dry-run
anvil-serving models recipes unload MODEL --container NAME --confirm

Use anvil-serving serves switch ROLE MODEL --dry-run only for a declared deployment-role change. Apply with --confirm only after the preview identifies the exact source artifacts and rollback state. A recipe load does not change router policy, and an isolated test does not require an alias change.

Recover an unhealthy candidate

  1. Run recipe status, then bounded recipe logs. Capture the earliest actionable startup error and classify authentication, authorization/license, missing dependency, incompatible engine/model, resource exhaustion, or failed health contract. Start with repeatable --contains literals for the expected failure signatures so progress bars and routine startup output do not flood agent context. Widen the filter or remove it only when the cause remains unclear; retain an unfiltered artifact outside model context when a complete raw log is required for evidence.
  2. Do not "fix unhealthy" by immediately recreating the container. First prove the owning cause and whether the recipe, image, environment handoff, health check, or product status surface is defective.
  3. If replacement is justified, verify the exact model/container identity, preview unload, obtain the mutation gate, unload, re-check the GPU lane, preview load, load, and follow bounded logs from first startup.
  4. Use raw Docker only for the narrowest read-only diagnosis when the Anvil surface cannot expose required status or logs. Record that missing capability as a product gap and return to managed commands for mutation.
  5. On Windows/WSL, fix one startup layer per managed recipe: establish NCCL transport first, then runner compatibility, then model load/compile, and only then KV/context capacity. Preserve each exact failure. A transport fix is not evidence that the model fits, and a compile OOM before KV allocation is not a measured context limit.

Qualify the candidate

  1. Require recipe status and the candidate's native health endpoint to pass.
  2. Run anvil-serving eval preflight against the exact direct endpoint/model, starting with thinking disabled and the smallest deterministic checks.
  3. Expand to the profile's context, tools, streaming, modality, and concurrency gates only after smoke passes. Use the LLM qualification skill for capacity, repeated quality evidence, and publication.
  4. Keep external recipe claims advisory. If the test came from a gist or runbook, use workflow-intake:external-procedure-intake when installed to retain its exact revision and the local adaptation matrix.

Close client context propagation

An isolated recipe load does not require client changes. A managed serve switch, router-profile change, promotion, rollback, or fix-forward recovery does whenever it changes a client-facing alias's backing identity, context_limit_tokens, max_output_tokens, reasoning, or modality contract.

Every approved promotion also requires the infrastructure capture and all-host provisioning gate in skills/anvil-serving-release-readiness/SKILL.md, even when limits or aliases are unchanged. Bind rebuilt server dependencies and every local/remote harness catalog to the same approved profile and effective router hash, then prove repeat convergence. Never mark a healthy route complete while provisioning can reinstall a stale model or client contract. Return harness_convergence_pending with the accepted serve preserved for any gap.

For such changes on Companion Node, use converge-mini-vision-clients when installed; otherwise enforce the same closure directly before reporting the operation restored or complete. Reconcile Hermes, Pi, and OpenClaw from the same authenticated router snapshot; enumerate every retained Anvil-backed Hermes profile; preserve provider/auth, unrelated settings, and compaction; and require real-client acceptance plus an idempotent second dry-run. Also verify the durable scheduled reconciliation job covers openclaw,hermes,pi, uses --hermes-profiles all, and last exited zero.

If the operation lacks authority to update/restart live clients, do not stage config in live paths. Record exact candidate bindings and return client_convergence=pending; a healthy model or router is not deployment closure.

Close an authorized promotion's publication

When the user separately approves promotion, complete the publication closure gate after live model/client acceptance. Publish the Anvil Serving benchmark narrative and update the declared Grafana and Workbench benchmark views through the operator's dashboard infrastructure repository. Verify actual result data and retain the live publication receipt; updating model labels or a private runbook is insufficient. Preserve the exact rollback and serving state while publishing. Report promoted_publication_pending if a required surface remains unpublished; do not represent the entire promotion workflow as complete.

Restore and report

For isolated or unpromoted trials, stop the candidate through its owning recipe/serve lifecycle, restore the exact starting serve and router state, and prove the GPU returns to the recorded baseline. For an explicitly approved promotion, preserve the accepted promoted serve and retained rollback instead; do not restore the starting serve. Complete client and publication closure without undoing the approved cutover. Do not remove model caches or Docker volumes as cleanup.

When restoration is required for an unpromoted trial or separately authorized rollback, treat it as a fresh managed deployment, not as an assumed container restart. Use a preserved --keep-container baseline only when its startup preparation is proven restart-idempotent; an image that verifies and patches a file in place may fail its checksum on restart. Otherwise unload the preserved container and recreate the exact starting recipe. Load the router credential from its declared user-local environment source before a mode transaction that must drain or readmit a tier. After direct health and model identity pass, require authenticated routed smoke and structured-output acceptance, exact mode ownership, GPU residency, and shared-memory status. Preserve failed restore attempts and the root cause rather than overwriting them with the successful retry.

Return the start-state fingerprint, credential presence boundary, GPU lane classification, recipe/profile identity, preview and confirmation gates, health and log cause, preflight/benchmark artifacts, restoration checks, whether client-facing metadata changed, client convergence state, product gaps, and promoted=false for isolated/unpromoted trials. For an approved promotion, report promoted=true and the retained rollback, with completion status promoted_publication_pending until the publication gate passes. Promotion remains a separate human gate.

04 同仓库更多 Skills