anvil-serving-llm-qualification
Qualify a pinned local LLM or VLM on Anvil Serving with reproducible text, image, video, capacity, router, and storage evidence. Use for model bakeoffs, failure diagnosis and researched configuration trials, quant comparisons, context/concurrency qualification, multimodal enablement, or production-role recommendations that must remain human-gated.
- License 许可
- MIT
- 最近更新
- 2026-11-14
- Stars
- ★ 1
- Forks
- 0
01 前言 Front Matter
- name
- anvil-serving-llm-qualification
- description
- Qualify a pinned local LLM or VLM on Anvil Serving with reproducible text, image, video, capacity, router, and storage evidence. Use for model bakeoffs, failure diagnosis and researched configuration trials, quant comparisons, context/concurrency qualification, multimodal enablement, or production-role recommendations that must remain human-gated.
02 资源文件 2
- 参考文档references/evidence-contract.md5.3 KB
- 参考文档references/runtime-investigation.md10.2 KB
03 SKILL.md 正文
Anvil Serving LLM Qualification
Use the product CLI for lifecycle, cache, preflight, benchmark, and router work. Do not create skill-local operational scripts or promote a route.
The maintained layout is PRODUCT_ROOT/skills/anvil-serving-llm-qualification.
For global discovery, symlink that directory rather than copying it. Resolve
its real path and verify the product root contains pyproject.toml and
anvil_serving/ before using repository-relative paths. For a copied install,
use the available workspace-resolution skill to locate the product checkout
and read its canonical skill; never infer the checkout from the caller's cwd.
Read references/evidence-contract.md and references/configuration-search.md
before starting. The latter owns failure investigation and candidate stopping
rules; preserve failed configurations while testing supported successors.
For runtime crashes, configuration improvement, or hardware-limit exploration,
read runtime-investigation.md before live
requests. Validate its incident scenario offline first; finish the incident
branch separately from a replacement-model comparison.
Apply the shared session-improvement-loop:research-synthesis policy through its Anvil
bindings without requiring the
user to ask for fusion: use one initial researcher and shared source registry,
then independent answers only for a named unresolved material decision.
Budget one synthesis and keep local-only questions on retained local evidence.
Workflow
-
Record the repository revision, dirty state, live serve/router state, exact model/runtime revisions, GPU identity, and cache inventory.
-
Work in an isolated
codex/worktree. Preserve unrelated and untracked files. Before downloading or loading a candidate, validate the offline harness:python scripts/run_tests.py tests/test_agentic_benchmark.py tests/test_benchmark_suite_runner.py -qStop on a failed fixture/oracle check. Strict tool protocol, fixture state, final-answer formatting and independent coding quality remain separate evidence. Record the oracle revision; never rewrite historical scores as if they had used a newer scorer.
-
Gate storage with
models cache inventory. For any removal, use an exact revision dry-run and explicit confirmation. Never broad-prune Docker or delete volumes. -
Record every source with observed date, evidence class, hardware/runtime relevance, and decision impact. Treat model cards and discussions as priors, not local proof.
-
Pull exact revisions with
models pull; a shell/client timeout does not prove failure. Before retrying, inspect whether the one downloader container remains active. Never start competing writers against the same cache. -
Load one isolated candidate at a time with a pinned recipe and discovered GPU UUID. For a Windows/WSL GPU lane, credential handoff, unhealthy loaded recipe, managed switch, or restore transaction, use
.agents/skills/anvil-serving-candidate-operations/SKILL.md. Do not change a live alias. -
Diagnose startup and request failures down-stack: caller, product status/logs, container exit/health, then engine/model-download error. Retain the earliest actionable error, research it, and test a versioned configuration remedy under
references/configuration-search.mdbefore rejecting a candidate. -
For a hardware-gated upstream kernel patch, pin the upstream commit, verify the exact source and result hashes, and fail startup unless the required symbol imports or the engine logs prove the intended path loaded. Treat the patch as an exact recipe artifact, not generic runtime guidance.
-
For missing hardware-specific MoE/GEMM config warnings or measured kernel bottlenecks, use
skills/anvil-serving-kernel-tuning/SKILL.md. Do not recommend a generated tune without the identical untuned-versus-tuned A/B. -
Declare and qualify the model-native reasoning and sampling policy intended for the role. Use thinking-disabled probes only when supported and relevant; they do not qualify a deployment that will use a different reasoning policy.
-
Before requests for a serving comparison, retain the content-bound frozen plan and matching
metrics_coveragefor its profile × workload × context × requested-concurrency matrix. Run capacity at the declared points with the campaign’s output/reasoning headroom. Keep performance measurement, quality, and promotion independent: preserve quality misses, stop unsafe infrastructure/resource failures, and never call the comparison complete with required non-measured metrics. -
For images/video, run direct endpoint preflight and
eval benchmark multimodalbefore router work. If directvideo_urlfails, retain the failure and pause routed video qualification while investigating the direct configuration. Resume only after its direct gates pass. -
If direct video passes, verify same-dialect preservation, fail-closed unsupported translation, admission/count/token limits, streaming, tools, malformed media, and an isolated router configuration.
-
A profile passes only with exact identity, all deterministic assertions, visible answers, allowed finish reasons, matching media hashes, valid tools, and no OOM/parser corruption.
-
Explicitly classify the result as either isolated-only or client-facing. If no public alias, router metadata, or selected backing identity changed, prove that fact and do not rewrite client catalogs. If an authorized promotion, rollback, or route switch changes any client-facing alias's backing identity,
context_limit_tokens,max_output_tokens, reasoning, or modality contract, client propagation is a required qualification closure gate even when the request did not mention clients. -
Every approved promotion must also pass the all-host provisioning and rebuild-snapshot gate in
skills/anvil-serving-release-readiness/SKILL.md, even when advertised limits are unchanged. Record source bindings, inventory coverage and repeat convergence; missing coverage isharness_convergence_pending, not completed promotion. For client-facing changes on Companion Node, useconverge-mini-vision-clientswhen installed; otherwise enforce the same closure directly. Reconcile Hermes, Pi, and OpenClaw from one authenticated router snapshot. Enumerate every retained Anvil-backed Hermes profile, preserve provider/auth and compaction settings, validate real client behavior, and require an idempotent final dry-run. If live client mutation/restart authority is not present, report the qualification as client-convergence-pending rather than complete or deployed. -
Restore the exact starting serve/router state. Freeze the final candidate source, run focused and full repository gates once, then delegate publication to
skills/anvil-serving-benchmark-docs/SKILL.md. Apply itsreferences/artifact-set-contract.mdandanvil-serving.benchmark-artifact-set/v1template around the native raw artifacts, dated finding, precision/modality decision table, and required publication-ready result card. Add a publication summary only when compact public communication is requested; otherwise mark that rolenot-applicablewith a reason. For a format-only refresh backed by complete retained artifacts, reconcile the published values without restarting the serve or rerunning the benchmark.
Improve the workflow from evidence
At campaign closure or after a material process failure, read
references/improvement-loop.md. Capture scoped lessons in existing evidence,
then evaluate authorized skill changes independently. Preserve the running
qualification baseline and the budget for the requested model outcome.
Decisions
- Separate official/community claims from locally measured results.
- Compare quantized profiles only after every hard gate passes for the exact frozen profile. A failed profile returns to configuration search; it does not automatically reject the checkpoint or engine.
- Compare speculation to an otherwise identical no-speculation control. Keep model, revision, image, patch, TP, context, concurrency, KV dtype, memory fraction, backends, batching, graph capture, transport, parsers, and offload policy fixed; do not inherit a KV dtype already known to fail that hardware path.
- Prefer a quant only when it meets the campaign’s declared memory or throughput improvement threshold relative to the next higher precision.
- Limit a known-crashing vision quant to a text-only role; do not infer future support from shipped vision weights.
- Keep promotion separately human-gated.
04 同仓库更多 Skills
voice
暂无简介
workbench
暂无简介
workbench
暂无简介
workbench
暂无简介
anvil-serving-voice-ops
Operate topology-owned anvil-serving voice audio and Realtime proxy lifecycle, Mini-to-Dark loopback forwarding, profiles, sidecars, and bounded voice benchmarks. Use for voice status, logs, start/stop/restart, OpenClaw Talk topology checks, and voice evidence that must remain separate from LLM serve qualification evidence.
anvil-serving-stt-benchmark
Prepare, validate, execute, compare, and publish reproducible multi-sample speech-to-text qualifications in anvil-serving. Use for STT model bakeoffs, corpus-based WER/CER and latency testing, candidate overlays, concurrency tests, restoration checks, and fail-closed stt-benchmark-evidence/v1 artifacts. Do not use for a single voice-pipeline turn or LLM qualification.