execution-and-monitoring
Select and operate local, SSH, or Slurm experiment execution with bounded, zero-LLM monitoring, safe paths, truthful terminal outcomes, and actionable recovery.
- License 许可
- Apache-2.0
- 最近更新
- 2026-09-03
- Stars
- ★ 351
- Forks
- 27
01 前言 Front Matter
- name
- execution-and-monitoring
- license
- Apache 2.0
- metadata
- { "disco-role": "operating" }
- description
- Select and operate local, SSH, or Slurm experiment execution with bounded, zero-LLM monitoring, safe paths, truthful terminal outcomes, and actionable recovery.
- disable-model-invocation
- true
02 资源文件 4
- 参考文档references/backends.md6.9 KB
- 参考文档references/monitoring.md3.8 KB
- 参考文档references/troubleshooting.md5.6 KB
- 脚本scripts/check_backend_config.py7.0 KB
03 SKILL.md 正文
Execution and monitoring
Use this skill when an experiment must be launched, inspected, or recovered and backend choice or liveness semantics matter. It covers the execution boundary, not the research loop or provider/tool-selection policy.
Route first
- Read the
execution.modevalue and the monitor settings. - Select exactly one backend:
local,ssh, orslurm. - Validate required fields before launching anything. For
sshandslurm, controller state remains local while the tool-visible workspace and logs are remote. A Slurm login node is a submit/file-operation host, not a place to keep a watcher,tmux,srun --wait, or another persistent job process. - Pass command arguments as an argument vector at the backend boundary. Never
turn untrusted command text into a shell string. Treat
slurm_setupand extra scheduler directives as trusted operator configuration. - Record the returned
pidandlog_file; for Slurm,pidis the integer Slurm job id. Then use the zero-LLM monitor until a terminal decision.
See backends.md for constructors, factory fields, transport, path rules, and scheduler state mapping. See monitoring.md for the polling contract and result shape. See troubleshooting.md before retrying.
Configuration gate
The execution block uses these fields:
mode:local(default),ssh, orslurm.ssh_host,remote_workspace,remote_python(defaultpython3), andssh_args(default[]) for remote modes.- Slurm additionally requires
slurm_partitionandslurm_time; it acceptsslurm_gpus_per_job,slurm_gres,slurm_qos,slurm_account,slurm_setup,slurm_extra_sbatch,slurm_unknown_grace_polls, andslurm_time_buffer. monitor.poll_intervaldefaults to 900 seconds,monitor.zero_llmdefaults to true, andmonitor.notify_on_completedefaults to true.
Run the bundled read-only validator when changing these values:
python scripts/check_backend_config.py --config config.yaml
It does not connect, submit, launch, or write. Add --workspace-root DIR --path RELATIVE_PATH to check a path against a workspace, including symlink
escape detection.
Safe launch and monitor contract
The launch tool parses command text with shell-like quoting into argv, rejects
an empty command and a small set of destructive executable names, and delegates
with shell=False. Semicolons and pipes therefore remain ordinary argument
text; they do not create a second command. A log path is normalized and checked
before the backend creates its parent or opens it.
The monitor performs only backend PID/job liveness, a short log tail, and GPU
status while the job is alive. It makes no LLM call. On completion it reads up
to 50 log lines, extracts common loss, accuracy, FGD, FID, epoch, and
step values, and asks final_status() for the backend's terminal evidence.
status is failed only when success is explicitly false; otherwise it is
completed with success possibly unknown. Treat success: null as
indeterminate, not proof of successful training.
For Slurm, never reap a job while sacct reports PENDING or any running-bucket
state, even after slurm_time + slurm_time_buffer: queue time is not bounded by
--time. Only unconfirmed probes are bounded by consecutive unknown grace and
the wall-clock backstop. A confirmed Slurm FAILED, TIMEOUT, CANCELLED,
OUT_OF_MEMORY, or other failure state must remain failed and be carried into
state/ledger/reporting. A confirmed COMPLETED is the only Slurm success.
Recovery checklist
local: inspect the final log tail and rerun only after deciding whether the process ended normally; PID-only final status is inherently indeterminate.ssh: first verify host/workspace/helper reachability; retry transient SSH actions without changing the remote command. Do not create a remote watcher.slurm: distinguishPENDINGfrom unknown. Querysacctfirst; use its state as truth, and expect the monitor's bounded unknown handling only when accounting/transport cannot confirm the job. A job reaped by the backstop has unknown terminal state; reconcile the scheduler and log before calling it a success or resubmitting.- Any path error mentioning “escapes workspace” is a safety stop. Correct the relative path; do not weaken normalization or follow a symlink.
04 同仓库更多 Skills
passnet-triton-opt
Make a single PassNet Triton kernel fast AND numerically faithful: performance model (when a replacement can win at all), block/grid/warp tuning, autotune policy, launch overhead, and per-op numeric recipes to pass the dtype baseline tolerances. Use when a pass matches and is correct but speedup below expected, or when correctness fails by small numeric margins.
passnet-solve
END-TO-END playbook for solving one PassNet sample: analyze the computation graphs, decide the optimization strategy (what to fuse, what to replace, what to leave alone), drive the iteration loop, and maximize the sample score. This is the ENTRY skill — invoke it first for any PassNet optimization task; it tells you when to use passnet-pattern-fusion, passnet-triton-opt and passnet-feedback.
passnet-skill
PassNet GPU kernel optimization via compiler passes. Design and implement Triton-based optimization passes, create pass files under ./pass_dir/, self-evaluate with pass_evaluator, and iterate to maximize GPU speedup.
passnet-pattern-fusion
Author PassNet pass files: write patterns that actually MATCH the FX graph, pick fusion regions (which ops to absorb into one Triton kernel), and structure multi-pass submissions with the mandatory shared-dispatch architecture. Use when creating or fixing pattern()/replacement_args()/replacement_func() files, when a pass "failed to match", or when deciding how to fuse multiple kernels/ops.
passnet-orchestrate
Round/batch planning and decision support for PassNet graph optimization. Use for multi-sample triage, worker allocation, eval-budget policy, and post-eval keep/revert decisions. For solving one concrete sample end to end, use passnet-solve as the entry skill; consult this skill only when you need broader planning or a specific decision gate.
passnet-feedback
Fast, deterministic feedback for PassNet work: pre-flight pattern/match verification WITHOUT burning a GPU evaluation, per-node bottleneck analysis of a sample's graphs, and eval-log parsing into per-variant status + estimated score + failure classification. Use BEFORE every GPU evaluation (check_pattern), at the START of a sample (analyze_graph), and AFTER every evaluation (parse_eval_log).