AGENT SKILL

execution-and-monitoring

Select and operate local, SSH, or Slurm experiment execution with bounded, zero-LLM monitoring, safe paths, truthful terminal outcomes, and actionable recovery.

License 许可
Apache-2.0
最近更新
2026-09-03
Stars
★ 351
Forks
27

01 前言 Front Matter

name
execution-and-monitoring
license
Apache 2.0
metadata
{ "disco-role": "operating" }
description
Select and operate local, SSH, or Slurm experiment execution with bounded, zero-LLM monitoring, safe paths, truthful terminal outcomes, and actionable recovery.
disable-model-invocation
true

02 资源文件 4

03 SKILL.md 正文

Execution and monitoring

Use this skill when an experiment must be launched, inspected, or recovered and backend choice or liveness semantics matter. It covers the execution boundary, not the research loop or provider/tool-selection policy.

Route first

  1. Read the execution.mode value and the monitor settings.
  2. Select exactly one backend: local, ssh, or slurm.
  3. Validate required fields before launching anything. For ssh and slurm, controller state remains local while the tool-visible workspace and logs are remote. A Slurm login node is a submit/file-operation host, not a place to keep a watcher, tmux, srun --wait, or another persistent job process.
  4. Pass command arguments as an argument vector at the backend boundary. Never turn untrusted command text into a shell string. Treat slurm_setup and extra scheduler directives as trusted operator configuration.
  5. Record the returned pid and log_file; for Slurm, pid is the integer Slurm job id. Then use the zero-LLM monitor until a terminal decision.

See backends.md for constructors, factory fields, transport, path rules, and scheduler state mapping. See monitoring.md for the polling contract and result shape. See troubleshooting.md before retrying.

Configuration gate

The execution block uses these fields:

  • mode: local (default), ssh, or slurm.
  • ssh_host, remote_workspace, remote_python (default python3), and ssh_args (default []) for remote modes.
  • Slurm additionally requires slurm_partition and slurm_time; it accepts slurm_gpus_per_job, slurm_gres, slurm_qos, slurm_account, slurm_setup, slurm_extra_sbatch, slurm_unknown_grace_polls, and slurm_time_buffer.
  • monitor.poll_interval defaults to 900 seconds, monitor.zero_llm defaults to true, and monitor.notify_on_complete defaults to true.

Run the bundled read-only validator when changing these values:

python scripts/check_backend_config.py --config config.yaml

It does not connect, submit, launch, or write. Add --workspace-root DIR --path RELATIVE_PATH to check a path against a workspace, including symlink escape detection.

Safe launch and monitor contract

The launch tool parses command text with shell-like quoting into argv, rejects an empty command and a small set of destructive executable names, and delegates with shell=False. Semicolons and pipes therefore remain ordinary argument text; they do not create a second command. A log path is normalized and checked before the backend creates its parent or opens it.

The monitor performs only backend PID/job liveness, a short log tail, and GPU status while the job is alive. It makes no LLM call. On completion it reads up to 50 log lines, extracts common loss, accuracy, FGD, FID, epoch, and step values, and asks final_status() for the backend's terminal evidence. status is failed only when success is explicitly false; otherwise it is completed with success possibly unknown. Treat success: null as indeterminate, not proof of successful training.

For Slurm, never reap a job while sacct reports PENDING or any running-bucket state, even after slurm_time + slurm_time_buffer: queue time is not bounded by --time. Only unconfirmed probes are bounded by consecutive unknown grace and the wall-clock backstop. A confirmed Slurm FAILED, TIMEOUT, CANCELLED, OUT_OF_MEMORY, or other failure state must remain failed and be carried into state/ledger/reporting. A confirmed COMPLETED is the only Slurm success.

Recovery checklist

  • local: inspect the final log tail and rerun only after deciding whether the process ended normally; PID-only final status is inherently indeterminate.
  • ssh: first verify host/workspace/helper reachability; retry transient SSH actions without changing the remote command. Do not create a remote watcher.
  • slurm: distinguish PENDING from unknown. Query sacct first; use its state as truth, and expect the monitor's bounded unknown handling only when accounting/transport cannot confirm the job. A job reaped by the backstop has unknown terminal state; reconcile the scheduler and log before calling it a success or resubmitting.
  • Any path error mentioning “escapes workspace” is a safety stop. Correct the relative path; do not weaken normalization or follow a symlink.

04 同仓库更多 Skills

VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-triton-opt

Make a single PassNet Triton kernel fast AND numerically faithful: performance model (when a replacement can win at all), block/grid/warp tuning, autotune policy, launch overhead, and per-op numeric recipes to pass the dtype baseline tolerances. Use when a pass matches and is correct but speedup below expected, or when correctness fails by small numeric margins.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-solve

END-TO-END playbook for solving one PassNet sample: analyze the computation graphs, decide the optimization strategy (what to fuse, what to replace, what to leave alone), drive the iteration loop, and maximize the sample score. This is the ENTRY skill — invoke it first for any PassNet optimization task; it tells you when to use passnet-pattern-fusion, passnet-triton-opt and passnet-feedback.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-skill

PassNet GPU kernel optimization via compiler passes. Design and implement Triton-based optimization passes, create pass files under ./pass_dir/, self-evaluate with pass_evaluator, and iterate to maximize GPU speedup.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-pattern-fusion

Author PassNet pass files: write patterns that actually MATCH the FX graph, pick fusion regions (which ops to absorb into one Triton kernel), and structure multi-pass submissions with the mandatory shared-dispatch architecture. Use when creating or fixing pattern()/replacement_args()/replacement_func() files, when a pass "failed to match", or when deciding how to fuse multiple kernels/ops.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-orchestrate

Round/batch planning and decision support for PassNet graph optimization. Use for multi-sample triage, worker allocation, eval-budget policy, and post-eval keep/revert decisions. For solving one concrete sample end to end, use passnet-solve as the entry skill; consult this skill only when you need broader planning or a specific decision gate.

★ 351
VectorSpaceLab
VectorSpaceLab/AREX-Skill

passnet-feedback

Fast, deterministic feedback for PassNet work: pre-flight pattern/match verification WITHOUT burning a GPU evaluation, per-node bottleneck analysis of a sample's graphs, and eval-log parsing into per-variant status + estimated score + failure classification. Use BEFORE every GPU evaluation (check_pattern), at the START of a sample (analyze_graph), and AFTER every evaluation (parse_eval_log).

★ 351