[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6PCQMAV72WYZTYSE35RE4TH":3,"skill-same:01M6PCQMAV72WYZTYSE35RE4TH":273},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":15,"categories":16,"keywords":27,"publishedAt":48,"linkOnly":49,"markdownContent":50,"markdownBody":51,"frontMatter":52,"refName":37,"commitSha":59,"skillDirectory":60,"forksCount":13,"repositoryHtmlUrl":61,"repositoryHomepageUrl":6,"ownerHtmlUrl":62,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":63,"install":64,"resources":77},"01M6PCQMAV72WYZTYSE35RE4TH","remote-gpu-trainer",null,"Deploy, monitor, and debug long GPU jobs on RENTED\u002Fremote instances (AutoDL, RunPod, vast.ai, Lambda, Slurm, K8s): teardown\u002Fbilling safety, spot resilience, resumable checkpointing, OOM\u002FNaN triage.","duc-duy-1612\u002FOnline-Delivery-Systems-Web","duc-duy-1612","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F192231384?v=4","01M6NSDKQQDKVTHZ8DZMZ1VYY6","01M6NSDKW8DDVTQHBT1HBDDJPT",0,"MIT","https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002FSKILL.md",[17,22],{"categoryNo":18,"categoryName":19,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000023","Skill与智能体开发","SCENARIO","使用场景",{"categoryNo":23,"categoryName":24,"taxonomyType":25,"taxonomyTypeDesc":26},"01JBSEEDCAT000000000000030","开发工程师","PROFESSION","适用职业",[28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47],"gpu","remote gpu trainer","remote","instances","dl","billing","host","teardown","map","main","runner","resilience","community","references","superpowers","Bash","api","lambda","monitoring","code","2026-11-10 10:12:30",false,"---\nname: remote-gpu-trainer\ndescription: \"Deploy, monitor, and debug long GPU jobs on RENTED\u002Fremote instances (AutoDL, RunPod, vast.ai, Lambda, Slurm, K8s): teardown\u002Fbilling safety, spot resilience, resumable checkpointing, OOM\u002FNaN triage.\"\nrisk: safe\nsource: community\nsource_type: community\nsource_repo: Hanyuyuan6\u002Fremote-gpu-trainer\ndate_added: \"2026-06-20\"\ncategory: ml-ops\nlicense: \"MIT\"\nlicense_source: \"https:\u002F\u002Fgithub.com\u002FHanyuyuan6\u002Fremote-gpu-trainer\u002Fblob\u002Fmain\u002FLICENSE\"\ncompatibility: |\n  Any Agent-Skills (SKILL.md)-compatible agent — Claude Code, Codex, Cursor, Trae, Gemini CLI, etc.\n  Needs a shell + SSH (or a platform CLI\u002FAPI) to drive the remote box; scripts are bash\u002Fpython. A few\n  durable-monitoring recipes assume a host background-task runner + scheduler — map to the running\n  agent's equivalents (references\u002Fmonitoring_patterns.md §7). Companion skills (verifying-dl-experiments,\n  superpowers:*, huggingface-skills:*) are optional separate installs.\n---\n\n# remote-gpu-trainer — Remote GPU Job Orchestration\n\n## Overview\n\nDeploy and babysit long-running GPU jobs on **rented boxes you don't own**, across any platform, and\nget the result off the box before the meter or a preemption kills it. The core insight: **you are a\nshort-term tenant on someone else's machine** — so the job is to *detach the work, make the result\noutlive the instance, and stop the meter safely*, not to provision a cluster.\n\nThis skill is **platform-agnostic at the core, platform-specific at the edges**: a fixed set of\noperating principles + a 6-phase lifecycle that hold everywhere, plus one **profile per platform**\n(`profiles\u002F\u003Cplatform>.md`) that owns every concrete path, proxy, billing verb, and spot semantic. Its\ndefensible value is the union the big orchestrators skip: **Chinese cgroup-isolated rentals + bare-SSH\ncheap boxes + the disk-budget \u002F monitoring \u002F teardown reality** that *is* the job on metered hardware.\n\n## When to Use This Skill\n\nUse whenever the user deploys, trains, monitors, or troubleshoots a long-running GPU job on a **RENTED\nor remote instance they do not own** — training, eval, ablation sweeps, batch inference, or large data\nprocessing — on AutoDL, RunPod, vast.ai, Lambda, Paperspace, Chinese platforms (恒源云\u002F矩池云\u002FFeaturize\u002F\n揽睿星舟), a bare SSH box, Slurm, or Kubernetes; single OR multi-instance. Triggers (multilingual):\n\"远程 GPU 训练\", \"GPU 租赁\", \"GPU rental\", \"租卡\", \"spot 抢占\", \"spot preemption\", \"断点续训\",\n\"resumable training\", \"tmux 训练守护\", \"防 SSH 断线\", \"scp\u002Frsync 上传\", \"多实例 ablation\",\n\"远程 GPU 监控\", \"省钱关机\u002F销毁实例\", \"stop vs terminate billing\", \"checkpoint 磁盘满\",\n\"CUDA OOM\u002F显存不足\", \"loss NaN\u002Floss spike\", \"loss 不下降\u002F不收敛\", \"overfit 单 batch\",\n\"FSDP\u002FDeepSpeed 配置\", \"多卡训练 hang\", \"dataloader worker\u002F数据增广 bug\". **NOT** for purely local\nsingle-GPU training, in-instance multi-GPU DDP (use torchrun\u002Faccelerate), managed multi-cloud\nprice-shopping (use SkyPilot's skill), or zero-ops serverless (use Modal).\n\n## When NOT to use — and what to use instead\n\n| Situation | Use instead |\n|---|---|\n| Local single-GPU, or multi-GPU **DDP inside one box** | `torchrun` \u002F `accelerate` directly |\n| Managed multi-cloud price-shopping + auto spot-recovery across **Western** clouds | **SkyPilot** (has its own Agent Skill) — then come back here to make your *code* resume-correct so its recovery actually works |\n| Open BYOC dev environments | **dstack** |\n| Zero-ops serverless inference | **Modal** |\n| \"Is this metric \u002F ablation delta real?\" | **REQUIRED:** `verifying-dl-experiments` (this skill owns *running* the job; that one owns *whether the number is true*) |\n\n**This skill is for the blind spot those tools leave:** AutoDL + Chinese platforms, bare SSH\u002FSlurm\u002FK8s\nrentals, and the operational gotchas (inode caps, mirror stalls, cgroup OOM, silent sync, spot grace\nwindows, irreversible teardown) that survive whichever provisioner you use.\n\n## Operating principles (the WHY — 10 invariants)\n\nThese hold on every metered, isolated, rented GPU; only the paths\u002FCLI change. One line each; the deep\nform with cross-platform nuance is in **`references\u002Fprinciples.md`** (read it before Phase 0).\n\n1. **Minimize paid wall-clock.** The meter runs the whole time — smoke locally on CPU before renting, launch detached, release the instant verification passes.\n2. **Cheap checks before expensive compute.** A 1–2 batch CPU smoke (logger off) kills import\u002Fconfig\u002Fshape\u002Fscale bugs for ~free. (Smoke *content* → `verifying-dl-experiments`.)\n3. **Trust artifacts you loaded, not log lines that claim success.** \"synced\u002Fsaved\u002Fdone\" lies under a silently-failed write; a watcher's own state is also a claim — reconcile it against the real process\u002Fartifact.\n4. **Know what survives stop vs destroy.** Per platform, identify exactly which mount survives a *stop* and which survives a *terminate* — the data you need often lives on the volatile one. (The single biggest portability trap.)\n5. **Storage fails on the dimension you're not watching.** Disk dies on **inodes** before bytes; the real hog hides in a symlinked cache; clean by value (keep tiny evidence, drop big scratch); monitor `df -i`, not just `df -h`.\n6. **Never mutate inputs under a live run.** A running job holds its scripts in memory by byte-offset; overwriting one mid-run re-executes blocks. Version filenames.\n7. **Design for retry — failure is probabilistic, transfers are flaky, mirrors are route-specific.** Make wrappers idempotent + resumable; retry the *identical* config; wrap bulk transfers in `timeout`+resume loops; a mirror\u002Fproxy speeds ONE route — validate on the same route the real transfer uses.\n8. **Checkpoint-to-durable + idempotent resume is the universal spine.** File checkpoint to the platform's durable location + unconditional load-latest-on-startup is the *one* mechanism that survives an SSH drop, a Slurm walltime kill, a K8s reschedule, a spot preemption, and a Colab disconnect. The detach primitive (tmux\u002Fsbatch\u002FJob\u002Fcommit) is the swappable plug; this is the invariant.\n9. **Cost and destructive actions are the user's call.** Never auto-release\u002Fterminate, never delete durable files without confirmation; if cleanup can't free space, **ask to expand the disk** rather than silently shrink the experiment.\n10. **Teach the user the platform, don't just drive it.** Most users don't know a platform's non-obvious **conveniences** (one-click SSH-key registration, GPU-availability notifications, built-in panels) or its **danger clocks** (auto-release\u002Fauto-delete timers on a *stopped* box — AutoDL releases a 关机 instance after 15 days → data disk gone; a stop that keeps billing; low-balance purge). Surface them on first contact — #9 stops the agent *doing* the dangerous thing, #10 *warns the human* before the clock fires. Per-platform list → each profile's **Surface to the user** block.\n\n> **Monitoring physics (substrate for #3):** foreground Bash hard-caps at 600 s; `run_in_background` has no cap and notifies on exit; a never-exiting watcher never notifies; an unquoted `|` in a poll regex reads stdin and hangs forever. The four-layer monitoring architecture is built on these facts → `references\u002Fmonitoring_patterns.md`.\n\n## Code discipline (the wrapper & training scripts you write)\n\nTwo rules govern the launch\u002Fwrapper\u002Ftraining code this skill has you write — corollaries of #1 and #8, not new invariants:\n\n1. **Reuse before writing.** Take the lowest rung that already works before adding code: the base image's pre-installed stack + platform features → a framework\u002Flibrary utility (`torchrun` \u002F `accelerate` \u002F HF) → your existing `scripts\u002F` templates → minimal new code. On a metered box a needless `pip install` also burns paid wall-clock and can break the image's ABI — Phase 1's rule (*the prebuilt image **is** the env; don't `conda create` on a rental*) is exactly this principle applied to dependencies.\n2. **Floor — `minimum` bounds scope, not correctness.** Shrinking code must never drop what makes an expensive run survivable: checkpoint-to-durable + idempotent resume (#8), atomic writes, the error handling that prevents losing a long run, or seed\u002Fdeterminism logging. Keep one minimal self-check for non-trivial logic.\n\n## Pick your platform profile FIRST\n\nRead the matching profile **before Phase 0** — it owns every path, proxy, credential location, billing\nverb, and spot rule the phases below delegate to. Each follows the same 8-field schema\n(`profiles\u002F_schema.md`).\n\n> **New here? The path is:** (1) find your platform in the table below → (2) read that profile's **LAUNCH**\n> section (it walks rent → register SSH key → reach the box) → (3) come back and run the 6 phases from Phase 0.\n> Already have a box you can `ssh` into? Skip straight to Phase 0.\n\n| You're on… | Profile | Kind | Detach primitive | Meter-stop verb |\n|---|---|---|---|---|\n| AutoDL (deepest, battle-tested) | `profiles\u002Fautodl.md` | ssh-rental | tmux | 关机 (stops meter, **keeps disk** — the AutoDL exception) |\n| RunPod | `profiles\u002Frunpod.md` | ssh-rental | tmux | **terminate** (stop still bills 2×; destroys volume disk) |\n| vast.ai | `profiles\u002Fvastai.md` | ssh-rental (spot) | tmux | **destroy** (stop bills disk forever) |\n| Lambda | `profiles\u002Flambda.md` | cloud-api | tmux | **terminate** (no stop state) |\n| Paperspace | `profiles\u002Fpaperspace.md` | cloud-api | tmux | **destroy + release IP + delete storage** (shut-down stops compute only) |\n| 恒源云 \u002F 矩池云 \u002F Featurize \u002F 揽睿星舟 | `profiles\u002Fchina.md` | ssh-rental | tmux | per-platform (data disk often bills while stopped) |\n| Bare SSH box \u002F Slurm \u002F K8s \u002F Colab-Kaggle | `profiles\u002Fgeneric-ssh.md` | ssh \u002F slurm \u002F k8s | tmux \u002F sbatch \u002F Job \u002F commit | **manual** (a forgotten box bills 24\u002F7) |\n\n> **Profile confidence:** AutoDL is battle-tested from the author's daily use; the other six profiles are\n> built from each platform's official docs + community reports (cited inline, `verified \u003Cmonth>`) and not\n> yet independently live-tested — lean on the Phase-0 live measurements and **re-verify any teardown\u002F\n> billing fact against current docs before betting money or data** (`references\u002Fself-improvement.md` §5).\n\n**Mental verb model** (one API across all platforms; the profile binds each verb to real commands):\n`up` (rent+reach) → `push` (code\u002Fdata on) → `run` (detached + checkpointing) → `watch` (durable monitor) → `pull` (results off + verify) → `down` (stop the meter).\n\n## Default workflow (6 phases)\n\nSkip phases already done. Each phase delegates substrate to the profile and **ends in a runnable check**.\n\n**Phase 0 — Environment audit.** Read the profile's STORAGE survival-matrix + region\u002FDC-lock. Measure live:\n`df -h && df -i \u003Cdata-mount>`, cgroup `memory.max`, `nvidia-smi`. Pre-compute the checkpoint disk budget\n(`ckpt_size × N + scratch`). → **verify:** `nvidia-smi` shows the expected GPU and `df -i` is not near 100%.\n\n**Phase 1 — SSH + credentials.** Set the alias\u002Fenv per the profile (the prebuilt image\u002Fbase IS the env —\ndo not `conda create` on a rental). **Never rented before? the profile's LAUNCH section walks rent → register SSH key → connect.** Push secrets via **stdin, never onto a shared\u002Fdurable FS**\n(`references\u002Fssh_transport.md`). → **verify:** `ssh \u003Calias> 'python -c \"import torch;print(torch.cuda.is_available())\"'`.\n\n**Phase 2 — Wrapper + CPU-smoke gate.** Build an idempotent `run_one`\u002F`run_queue` from `scripts\u002F` (parameterized\nfrom the profile's OVERRIDES; **size batch\u002Fworkers to the box for a standalone run, but PIN them across cells for a fair comparison** — `references\u002Ftraining\u002Fthroughput-profiling.md`). **Run the cheap CPU smoke locally BEFORE renting** — it kills the dumb,\nexpensive failures (e.g. `python -m \u003Cyour.train.module> --limit-batches 2 --epochs 1` — substitute your own entrypoint; this gate needs your training code plugged in). → **verify:** that smoke exits 0 on 2 batches with the logger disabled.\n\n**Phase 3 — Detached launch.** Launch via the profile's detach primitive; probe briefly (log head + alive +\nno traceback), then **hand back** — never a blocking foreground `sleep`. → **verify:** within 60 s, the detach\nsession is alive and the first log line shows the expected step\u002Fepoch.\n\n**Phase 4 — Durable monitoring.** For anything over ~1–2 h, deploy the **four-layer architecture**\n(`references\u002Fmonitoring_patterns.md`): on-box self-completion chain + session patrol loop + event sentinels +\nrecovery handbook. **On Claude Code, fire the L2 patrol via `\u002Floop 30m` (or `ScheduleWakeup`) running `scripts\u002Fhealth_patrol.sh.template`**; a host with no local recurring runner wires the on-box self-push instead (`references\u002Fmonitoring_patterns.md` §7). A session-bound watcher alone dies with the session. Classify each outcome →\nfixed remediation; **never blind-retry**. → **verify:** the patrol reports even when nothing changed.\n\n**Phase 5 — Aggregate + verify + teardown.** Checked-sync to durable storage (gate the success line on the\ncopy result — principle #3), then **load-and-verify each artifact** (`scripts\u002Fverify_local.py`), THEN the profile's\nmeter-stopping action. → **verify:** `verify_local.py` reports 100% OK *before* any teardown.\n\n> **Iron Law — teardown gate:** NO `release` \u002F `terminate` \u002F `destroy` \u002F file-delete until checkpoints are\n> **pulled to local AND verified by load**, and the user has explicitly approved the cost-affecting action.\n> \"It looked done in the log\" is not evidence (principle #3). On most platforms the meter-stopping action is\n> **irreversible** (deletes the disk) — confirmation matters more, not less.\n\n## Parallel ablation fan-out\n\nFor N ablation cells: one job per cell, an **isolated write path per job** (no shared mutable output), launched\nacross instances\u002Fqueues. **REQUIRED:** `superpowers:dispatching-parallel-agents` supplies the independence\npredicate (don't fan out onto shared state) and the mandatory post-fan-out reconciliation. FS-shared deployment\npattern → `references\u002Fparallel_ablation.md`.\n\n## Quick reference — the four facts that bite per platform\n\nFull detail in each profile; this table is the at-a-glance.\n\n| Platform | Survives **stop** | Survives **destroy** | Spot grace | China mirror needed |\n|---|---|---|---|---|\n| AutoDL | \u002Froot + data + FS | FS only | n\u002Fa | yes (`\u002Fetc\u002Fnetwork_turbo`, hf-mirror) |\n| RunPod | volume disk (bills 2×) | Network Volume only | ~5 s SIGTERM→KILL | no (`hf_transfer`) |\n| vast.ai | disk (bills forever) | nothing | ~0 s (abrupt) | no |\n| Lambda | n\u002Fa (no stop) | nothing | n\u002Fa (on-demand) | no |\n| China (恒源云\u002F矩池云\u002F…) | varies; data disk bills | per-platform persistent vol | n\u002Fa | yes |\n| generic-SSH\u002FSlurm\u002FK8s | you own it | you own it | Slurm SIGTERM→KillWait (def 30 s) | only if in China |\n\n## Common gotchas (top 8 inline — full catalog in references\u002F)\n\nThe universal ones that cost the most GPU-hours. Symptom → fix; root cause + the rest in\n**`references\u002Fgotchas_universal.md`** (run `grep -i '\u003Ckeyword>' references\u002Fgotchas_universal.md` to jump).\n\n1. **SSH drops on `pkill -9`** (exit 255 + \"Connection reset\") — normal; re-ssh to verify, don't panic.\n2. **tmux holds the script in memory** — editing it mid-run re-executes blocks; version the filename.\n3. **Disk-full crashes `torch.save`** (`iostream error`) — pre-budget; auto-prune `latest.pth`, keep `best`.\n4. **cgroup OOM with no traceback** (bare `Killed` \u002F exit 137) — `num_workers × big-tensor`; size workers vs `memory.max`, not CPU count.\n5. **Silent sync failure** — `cp … 2>\u002Fdev\u002Fnull; echo synced` lies on a full\u002Finode-exhausted FS; gate the success line on the actual copy result.\n6. **Spot preemption grace is tiny (~5 s → ~0 s on the platforms profiled here; AWS-style 2-min grace only on clouds not profiled)** — a SIGTERM-flush handler is NOT a safety net; checkpoint on a timer to durable storage, load-latest unconditionally (`references\u002Fspot-resilience.md`).\n7. **\"Stop\" rarely stops the meter** — only `terminate`\u002F`destroy` does, and it's irreversible (deletes the disk). Know the verb from the profile before you click, and on RunPod a stopped Pod can even restart with zero GPUs.\n8. **CRLF breaks `.sh` on Linux** — author on Windows → `.gitattributes` `*.sh text eol=lf`; on-box unblock `sed -i 's\u002F\\r$\u002F\u002F'`.\n\n## When training itself breaks (the model, not the platform)\n\nPlatform ops is only half the job — once the box is running, training breaks in its own ways. The\n`references\u002Ftraining\u002F` layer is the debug knowledge for the run itself. Boundary: **this layer owns\n\"make it run, fast, and not crash\"; `verifying-dl-experiments` owns \"is the *number* real\"** —\ncross-link it for collapse \u002F leakage \u002F metric-validity. Every entry is symptom → root cause → fix with\ncited current docs.\n\n- `references\u002Ftraining\u002Foom-memory.md` — CUDA\u002FVRAM + host-RAM OOM and the fit-it ladder (grad-accum → bf16 → activation-checkpointing → `expandable_segments` → FSDP\u002FZeRO → CPU\u002FNVMe offload → LoRA\u002FQLoRA); OOM-at-a-specific-step (first backward \u002F val \u002F longest batch); the memory snapshot + visualizer.\n- `references\u002Ftraining\u002Fdistributed-launch.md` — `torchrun`\u002F`accelerate`\u002F`deepspeed` launch + env contract, DDP\u002FFSDP\u002FZeRO config, and the multi-GPU **HANGS** toolkit (one-rank-diverged, rank-conditional collective, dataloader-length mismatch). Multi-node wire → `references\u002Fmultinode.md`.\n- `references\u002Ftraining\u002Fprecision-stability.md` — fp16\u002Fbf16\u002Ftf32 + AMP\u002FGradScaler, NaN\u002FInf hunting (`detect_anomaly`), LLM **loss spikes** + divergence (warmup, clip, init, z-loss).\n- `references\u002Ftraining\u002Fthroughput-profiling.md` — GPU-bound vs data-bound vs comms-bound; dataloader knobs; `torch.compile` traps; flash-attention; `torch.profiler` \u002F Nsight.\n- `references\u002Ftraining\u002Fcheckpoint-resume.md` — full-state save\u002Fresume mechanics, sharded (FSDP\u002FDeepSpeed) checkpoints, and the resume bugs (epoch restart, data reshuffle, scaler\u002FEMA dropped). Spot cadence → `references\u002Fspot-resilience.md`.\n- `references\u002Ftraining\u002Fby-domain.md` — per-domain gotchas: LLM\u002Ftransformer, vision (det\u002Fseg), diffusion, RL, multimodal\u002FVLM.\n- `references\u002Ftraining\u002Fconvergence-debugging.md` — the **\"runs but won't learn \u002F learns badly\"** layer: the overfit-one-batch smoke, params-not-updating, optimizer\u002FLR\u002Fweight-decay\u002Fschedule config, loss-function footguns (double-softmax, BCEWithLogits, CE-target form), fine-tuning\u002Ffreezing (frozen-BN drift, discriminative LR, LoRA wiring), and the training-dynamics dashboard (update:weight ratio, dead-ReLU, GradScaler-scale).\n- `references\u002Ftraining\u002Fdata-pipeline.md` — dataloader\u002Fdataset **correctness** (not speed): the worker-RNG augmentation-duplication bug, IterableDataset worker\u002Frank sharding, collate\u002F`__len__`\u002F`pin_memory`\u002F`spawn` contracts, and preprocessing\u002Flabel\u002Fshuffle traps (RGB-vs-BGR, ToTensor ÷255, `set_epoch`).\n\n## Companion skills (separate installs; REQUIRED reading where present)\n\nThese are **separate** Agent Skills, not bundled here — install them for the full experience. On an\nagent where a companion isn't installed, treat its pointer below as an optional cross-reference; this\nskill still works standalone.\n\n- **`verifying-dl-experiments`** — owns *is-the-number-real*: smoke content, retry-vs-safeguard, keepable-checkpoint, eval sizing, tracker forensics, GPU-0%-util diagnosis. This skill owns *where\u002Fwhen\u002Fhow-much-$*.\n- **`huggingface-skills:hf-cli`** — the transport verbs (`hf download --resume`, `hf upload-large-folder`, `hf cache verify`); this skill owns the China-mirror swap + stall-retry (`references\u002Fchina-network.md`).\n- **`huggingface-skills:huggingface-trackio`** — hosted tracker so metrics survive teardown (gotcha U20); poll `trackio` alerts as a structured monitor instead of brittle ssh-tail.\n- **`superpowers:verification-before-completion`** — the Iron Law's general form; gates every \"training done \u002F synced \u002F teardown complete\" claim.\n- **`superpowers:dispatching-parallel-agents`** — independence predicate + reconciliation for ablation fan-out.\n\n## Getting better over time (capture new gotchas + personalize)\n\nThis skill is static, but every run can teach it something — without corrupting it.\nProtocol → **`references\u002Fself-improvement.md`**. In short: when a run surfaces a gotcha the catalog\nlacks, **only sediment a root-caused, reproduced, generalizable one** (a one-off flake is a hypothesis,\nnot a gotcha — principle #3); **route it** — user\u002Fproject-specific → the host's memory system,\ngeneralizable → propose adding to `references\u002Fgotchas_universal.md` \u002F the profile §7 \u002F\n`references\u002Ftraining\u002F` (and offer an upstream PR); **never silently rewrite a skill file — draft the\n`symptom → root cause → fix` and let the user approve.** On first use, capture the user's platforms +\npaths + tracker entity into memory so later runs are pre-parameterized. Platform facts carry a `verified\n\u003Cmonth>` stamp — re-verify any teardown\u002Fbilling fact against current docs before betting money or data.\n\n## Limitations\n\n- Does not replace a real cloud orchestrator or managed provisioner; use it to make rented-box work survivable, not to optimize multi-cloud procurement.\n- Platform billing, stop, destroy, and data-retention behavior can drift; re-check current provider docs before destructive or money-impacting actions.\n- Requires user-owned credentials, SSH\u002FAPI access, and explicit confirmation before teardown, deletion, or other irreversible cleanup.\n- Companion skills named above are not bundled here; treat them as optional references unless installed in the current agent environment.\n\n## Bundled resources\n\nLoad only what the current phase needs.\n\n- `references\u002Fprinciples.md` — the 10 invariants expanded, with the cross-platform nuance behind each.\n- `references\u002Flifecycle_checklist.md` — the 6-phase runbook as a per-platform checklist.\n- `references\u002Fgotchas_universal.md` — universal + mixed gotchas (TOC + grep index at top).\n- `references\u002Fmonitoring_patterns.md` — the four-layer durable-monitoring architecture + robust ssh-poll template.\n- `references\u002Fssh_transport.md` — ssh config, rsync\u002Fscp resumable patterns, secrets-via-stdin, CRLF, two-SSH-flavor caveat.\n- `references\u002Fchina-network.md` — mirrors table + HF_ENDPOINT + resumable-download ladder + the `no_proxy` trap (all CN platforms).\n- `references\u002Fspot-resilience.md` — preemption signals, Young\u002FDaly checkpoint cadence, atomic-write resume.\n- `references\u002Fparallel_ablation.md` — FS-shared fan-out + the independence predicate + reconciliation.\n- `references\u002Fmultinode.md` — (advanced) NCCL \u002F fabric-manager \u002F elastic-training gotchas; single-box users skip.\n- `references\u002Ftraining\u002F` — the **DL-training debug layer** (8 files: oom-memory, distributed-launch, precision-stability, throughput-profiling, checkpoint-resume, by-domain, convergence-debugging, data-pipeline) — see \"When training breaks\" above.\n- `references\u002Fself-improvement.md` — the feedback loop: capture a new gotcha (at a bar) into memory or the catalog, personalize on first run, keep platform facts fresh.\n- `scripts\u002F` — wrapper templates (`run_one`\u002F`run_queue`), monitors (`mem_monitor`, `gpu_health`, `reap_vram_zombies`), the read-only patrol (`health_patrol.sh.template`), transfer\u002Faggregation (`download_loop`, `aggregate_to_fs`, `setup-china-mirrors`), the load-and-verify checker (`verify_local.py`), and the `verified`-stamp freshness linter (`check_staleness.py`).\n- `profiles\u002F\u003Cplatform>.md` — the per-platform substrate (one per platform; `_schema.md` defines the 8 fields).\n- `examples\u002Fautodl_sweep\u002F` — one complete, runnable worked case end to end.\n","# remote-gpu-trainer — Remote GPU Job Orchestration\n\n## Overview\n\nDeploy and babysit long-running GPU jobs on **rented boxes you don't own**, across any platform, and\nget the result off the box before the meter or a preemption kills it. The core insight: **you are a\nshort-term tenant on someone else's machine** — so the job is to *detach the work, make the result\noutlive the instance, and stop the meter safely*, not to provision a cluster.\n\nThis skill is **platform-agnostic at the core, platform-specific at the edges**: a fixed set of\noperating principles + a 6-phase lifecycle that hold everywhere, plus one **profile per platform**\n(`profiles\u002F\u003Cplatform>.md`) that owns every concrete path, proxy, billing verb, and spot semantic. Its\ndefensible value is the union the big orchestrators skip: **Chinese cgroup-isolated rentals + bare-SSH\ncheap boxes + the disk-budget \u002F monitoring \u002F teardown reality** that *is* the job on metered hardware.\n\n## When to Use This Skill\n\nUse whenever the user deploys, trains, monitors, or troubleshoots a long-running GPU job on a **RENTED\nor remote instance they do not own** — training, eval, ablation sweeps, batch inference, or large data\nprocessing — on AutoDL, RunPod, vast.ai, Lambda, Paperspace, Chinese platforms (恒源云\u002F矩池云\u002FFeaturize\u002F\n揽睿星舟), a bare SSH box, Slurm, or Kubernetes; single OR multi-instance. Triggers (multilingual):\n\"远程 GPU 训练\", \"GPU 租赁\", \"GPU rental\", \"租卡\", \"spot 抢占\", \"spot preemption\", \"断点续训\",\n\"resumable training\", \"tmux 训练守护\", \"防 SSH 断线\", \"scp\u002Frsync 上传\", \"多实例 ablation\",\n\"远程 GPU 监控\", \"省钱关机\u002F销毁实例\", \"stop vs terminate billing\", \"checkpoint 磁盘满\",\n\"CUDA OOM\u002F显存不足\", \"loss NaN\u002Floss spike\", \"loss 不下降\u002F不收敛\", \"overfit 单 batch\",\n\"FSDP\u002FDeepSpeed 配置\", \"多卡训练 hang\", \"dataloader worker\u002F数据增广 bug\". **NOT** for purely local\nsingle-GPU training, in-instance multi-GPU DDP (use torchrun\u002Faccelerate), managed multi-cloud\nprice-shopping (use SkyPilot's skill), or zero-ops serverless (use Modal).\n\n## When NOT to use — and what to use instead\n\n| Situation | Use instead |\n|---|---|\n| Local single-GPU, or multi-GPU **DDP inside one box** | `torchrun` \u002F `accelerate` directly |\n| Managed multi-cloud price-shopping + auto spot-recovery across **Western** clouds | **SkyPilot** (has its own Agent Skill) — then come back here to make your *code* resume-correct so its recovery actually works |\n| Open BYOC dev environments | **dstack** |\n| Zero-ops serverless inference | **Modal** |\n| \"Is this metric \u002F ablation delta real?\" | **REQUIRED:** `verifying-dl-experiments` (this skill owns *running* the job; that one owns *whether the number is true*) |\n\n**This skill is for the blind spot those tools leave:** AutoDL + Chinese platforms, bare SSH\u002FSlurm\u002FK8s\nrentals, and the operational gotchas (inode caps, mirror stalls, cgroup OOM, silent sync, spot grace\nwindows, irreversible teardown) that survive whichever provisioner you use.\n\n## Operating principles (the WHY — 10 invariants)\n\nThese hold on every metered, isolated, rented GPU; only the paths\u002FCLI change. One line each; the deep\nform with cross-platform nuance is in **`references\u002Fprinciples.md`** (read it before Phase 0).\n\n1. **Minimize paid wall-clock.** The meter runs the whole time — smoke locally on CPU before renting, launch detached, release the instant verification passes.\n2. **Cheap checks before expensive compute.** A 1–2 batch CPU smoke (logger off) kills import\u002Fconfig\u002Fshape\u002Fscale bugs for ~free. (Smoke *content* → `verifying-dl-experiments`.)\n3. **Trust artifacts you loaded, not log lines that claim success.** \"synced\u002Fsaved\u002Fdone\" lies under a silently-failed write; a watcher's own state is also a claim — reconcile it against the real process\u002Fartifact.\n4. **Know what survives stop vs destroy.** Per platform, identify exactly which mount survives a *stop* and which survives a *terminate* — the data you need often lives on the volatile one. (The single biggest portability trap.)\n5. **Storage fails on the dimension you're not watching.** Disk dies on **inodes** before bytes; the real hog hides in a symlinked cache; clean by value (keep tiny evidence, drop big scratch); monitor `df -i`, not just `df -h`.\n6. **Never mutate inputs under a live run.** A running job holds its scripts in memory by byte-offset; overwriting one mid-run re-executes blocks. Version filenames.\n7. **Design for retry — failure is probabilistic, transfers are flaky, mirrors are route-specific.** Make wrappers idempotent + resumable; retry the *identical* config; wrap bulk transfers in `timeout`+resume loops; a mirror\u002Fproxy speeds ONE route — validate on the same route the real transfer uses.\n8. **Checkpoint-to-durable + idempotent resume is the universal spine.** File checkpoint to the platform's durable location + unconditional load-latest-on-startup is the *one* mechanism that survives an SSH drop, a Slurm walltime kill, a K8s reschedule, a spot preemption, and a Colab disconnect. The detach primitive (tmux\u002Fsbatch\u002FJob\u002Fcommit) is the swappable plug; this is the invariant.\n9. **Cost and destructive actions are the user's call.** Never auto-release\u002Fterminate, never delete durable files without confirmation; if cleanup can't free space, **ask to expand the disk** rather than silently shrink the experiment.\n10. **Teach the user the platform, don't just drive it.** Most users don't know a platform's non-obvious **conveniences** (one-click SSH-key registration, GPU-availability notifications, built-in panels) or its **danger clocks** (auto-release\u002Fauto-delete timers on a *stopped* box — AutoDL releases a 关机 instance after 15 days → data disk gone; a stop that keeps billing; low-balance purge). Surface them on first contact — #9 stops the agent *doing* the dangerous thing, #10 *warns the human* before the clock fires. Per-platform list → each profile's **Surface to the user** block.\n\n> **Monitoring physics (substrate for #3):** foreground Bash hard-caps at 600 s; `run_in_background` has no cap and notifies on exit; a never-exiting watcher never notifies; an unquoted `|` in a poll regex reads stdin and hangs forever. The four-layer monitoring architecture is built on these facts → `references\u002Fmonitoring_patterns.md`.\n\n## Code discipline (the wrapper & training scripts you write)\n\nTwo rules govern the launch\u002Fwrapper\u002Ftraining code this skill has you write — corollaries of #1 and #8, not new invariants:\n\n1. **Reuse before writing.** Take the lowest rung that already works before adding code: the base image's pre-installed stack + platform features → a framework\u002Flibrary utility (`torchrun` \u002F `accelerate` \u002F HF) → your existing `scripts\u002F` templates → minimal new code. On a metered box a needless `pip install` also burns paid wall-clock and can break the image's ABI — Phase 1's rule (*the prebuilt image **is** the env; don't `conda create` on a rental*) is exactly this principle applied to dependencies.\n2. **Floor — `minimum` bounds scope, not correctness.** Shrinking code must never drop what makes an expensive run survivable: checkpoint-to-durable + idempotent resume (#8), atomic writes, the error handling that prevents losing a long run, or seed\u002Fdeterminism logging. Keep one minimal self-check for non-trivial logic.\n\n## Pick your platform profile FIRST\n\nRead the matching profile **before Phase 0** — it owns every path, proxy, credential location, billing\nverb, and spot rule the phases below delegate to. Each follows the same 8-field schema\n(`profiles\u002F_schema.md`).\n\n> **New here? The path is:** (1) find your platform in the table below → (2) read that profile's **LAUNCH**\n> section (it walks rent → register SSH key → reach the box) → (3) come back and run the 6 phases from Phase 0.\n> Already have a box you can `ssh` into? Skip straight to Phase 0.\n\n| You're on… | Profile | Kind | Detach primitive | Meter-stop verb |\n|---|---|---|---|---|\n| AutoDL (deepest, battle-tested) | `profiles\u002Fautodl.md` | ssh-rental | tmux | 关机 (stops meter, **keeps disk** — the AutoDL exception) |\n| RunPod | `profiles\u002Frunpod.md` | ssh-rental | tmux | **terminate** (stop still bills 2×; destroys volume disk) |\n| vast.ai | `profiles\u002Fvastai.md` | ssh-rental (spot) | tmux | **destroy** (stop bills disk forever) |\n| Lambda | `profiles\u002Flambda.md` | cloud-api | tmux | **terminate** (no stop state) |\n| Paperspace | `profiles\u002Fpaperspace.md` | cloud-api | tmux | **destroy + release IP + delete storage** (shut-down stops compute only) |\n| 恒源云 \u002F 矩池云 \u002F Featurize \u002F 揽睿星舟 | `profiles\u002Fchina.md` | ssh-rental | tmux | per-platform (data disk often bills while stopped) |\n| Bare SSH box \u002F Slurm \u002F K8s \u002F Colab-Kaggle | `profiles\u002Fgeneric-ssh.md` | ssh \u002F slurm \u002F k8s | tmux \u002F sbatch \u002F Job \u002F commit | **manual** (a forgotten box bills 24\u002F7) |\n\n> **Profile confidence:** AutoDL is battle-tested from the author's daily use; the other six profiles are\n> built from each platform's official docs + community reports (cited inline, `verified \u003Cmonth>`) and not\n> yet independently live-tested — lean on the Phase-0 live measurements and **re-verify any teardown\u002F\n> billing fact against current docs before betting money or data** (`references\u002Fself-improvement.md` §5).\n\n**Mental verb model** (one API across all platforms; the profile binds each verb to real commands):\n`up` (rent+reach) → `push` (code\u002Fdata on) → `run` (detached + checkpointing) → `watch` (durable monitor) → `pull` (results off + verify) → `down` (stop the meter).\n\n## Default workflow (6 phases)\n\nSkip phases already done. Each phase delegates substrate to the profile and **ends in a runnable check**.\n\n**Phase 0 — Environment audit.** Read the profile's STORAGE survival-matrix + region\u002FDC-lock. Measure live:\n`df -h && df -i \u003Cdata-mount>`, cgroup `memory.max`, `nvidia-smi`. Pre-compute the checkpoint disk budget\n(`ckpt_size × N + scratch`). → **verify:** `nvidia-smi` shows the expected GPU and `df -i` is not near 100%.\n\n**Phase 1 — SSH + credentials.** Set the alias\u002Fenv per the profile (the prebuilt image\u002Fbase IS the env —\ndo not `conda create` on a rental). **Never rented before? the profile's LAUNCH section walks rent → register SSH key → connect.** Push secrets via **stdin, never onto a shared\u002Fdurable FS**\n(`references\u002Fssh_transport.md`). → **verify:** `ssh \u003Calias> 'python -c \"import torch;print(torch.cuda.is_available())\"'`.\n\n**Phase 2 — Wrapper + CPU-smoke gate.** Build an idempotent `run_one`\u002F`run_queue` from `scripts\u002F` (parameterized\nfrom the profile's OVERRIDES; **size batch\u002Fworkers to the box for a standalone run, but PIN them across cells for a fair comparison** — `references\u002Ftraining\u002Fthroughput-profiling.md`). **Run the cheap CPU smoke locally BEFORE renting** — it kills the dumb,\nexpensive failures (e.g. `python -m \u003Cyour.train.module> --limit-batches 2 --epochs 1` — substitute your own entrypoint; this gate needs your training code plugged in). → **verify:** that smoke exits 0 on 2 batches with the logger disabled.\n\n**Phase 3 — Detached launch.** Launch via the profile's detach primitive; probe briefly (log head + alive +\nno traceback), then **hand back** — never a blocking foreground `sleep`. → **verify:** within 60 s, the detach\nsession is alive and the first log line shows the expected step\u002Fepoch.\n\n**Phase 4 — Durable monitoring.** For anything over ~1–2 h, deploy the **four-layer architecture**\n(`references\u002Fmonitoring_patterns.md`): on-box self-completion chain + session patrol loop + event sentinels +\nrecovery handbook. **On Claude Code, fire the L2 patrol via `\u002Floop 30m` (or `ScheduleWakeup`) running `scripts\u002Fhealth_patrol.sh.template`**; a host with no local recurring runner wires the on-box self-push instead (`references\u002Fmonitoring_patterns.md` §7). A session-bound watcher alone dies with the session. Classify each outcome →\nfixed remediation; **never blind-retry**. → **verify:** the patrol reports even when nothing changed.\n\n**Phase 5 — Aggregate + verify + teardown.** Checked-sync to durable storage (gate the success line on the\ncopy result — principle #3), then **load-and-verify each artifact** (`scripts\u002Fverify_local.py`), THEN the profile's\nmeter-stopping action. → **verify:** `verify_local.py` reports 100% OK *before* any teardown.\n\n> **Iron Law — teardown gate:** NO `release` \u002F `terminate` \u002F `destroy` \u002F file-delete until checkpoints are\n> **pulled to local AND verified by load**, and the user has explicitly approved the cost-affecting action.\n> \"It looked done in the log\" is not evidence (principle #3). On most platforms the meter-stopping action is\n> **irreversible** (deletes the disk) — confirmation matters more, not less.\n\n## Parallel ablation fan-out\n\nFor N ablation cells: one job per cell, an **isolated write path per job** (no shared mutable output), launched\nacross instances\u002Fqueues. **REQUIRED:** `superpowers:dispatching-parallel-agents` supplies the independence\npredicate (don't fan out onto shared state) and the mandatory post-fan-out reconciliation. FS-shared deployment\npattern → `references\u002Fparallel_ablation.md`.\n\n## Quick reference — the four facts that bite per platform\n\nFull detail in each profile; this table is the at-a-glance.\n\n| Platform | Survives **stop** | Survives **destroy** | Spot grace | China mirror needed |\n|---|---|---|---|---|\n| AutoDL | \u002Froot + data + FS | FS only | n\u002Fa | yes (`\u002Fetc\u002Fnetwork_turbo`, hf-mirror) |\n| RunPod | volume disk (bills 2×) | Network Volume only | ~5 s SIGTERM→KILL | no (`hf_transfer`) |\n| vast.ai | disk (bills forever) | nothing | ~0 s (abrupt) | no |\n| Lambda | n\u002Fa (no stop) | nothing | n\u002Fa (on-demand) | no |\n| China (恒源云\u002F矩池云\u002F…) | varies; data disk bills | per-platform persistent vol | n\u002Fa | yes |\n| generic-SSH\u002FSlurm\u002FK8s | you own it | you own it | Slurm SIGTERM→KillWait (def 30 s) | only if in China |\n\n## Common gotchas (top 8 inline — full catalog in references\u002F)\n\nThe universal ones that cost the most GPU-hours. Symptom → fix; root cause + the rest in\n**`references\u002Fgotchas_universal.md`** (run `grep -i '\u003Ckeyword>' references\u002Fgotchas_universal.md` to jump).\n\n1. **SSH drops on `pkill -9`** (exit 255 + \"Connection reset\") — normal; re-ssh to verify, don't panic.\n2. **tmux holds the script in memory** — editing it mid-run re-executes blocks; version the filename.\n3. **Disk-full crashes `torch.save`** (`iostream error`) — pre-budget; auto-prune `latest.pth`, keep `best`.\n4. **cgroup OOM with no traceback** (bare `Killed` \u002F exit 137) — `num_workers × big-tensor`; size workers vs `memory.max`, not CPU count.\n5. **Silent sync failure** — `cp … 2>\u002Fdev\u002Fnull; echo synced` lies on a full\u002Finode-exhausted FS; gate the success line on the actual copy result.\n6. **Spot preemption grace is tiny (~5 s → ~0 s on the platforms profiled here; AWS-style 2-min grace only on clouds not profiled)** — a SIGTERM-flush handler is NOT a safety net; checkpoint on a timer to durable storage, load-latest unconditionally (`references\u002Fspot-resilience.md`).\n7. **\"Stop\" rarely stops the meter** — only `terminate`\u002F`destroy` does, and it's irreversible (deletes the disk). Know the verb from the profile before you click, and on RunPod a stopped Pod can even restart with zero GPUs.\n8. **CRLF breaks `.sh` on Linux** — author on Windows → `.gitattributes` `*.sh text eol=lf`; on-box unblock `sed -i 's\u002F\\r$\u002F\u002F'`.\n\n## When training itself breaks (the model, not the platform)\n\nPlatform ops is only half the job — once the box is running, training breaks in its own ways. The\n`references\u002Ftraining\u002F` layer is the debug knowledge for the run itself. Boundary: **this layer owns\n\"make it run, fast, and not crash\"; `verifying-dl-experiments` owns \"is the *number* real\"** —\ncross-link it for collapse \u002F leakage \u002F metric-validity. Every entry is symptom → root cause → fix with\ncited current docs.\n\n- `references\u002Ftraining\u002Foom-memory.md` — CUDA\u002FVRAM + host-RAM OOM and the fit-it ladder (grad-accum → bf16 → activation-checkpointing → `expandable_segments` → FSDP\u002FZeRO → CPU\u002FNVMe offload → LoRA\u002FQLoRA); OOM-at-a-specific-step (first backward \u002F val \u002F longest batch); the memory snapshot + visualizer.\n- `references\u002Ftraining\u002Fdistributed-launch.md` — `torchrun`\u002F`accelerate`\u002F`deepspeed` launch + env contract, DDP\u002FFSDP\u002FZeRO config, and the multi-GPU **HANGS** toolkit (one-rank-diverged, rank-conditional collective, dataloader-length mismatch). Multi-node wire → `references\u002Fmultinode.md`.\n- `references\u002Ftraining\u002Fprecision-stability.md` — fp16\u002Fbf16\u002Ftf32 + AMP\u002FGradScaler, NaN\u002FInf hunting (`detect_anomaly`), LLM **loss spikes** + divergence (warmup, clip, init, z-loss).\n- `references\u002Ftraining\u002Fthroughput-profiling.md` — GPU-bound vs data-bound vs comms-bound; dataloader knobs; `torch.compile` traps; flash-attention; `torch.profiler` \u002F Nsight.\n- `references\u002Ftraining\u002Fcheckpoint-resume.md` — full-state save\u002Fresume mechanics, sharded (FSDP\u002FDeepSpeed) checkpoints, and the resume bugs (epoch restart, data reshuffle, scaler\u002FEMA dropped). Spot cadence → `references\u002Fspot-resilience.md`.\n- `references\u002Ftraining\u002Fby-domain.md` — per-domain gotchas: LLM\u002Ftransformer, vision (det\u002Fseg), diffusion, RL, multimodal\u002FVLM.\n- `references\u002Ftraining\u002Fconvergence-debugging.md` — the **\"runs but won't learn \u002F learns badly\"** layer: the overfit-one-batch smoke, params-not-updating, optimizer\u002FLR\u002Fweight-decay\u002Fschedule config, loss-function footguns (double-softmax, BCEWithLogits, CE-target form), fine-tuning\u002Ffreezing (frozen-BN drift, discriminative LR, LoRA wiring), and the training-dynamics dashboard (update:weight ratio, dead-ReLU, GradScaler-scale).\n- `references\u002Ftraining\u002Fdata-pipeline.md` — dataloader\u002Fdataset **correctness** (not speed): the worker-RNG augmentation-duplication bug, IterableDataset worker\u002Frank sharding, collate\u002F`__len__`\u002F`pin_memory`\u002F`spawn` contracts, and preprocessing\u002Flabel\u002Fshuffle traps (RGB-vs-BGR, ToTensor ÷255, `set_epoch`).\n\n## Companion skills (separate installs; REQUIRED reading where present)\n\nThese are **separate** Agent Skills, not bundled here — install them for the full experience. On an\nagent where a companion isn't installed, treat its pointer below as an optional cross-reference; this\nskill still works standalone.\n\n- **`verifying-dl-experiments`** — owns *is-the-number-real*: smoke content, retry-vs-safeguard, keepable-checkpoint, eval sizing, tracker forensics, GPU-0%-util diagnosis. This skill owns *where\u002Fwhen\u002Fhow-much-$*.\n- **`huggingface-skills:hf-cli`** — the transport verbs (`hf download --resume`, `hf upload-large-folder`, `hf cache verify`); this skill owns the China-mirror swap + stall-retry (`references\u002Fchina-network.md`).\n- **`huggingface-skills:huggingface-trackio`** — hosted tracker so metrics survive teardown (gotcha U20); poll `trackio` alerts as a structured monitor instead of brittle ssh-tail.\n- **`superpowers:verification-before-completion`** — the Iron Law's general form; gates every \"training done \u002F synced \u002F teardown complete\" claim.\n- **`superpowers:dispatching-parallel-agents`** — independence predicate + reconciliation for ablation fan-out.\n\n## Getting better over time (capture new gotchas + personalize)\n\nThis skill is static, but every run can teach it something — without corrupting it.\nProtocol → **`references\u002Fself-improvement.md`**. In short: when a run surfaces a gotcha the catalog\nlacks, **only sediment a root-caused, reproduced, generalizable one** (a one-off flake is a hypothesis,\nnot a gotcha — principle #3); **route it** — user\u002Fproject-specific → the host's memory system,\ngeneralizable → propose adding to `references\u002Fgotchas_universal.md` \u002F the profile §7 \u002F\n`references\u002Ftraining\u002F` (and offer an upstream PR); **never silently rewrite a skill file — draft the\n`symptom → root cause → fix` and let the user approve.** On first use, capture the user's platforms +\npaths + tracker entity into memory so later runs are pre-parameterized. Platform facts carry a `verified\n\u003Cmonth>` stamp — re-verify any teardown\u002Fbilling fact against current docs before betting money or data.\n\n## Limitations\n\n- Does not replace a real cloud orchestrator or managed provisioner; use it to make rented-box work survivable, not to optimize multi-cloud procurement.\n- Platform billing, stop, destroy, and data-retention behavior can drift; re-check current provider docs before destructive or money-impacting actions.\n- Requires user-owned credentials, SSH\u002FAPI access, and explicit confirmation before teardown, deletion, or other irreversible cleanup.\n- Companion skills named above are not bundled here; treat them as optional references unless installed in the current agent environment.\n\n## Bundled resources\n\nLoad only what the current phase needs.\n\n- `references\u002Fprinciples.md` — the 10 invariants expanded, with the cross-platform nuance behind each.\n- `references\u002Flifecycle_checklist.md` — the 6-phase runbook as a per-platform checklist.\n- `references\u002Fgotchas_universal.md` — universal + mixed gotchas (TOC + grep index at top).\n- `references\u002Fmonitoring_patterns.md` — the four-layer durable-monitoring architecture + robust ssh-poll template.\n- `references\u002Fssh_transport.md` — ssh config, rsync\u002Fscp resumable patterns, secrets-via-stdin, CRLF, two-SSH-flavor caveat.\n- `references\u002Fchina-network.md` — mirrors table + HF_ENDPOINT + resumable-download ladder + the `no_proxy` trap (all CN platforms).\n- `references\u002Fspot-resilience.md` — preemption signals, Young\u002FDaly checkpoint cadence, atomic-write resume.\n- `references\u002Fparallel_ablation.md` — FS-shared fan-out + the independence predicate + reconciliation.\n- `references\u002Fmultinode.md` — (advanced) NCCL \u002F fabric-manager \u002F elastic-training gotchas; single-box users skip.\n- `references\u002Ftraining\u002F` — the **DL-training debug layer** (8 files: oom-memory, distributed-launch, precision-stability, throughput-profiling, checkpoint-resume, by-domain, convergence-debugging, data-pipeline) — see \"When training breaks\" above.\n- `references\u002Fself-improvement.md` — the feedback loop: capture a new gotcha (at a bar) into memory or the catalog, personalize on first run, keep platform facts fresh.\n- `scripts\u002F` — wrapper templates (`run_one`\u002F`run_queue`), monitors (`mem_monitor`, `gpu_health`, `reap_vram_zombies`), the read-only patrol (`health_patrol.sh.template`), transfer\u002Faggregation (`download_loop`, `aggregate_to_fs`, `setup-china-mirrors`), the load-and-verify checker (`verify_local.py`), and the `verified`-stamp freshness linter (`check_staleness.py`).\n- `profiles\u002F\u003Cplatform>.md` — the per-platform substrate (one per platform; `_schema.md` defines the 8 fields).\n- `examples\u002Fautodl_sweep\u002F` — one complete, runnable worked case end to end.\n",{"name":5,"risk":53,"source":40,"license":14,"category":54,"date_added":55,"description":7,"source_repo":56,"source_type":40,"compatibility":57,"license_source":58},"safe","ml-ops","2026-06-20","Hanyuyuan6\u002Fremote-gpu-trainer","Any Agent-Skills (SKILL.md)-compatible agent — Claude Code, Codex, Cursor, Trae, Gemini CLI, etc.\nNeeds a shell + SSH (or a platform CLI\u002FAPI) to drive the remote box; scripts are bash\u002Fpython. A few\ndurable-monitoring recipes assume a host background-task runner + scheduler — map to the running\nagent's equivalents (references\u002Fmonitoring_patterns.md §7). Companion skills (verifying-dl-experiments,\nsuperpowers:*, huggingface-skills:*) are optional separate installs.","https:\u002F\u002Fgithub.com\u002FHanyuyuan6\u002Fremote-gpu-trainer\u002Fblob\u002Fmain\u002FLICENSE","f3312ba749561b6159f477dbfc9313c875d40a56",".agents\u002Fskills_archive\u002Fremote-gpu-trainer","https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web","https:\u002F\u002Fgithub.com\u002Fduc-duy-1612","2026-10-25 23:50:58",{"source":8,"skillName":5,"commands":65,"prompt":75,"zipAvailable":76},[66,69,72],{"tool":67,"command":68},"npx","npx skills add duc-duy-1612\u002FOnline-Delivery-Systems-Web --skill remote-gpu-trainer",{"tool":70,"command":71},"bunx","bunx skills add duc-duy-1612\u002FOnline-Delivery-Systems-Web --skill remote-gpu-trainer",{"tool":73,"command":74},"pnpm","pnpm dlx skills add duc-duy-1612\u002FOnline-Delivery-Systems-Web --skill remote-gpu-trainer","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002FSKILL.md",true,[78,84,88,92,98,102,108,112,116,121,125,129,133,137,141,145,149,153,157,161,165,169,173,177,181,185,189,193,197,201,205,209,213,217,221,225,229,233,237,241,245,249,253,257,261,265,269],{"path":79,"resourceType":80,"resourceTypeDesc":81,"sizeBytes":82,"sourceHtmlUrl":83},".gitattributes","OTHER","其他",332,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002F.gitattributes",{"path":85,"resourceType":80,"resourceTypeDesc":81,"sizeBytes":86,"sourceHtmlUrl":87},".gitignore",415,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002F.gitignore",{"path":89,"resourceType":80,"resourceTypeDesc":81,"sizeBytes":90,"sourceHtmlUrl":91},"evals\u002Fcases.jsonl",7302,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Fevals\u002Fcases.jsonl",{"path":93,"resourceType":94,"resourceTypeDesc":95,"sizeBytes":96,"sourceHtmlUrl":97},"evals\u002FREADME.md","REFERENCE","参考文档",3161,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Fevals\u002FREADME.md",{"path":99,"resourceType":94,"resourceTypeDesc":95,"sizeBytes":100,"sourceHtmlUrl":101},"evals\u002FRESULTS.md",3724,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Fevals\u002FRESULTS.md",{"path":103,"resourceType":104,"resourceTypeDesc":105,"sizeBytes":106,"sourceHtmlUrl":107},"evals\u002Frun_evals.py","SCRIPT","脚本",2521,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Fevals\u002Frun_evals.py",{"path":109,"resourceType":94,"resourceTypeDesc":95,"sizeBytes":110,"sourceHtmlUrl":111},"examples\u002Fautodl_sweep\u002Fqueue_1.txt",418,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Fexamples\u002Fautodl_sweep\u002Fqueue_1.txt",{"path":113,"resourceType":94,"resourceTypeDesc":95,"sizeBytes":114,"sourceHtmlUrl":115},"examples\u002Fautodl_sweep\u002FREADME.md",4241,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Fexamples\u002Fautodl_sweep\u002FREADME.md",{"path":117,"resourceType":117,"resourceTypeDesc":118,"sizeBytes":119,"sourceHtmlUrl":120},"LICENSE","许可证",1067,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002FLICENSE",{"path":122,"resourceType":94,"resourceTypeDesc":95,"sizeBytes":123,"sourceHtmlUrl":124},"profiles\u002F_schema.md",5078,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Fprofiles\u002F_schema.md",{"path":126,"resourceType":94,"resourceTypeDesc":95,"sizeBytes":127,"sourceHtmlUrl":128},"profiles\u002Fautodl.md",23189,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Fprofiles\u002Fautodl.md",{"path":130,"resourceType":94,"resourceTypeDesc":95,"sizeBytes":131,"sourceHtmlUrl":132},"profiles\u002Fchina.md",30238,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Fprofiles\u002Fchina.md",{"path":134,"resourceType":94,"resourceTypeDesc":95,"sizeBytes":135,"sourceHtmlUrl":136},"profiles\u002Fgeneric-ssh.md",36186,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Fprofiles\u002Fgeneric-ssh.md",{"path":138,"resourceType":94,"resourceTypeDesc":95,"sizeBytes":139,"sourceHtmlUrl":140},"profiles\u002Flambda.md",25360,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Fprofiles\u002Flambda.md",{"path":142,"resourceType":94,"resourceTypeDesc":95,"sizeBytes":143,"sourceHtmlUrl":144},"profiles\u002Fpaperspace.md",26900,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Fprofiles\u002Fpaperspace.md",{"path":146,"resourceType":94,"resourceTypeDesc":95,"sizeBytes":147,"sourceHtmlUrl":148},"profiles\u002Frunpod.md",26215,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Fprofiles\u002Frunpod.md",{"path":150,"resourceType":94,"resourceTypeDesc":95,"sizeBytes":151,"sourceHtmlUrl":152},"profiles\u002Fvastai.md",27133,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Fprofiles\u002Fvastai.md",{"path":154,"resourceType":94,"resourceTypeDesc":95,"sizeBytes":155,"sourceHtmlUrl":156},"README.md",16556,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002FREADME.md",{"path":158,"resourceType":94,"resourceTypeDesc":95,"sizeBytes":159,"sourceHtmlUrl":160},"references\u002Fchina-network.md",11606,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Freferences\u002Fchina-network.md",{"path":162,"resourceType":94,"resourceTypeDesc":95,"sizeBytes":163,"sourceHtmlUrl":164},"references\u002Fgotchas_universal.md",44724,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Freferences\u002Fgotchas_universal.md",{"path":166,"resourceType":94,"resourceTypeDesc":95,"sizeBytes":167,"sourceHtmlUrl":168},"references\u002Flifecycle_checklist.md",11339,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Freferences\u002Flifecycle_checklist.md",{"path":170,"resourceType":94,"resourceTypeDesc":95,"sizeBytes":171,"sourceHtmlUrl":172},"references\u002Fmonitoring_patterns.md",22337,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Freferences\u002Fmonitoring_patterns.md",{"path":174,"resourceType":94,"resourceTypeDesc":95,"sizeBytes":175,"sourceHtmlUrl":176},"references\u002Fmultinode.md",12716,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Freferences\u002Fmultinode.md",{"path":178,"resourceType":94,"resourceTypeDesc":95,"sizeBytes":179,"sourceHtmlUrl":180},"references\u002Fparallel_ablation.md",11138,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Freferences\u002Fparallel_ablation.md",{"path":182,"resourceType":94,"resourceTypeDesc":95,"sizeBytes":183,"sourceHtmlUrl":184},"references\u002Fprinciples.md",10784,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Freferences\u002Fprinciples.md",{"path":186,"resourceType":94,"resourceTypeDesc":95,"sizeBytes":187,"sourceHtmlUrl":188},"references\u002Fself-improvement.md",4975,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Freferences\u002Fself-improvement.md",{"path":190,"resourceType":94,"resourceTypeDesc":95,"sizeBytes":191,"sourceHtmlUrl":192},"references\u002Fspot-resilience.md",14305,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Freferences\u002Fspot-resilience.md",{"path":194,"resourceType":94,"resourceTypeDesc":95,"sizeBytes":195,"sourceHtmlUrl":196},"references\u002Fssh_transport.md",14151,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Freferences\u002Fssh_transport.md",{"path":198,"resourceType":94,"resourceTypeDesc":95,"sizeBytes":199,"sourceHtmlUrl":200},"references\u002Ftraining\u002Fby-domain.md",31452,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Freferences\u002Ftraining\u002Fby-domain.md",{"path":202,"resourceType":94,"resourceTypeDesc":95,"sizeBytes":203,"sourceHtmlUrl":204},"references\u002Ftraining\u002Fcheckpoint-resume.md",24419,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Freferences\u002Ftraining\u002Fcheckpoint-resume.md",{"path":206,"resourceType":94,"resourceTypeDesc":95,"sizeBytes":207,"sourceHtmlUrl":208},"references\u002Ftraining\u002Fconvergence-debugging.md",39845,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Freferences\u002Ftraining\u002Fconvergence-debugging.md",{"path":210,"resourceType":94,"resourceTypeDesc":95,"sizeBytes":211,"sourceHtmlUrl":212},"references\u002Ftraining\u002Fdata-pipeline.md",24057,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Freferences\u002Ftraining\u002Fdata-pipeline.md",{"path":214,"resourceType":94,"resourceTypeDesc":95,"sizeBytes":215,"sourceHtmlUrl":216},"references\u002Ftraining\u002Fdistributed-launch.md",25986,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Freferences\u002Ftraining\u002Fdistributed-launch.md",{"path":218,"resourceType":94,"resourceTypeDesc":95,"sizeBytes":219,"sourceHtmlUrl":220},"references\u002Ftraining\u002Foom-memory.md",22546,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Freferences\u002Ftraining\u002Foom-memory.md",{"path":222,"resourceType":94,"resourceTypeDesc":95,"sizeBytes":223,"sourceHtmlUrl":224},"references\u002Ftraining\u002Fprecision-stability.md",22083,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Freferences\u002Ftraining\u002Fprecision-stability.md",{"path":226,"resourceType":94,"resourceTypeDesc":95,"sizeBytes":227,"sourceHtmlUrl":228},"references\u002Ftraining\u002Fthroughput-profiling.md",28433,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Freferences\u002Ftraining\u002Fthroughput-profiling.md",{"path":230,"resourceType":104,"resourceTypeDesc":105,"sizeBytes":231,"sourceHtmlUrl":232},"scripts\u002Faggregate_to_fs.sh",2189,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Fscripts\u002Faggregate_to_fs.sh",{"path":234,"resourceType":104,"resourceTypeDesc":105,"sizeBytes":235,"sourceHtmlUrl":236},"scripts\u002Fcheck_staleness.py",2873,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Fscripts\u002Fcheck_staleness.py",{"path":238,"resourceType":104,"resourceTypeDesc":105,"sizeBytes":239,"sourceHtmlUrl":240},"scripts\u002Fdownload_loop.sh",3090,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Fscripts\u002Fdownload_loop.sh",{"path":242,"resourceType":104,"resourceTypeDesc":105,"sizeBytes":243,"sourceHtmlUrl":244},"scripts\u002Fgpu_health.sh",9234,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Fscripts\u002Fgpu_health.sh",{"path":246,"resourceType":104,"resourceTypeDesc":105,"sizeBytes":247,"sourceHtmlUrl":248},"scripts\u002Fhealth_patrol.sh.template",4544,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Fscripts\u002Fhealth_patrol.sh.template",{"path":250,"resourceType":104,"resourceTypeDesc":105,"sizeBytes":251,"sourceHtmlUrl":252},"scripts\u002Fmem_monitor.sh",2997,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Fscripts\u002Fmem_monitor.sh",{"path":254,"resourceType":104,"resourceTypeDesc":105,"sizeBytes":255,"sourceHtmlUrl":256},"scripts\u002Freap_vram_zombies.sh",7941,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Fscripts\u002Freap_vram_zombies.sh",{"path":258,"resourceType":104,"resourceTypeDesc":105,"sizeBytes":259,"sourceHtmlUrl":260},"scripts\u002Frun_one.sh.template",5500,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Fscripts\u002Frun_one.sh.template",{"path":262,"resourceType":104,"resourceTypeDesc":105,"sizeBytes":263,"sourceHtmlUrl":264},"scripts\u002Frun_queue.sh.template",3911,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Fscripts\u002Frun_queue.sh.template",{"path":266,"resourceType":104,"resourceTypeDesc":105,"sizeBytes":267,"sourceHtmlUrl":268},"scripts\u002Fsetup-china-mirrors.sh",2084,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Fscripts\u002Fsetup-china-mirrors.sh",{"path":270,"resourceType":104,"resourceTypeDesc":105,"sizeBytes":271,"sourceHtmlUrl":272},"scripts\u002Fverify_local.py",6493,"https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fremote-gpu-trainer\u002Fscripts\u002Fverify_local.py",[274,301,324,348,371,394],{"skillNo":275,"name":276,"nameZh":6,"description":277,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":278,"categories":279,"keywords":283,"publishedAt":48},"01M6Q0FA5FZ410Z33Z3TFQ4DBA","zero-trust","Implement zero-trust network architecture. Configure identity-based access, micro-segmentation, and continuous verification. Use when implementing modern security architectures.","https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fzero-trust\u002FSKILL.md",[280],{"categoryNo":281,"categoryName":282,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000011","运维与部署",[284,285,286,287,288,289,276,290,291,292,293,294,40,37,295,296,297,298,299,300],"ZeRO","trust","zero trust","architecture","assessment","verification","segmentation","devops-security","access","configure","only","active","critical","scope","scripts","relevant","implementing",{"skillNo":302,"name":303,"nameZh":6,"description":304,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":305,"categories":306,"keywords":307,"publishedAt":48},"01M6Q0F1YZJW3KX5FBKEVZ8CV4","zcode-delegate","Delegate coding tasks to the Z.AI ZCode CLI only when the user explicitly requests it, while the orchestrator retains review and landing responsibility.","https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fzcode-delegate\u002FSKILL.md",[],[308,309,310,311,312,313,314,315,43,316,317,318,319,294,320,40,321,322,296,323],"delegate","zcode","zcode delegate","review","cli","orchestration","tasks","run","model","node","read","orchestrator","files","desktop app","app","agent-orchestration",{"skillNo":325,"name":326,"nameZh":6,"description":327,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":328,"categories":329,"keywords":330,"publishedAt":48},"01M6Q0ETK41VGNMYEXW0T00AJN","youtube-transcript","Fetch YouTube transcripts through DeepAPI or local fallback tooling and save clean text output.","https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fyoutube-transcript\u002FSKILL.md",[],[331,332,333,53,334,335,336,337,338,339,340,341,342,40,37,343,344,345,346,347],"transcript","youtube transcript","youtube","github","codex","research","license","https","text","fetch","tooling","clean","local","output","deepapi","youtube transcripts","blob",{"skillNo":349,"name":350,"nameZh":6,"description":351,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":352,"categories":353,"keywords":361,"publishedAt":48},"01M6Q0EPPH2PVMDW40017F8WX0","youtube-transcript-skills","Fetch YouTube video transcripts, search videos\u002Fchannels, browse channels, and extract playlists via the getyoutubetranscript.com API - free tier, no card required.","https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fyoutube-transcript-skills\u002FSKILL.md",[354,357,360],{"categoryNo":355,"categoryName":356,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000010","后端与接口",{"categoryNo":358,"categoryName":359,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000016","图像与多媒体",{"categoryNo":23,"categoryName":24,"taxonomyType":25,"taxonomyTypeDesc":26},[331,332,333,362,53,363,334,335,337,364,44,365,366,340,367,40,37,368,369,370],"youtube transcript skills","gemini","search","Video","browse","api-integration","free","card","channels",{"skillNo":372,"name":373,"nameZh":6,"description":374,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":375,"categories":376,"keywords":381,"publishedAt":48},"01M6Q0EAB96PX83W5PTE8A0R8D","youtube-notetaker","Turn YouTube talks into local study notes with slides, transcripts, editable annotations, and a markdown-backed viewer.","https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fyoutube-notetaker\u002FSKILL.md",[377,380],{"categoryNo":378,"categoryName":379,"taxonomyType":20,"taxonomyTypeDesc":21},"01JBSEEDCAT000000000000003","演示文稿",{"categoryNo":358,"categoryName":359,"taxonomyType":20,"taxonomyTypeDesc":21},[333,382,383,312,53,384,385,386,47,337,365,387,388,389,390,37,391,343,392,393],"youtube notetaker","notetaker","workflow","official","notes","markdown","slides","annotations","README","study","dair-academy","viewer",{"skillNo":395,"name":396,"nameZh":6,"description":397,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":6,"repositoryNo":12,"starsCount":13,"licenseSpdxId":14,"licenseFileUrl":6,"sourceHtmlUrl":398,"categories":399,"keywords":401,"publishedAt":48},"01M6Q0E55F0GW5MX48V8HXS8B5","youtube-full","Fetch YouTube transcripts, search videos, browse channels, and extract playlists via TranscriptAPI — no yt-dlp, no Google API key, works from any cloud server.","https:\u002F\u002Fgithub.com\u002Fduc-duy-1612\u002FOnline-Delivery-Systems-Web\u002Fblob\u002Fmain\u002F.agents\u002Fskills_archive\u002Fyoutube-full\u002FSKILL.md",[400],{"categoryNo":355,"categoryName":356,"taxonomyType":20,"taxonomyTypeDesc":21},[402,333,403,404,53,363,364,405,406,44,365,366,340,407,367,40,37,368,408,346],"youtube full","full","google","server","dlp","credits","automatic"]