[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M75KVSYK7PRSNVJ7FT0CS92G":3,"skill-same:01M75KVSYK7PRSNVJ7FT0CS92G":57},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":29,"commitSha":30,"skillDirectory":31,"forksCount":32,"repositoryHtmlUrl":33,"repositoryHomepageUrl":6,"ownerHtmlUrl":34,"ownerBlogUrl":35,"ownerDescription":6,"updatedAt":36,"install":37,"resources":50},"01M75KVSYK7PRSNVJ7FT0CS92G","kermt-continue-pretrain",null,"Continue pretraining from an existing KERMT checkpoint. The skill validates the user's checkpoint and pretrain CSV, prepares the data into shard\u002Fvocab\u002Ffeatures form, then launches pretrain_ddp.py inside the kermt container (detached for long runs). Auto-dispatches `--pretrain_mode` based on the checkpoint type (grover_base vocab-only, cmim, or hybrid).","stanfish06\u002Fskillquarium","stanfish06","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F93099613?v=4","01M75F1CK4ZW109ZK3T26VN52F","Zhiyuan","01M75F1CM53VH6F79DZ8GXKKES",12,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fstanfish06\u002Fskillquarium\u002Fblob\u002Fmaster\u002Fskills\u002Fkermt-continue-pretrain\u002FSKILL.md",[],[],"2026-11-12 13:55:43",false,"---\nname: kermt-continue-pretrain\ndescription: Continue pretraining from an existing KERMT checkpoint. The skill validates the user's checkpoint and pretrain CSV, prepares the data into shard\u002Fvocab\u002Ffeatures form, then launches pretrain_ddp.py inside the kermt container (detached for long runs). Auto-dispatches `--pretrain_mode` based on the checkpoint type (grover_base vocab-only, cmim, or hybrid).\nlicense: Apache-2.0\ncompatibility: Requires docker, nvidia-container-toolkit, and a CUDA-capable NVIDIA GPU. Designed for Claude Code, Codex, and Nemotron.\nmetadata:\n  owner: evax@nvidia.com\n  classification: workflow-skill\n  risk_tier: skill\n# Line\u002Ftoken budget: this file is targeted at ~250 lines \u002F ~3000 tokens —\n# well within the 500-line \u002F 5000-token cap. Long examples live in\n# agent\u002Fscripts\u002Frun_pretrain_local.py's docstring.\n---\n\n# kermt-continue-pretrain\n\nContinue pretraining from a user-supplied KERMT checkpoint (grover_base \u002F\ncmim \u002F hybrid). The skill is the workflow orchestrator: it validates inputs,\nprepares the corpus, launches the runner, and returns a run directory.\n\n## Hardware requirements\n\n- **GPUs**: 1–N CUDA-capable NVIDIA GPUs. The runner auto-detects via\n  `torch.cuda.device_count()`; `--gpus 0,2` overrides. On a single GPU the\n  runner falls back to `--batch_size 32 --save_interval 500`; on multi-GPU\n  it uses the `defaults_pretrain.json` values (currently `batch_size 256`).\n  Note: `--gpus N` uses **torch.cuda** indexing, which can differ from\n  `nvidia-smi`'s display order on multi-GPU hosts (PCI bus vs. CUDA\n  enumeration). To target a specific physical GPU, set `CUDA_VISIBLE_DEVICES`\n  before invoking, or run\n  `python -c \"import torch; print([torch.cuda.get_device_name(i) for i in range(torch.cuda.device_count())])\"`\n  to confirm which device you're picking.\n- **VRAM**: the default `--batch-size 256` is sized for A100-class hardware\n  (80 GB VRAM). On smaller GPUs, downscale to avoid OOM:\n\n  | GPU class                | VRAM       | Suggested `--batch-size` |\n  |--------------------------|------------|--------------------------|\n  | L4, T4, V100 16 GB       | 16–24 GB   | 32–64                    |\n  | A100 40 GB, L40, A40     | 40–48 GB   | 128                      |\n  | A100 80 GB, H100, H200   | 80 GB      | 256 (default)            |\n\n  These are rough starting points — pass `--batch-size N` to override.\n- **Disk**: tens of GB depending on corpus size + epochs (each checkpoint\n  is several hundred MB).\n- **Driver \u002F CUDA**: any host supporting CUDA 12.6 (the kermt image base).\n  `kermt-setup` validates this up-front.\n\n## Inputs\n\nRequired:\n\n- `--csv \u003Cpath>` — the pretrain CSV (single column `smiles`). If you have\n  separate train\u002Fval CSVs, pass `--val-csv \u003Cpath>` too.\n\nCheckpoint (optional — defaults to the released model if omitted):\n\n- `--ckpt \u003Cpath>` — the input pretrain checkpoint to continue from. Must be\n  a grover_base (with vocab heads), cmim, or hybrid ckpt; the validator\n  rejects everything else with a redirect to the correct workflow. **If\n  omitted**, the skill offers to download the released pretrained hybrid model\n  **nvidia\u002FNV-KERMT-70M-v2** and continue-pretrain from it — see \"Resolve &\n  validate the checkpoint\" (workflow step 3). The released bundle ships its\n  three vocab files alongside the ckpt, so the authoritative-vocab pass-through\n  (step 5) works automatically.\n- `--pretrained-release` — explicit opt-in to use the released model without\n  the interactive prompt (for non-interactive \u002F agent runs). Mutually\n  exclusive with `--ckpt`.\n- `--model-dir \u003Cdir>` — where to save the downloaded bundle (default\n  `$KERMT_REPO\u002Fmodels\u002FNV-KERMT-70M-v2\u002F`). An already-complete bundle there is\n  reused, not re-downloaded.\n\nOptional:\n\n- `--val-csv \u003Cpath>` — separate validation CSV. Without it, the prep step\n  auto-splits the input by `--val-frac 0.1` (random shuffle with `--seed`).\n- `--epochs N` \u002F `--batch-size N` \u002F `--init-lr F` \u002F `--max-lr F` \u002F\n  `--final-lr F` \u002F `--warmup-epochs F` \u002F `--weight-decay F` \u002F `--dropout F` \u002F\n  `--save-interval N` \u002F `--seed N` — training-hyperparameter overrides.\n  Anything not given is filled from `agent\u002Fconfig\u002Fdefaults_pretrain.json`.\n- `--vocab-loss-weight F` (hybrid only) \u002F `--latent-dim N` \u002F\n  `--contrastive-temperature F` (cmim and hybrid only) — loss \u002F decoder\n  overrides.\n- `--wandb-project NAME` \u002F `--wandb-run-name NAME` — optional Weights & Biases\n  logging. When `--wandb-project` is set, rank 0 logs train\u002Fval losses; the run\n  name is honored only alongside a project. Off by default. (Independent of the\n  ckpt's `wandb_run_id` continuity handling under `--resume`.)\n- `--resume` — see \"Modes\" section below.\n- `--gpus 0,2` — restrict to a GPU subset. Default uses all visible GPUs.\n- `--from-prepare \u003Cdir>` — skip the prepare step and reuse an existing\n  `prepare_data.json` in `\u003Cdir>`. Useful when iterating on hyperparameters.\n\n## Modes\n\nThe runner has two modes for ingesting the input ckpt, dispatched on whether\n`--resume` is set. Pick based on intent:\n\n### Default (fresh-schedule continue-pretrain)\n\n**Use when**: you have a finished pretrain ckpt and want to continue training\nit — on a new corpus, with a different objective, or just for more epochs\nthan its original plan. The previous training's step counter and schedule\nshape are no longer relevant; you want a new learning-rate schedule for the\nnew run.\n\n**What gets loaded from the ckpt**:\n- ✓ Model weights (encoder + vocab heads + contrast head + decoder, whatever\n  is there)\n- ✓ Optimizer state (Adam's running m1\u002Fm2 moments — warm-starts the new\n  schedule so the first few hundred steps aren't dominated by noisy\n  gradient-estimate startup)\n- ✗ Scheduler step counter (reset to 0)\n- ✗ Epoch counter (reset to 0)\n- ✗ Batch counter (reset to 0)\n- ✗ wandb run id (new wandb run, not a continuation)\n\n**Schedule shape** (init\u002Fmax\u002Ffinal LR, warmup epochs, total epochs): from\nyour CLI args or `defaults_pretrain.json`. A fresh NoamLR is constructed\nfrom these values and starts at step 0.\n\n### `--resume` (true resume)\n\n**Use when**: a previous run was interrupted (crash, OOM, Ctrl-C) and you\nwant to pick up exactly where it left off — same dataset, same schedule,\nsame training trajectory.\n\n**What gets loaded from the ckpt**: **everything** in the\n`save_model_for_restart` format. Model weights + optimizer state +\nscheduler_step + epoch + batch_idx + wandb_run_id are all restored. The\nnew run continues from the saved step in the saved schedule (which is\nrecovered from the ckpt's `saved_args`). Mid-epoch resume works too —\n`pretrain_ddp.py`'s sampler skip-count picks up at the saved batch index\nwithin the saved epoch.\n\n**Schedule shape**: inherited from the ckpt's `saved_args`. CLI overrides\nof any schedule flag (`--epochs \u002F --warmup-epochs \u002F --init-lr \u002F --max-lr \u002F\n--final-lr`) are **rejected with a hard error** — pure resume means pure\nresume; if you want to change the schedule, drop `--resume` and start a\nfresh-schedule run.\n\n**Requirements**: the ckpt must have been saved via `save_model_for_restart`\n(i.e., carry `optimizer \u002F scheduler_step \u002F epoch \u002F batch_idx` keys). If\nany of these is missing, the runner errors with a clear message and\nsuggests dropping `--resume`.\n\nThe default mode is the right choice ~90% of the time. Reach for `--resume`\nonly when you genuinely need to continue a single interrupted training\nrun.\n\n## Workflow\n\nLet `$KERMT_REPO` be the path to your kermt repo checkout, and assume\n`kermt-setup` has already built `kermt:latest`. All paths below are on the\nhost; the helper bind-mounts them at known container paths.\n\n1. **Pre-flight: ensure container + system probe.**\n   ```\n   $KERMT_REPO\u002Fagent\u002Fscripts\u002Fkermt_container.sh check_system | python -c \"\n   import json, sys; d = json.load(sys.stdin)\n   if not d['ok']:\n       print('System check failed:', d['gaps']); sys.exit(1)\n   print(f'OK: {len(d[\\\"gpus\\\"])} GPU(s); {d[\\\"disk\\\"][\\\"free_gb\\\"]} GB free; CUDA via container toolkit')\n   \"\n   ```\n   Surface any `gaps` to the user. Refuse to proceed if `ok: false`.\n\n2. **Compute run directory.**\n   ```\n   RUN_DIR=$KERMT_REPO\u002Fruns\u002Fcontinue-pretrain_$(date -u +%Y-%m-%dT%H-%M-%SZ)\n   ```\n\n3. **Resolve & validate the checkpoint.**\n\n   **Resolve — only if `--ckpt` was omitted.** Default to the released\n   pretrained hybrid model **nvidia\u002FNV-KERMT-70M-v2**:\n   - **Consent gate.** Unless `--pretrained-release` was passed, ask the user:\n     \"No checkpoint given — download the released model nvidia\u002FNV-KERMT-70M-v2\n     (NVIDIA Open Model License, https:\u002F\u002Fhuggingface.co\u002Fnvidia\u002FNV-KERMT-70M-v2)\n     and continue-pretrain from it? [y\u002FN]\". **Never download without an\n     explicit yes** (or `--pretrained-release`). If both `--ckpt` and\n     `--pretrained-release` are given, abort — they conflict.\n   - **Save location.** Default `$KERMT_REPO\u002Fmodels\u002FNV-KERMT-70M-v2\u002F`; honor\n     `--model-dir \u003Cdir>` if given. An already-complete bundle is reused.\n   - **Download** (foreground; ~282 MB on first fetch):\n     ```\n     $KERMT_REPO\u002Fagent\u002Fscripts\u002Fkermt_container.sh run --model-dir \u003Csave-dir> -- \\\n         \"python agent\u002Fscripts\u002Ffetch_released_model.py --out \u002Fmodel\"\n     ```\n     Parse the JSON; abort on `ok: false` (surface `errors`). On success set\n     `\u003Cuser-ckpt> = \u003Csave-dir>\u002Fkermt_contrastive_v2.0.pt`. The bundle's three\n     vocab files land in `\u003Csave-dir>` too, so step 5's `--vocab-dir`\n     auto-detection (which looks in the ckpt's parent directory) finds them\n     with no extra work.\n\n   **Validate** the resolved (or user-provided) ckpt:\n   ```\n   $KERMT_REPO\u002Fagent\u002Fscripts\u002Fkermt_container.sh run --ckpt \u003Cuser-ckpt> -- \\\n       \"python agent\u002Fscripts\u002Fcheck_checkpoint.py --mode continue_pretrain --ckpt \u002Fckpt\"\n   ```\n   Parse the JSON. Abort on `ok: false`, showing the error verbatim. The error\n   message redirects the user to `kermt-add-cmim-pretrain` for encoder-only\n   ckpts, or to `kermt-finetune` for finetuned ckpts.\n\n4. **Validate the data.**\n   ```\n   $KERMT_REPO\u002Fagent\u002Fscripts\u002Fkermt_container.sh run --data \u003Cuser-csv> -- \\\n       \"python agent\u002Fscripts\u002Fcheck_data.py --mode pretrain --csv \u002Fdata\u002F\u003Cbasename>\"\n   ```\n   Abort on `ok: false`.\n\n5. **Prepare the data** (skip if `--from-prepare` given).\n   **Pass the ckpt's vocab through.** Look in the ckpt's parent directory for\n   the conventional `pretrain_atom_vocab.{json,pkl}`, `pretrain_bond_vocab.{json,pkl}`,\n   and `pretrain_smiles_vocab.pkl` files (the bundling convention for released\n   models; see `agent\u002FREADME.md` \"Released models\" section). If all three are\n   present, auto-pass via `--vocab-dir \u003Cckpt_parent_dir>`. If only some are\n   present, pass them via explicit flags (`--atom-vocab`, `--bond-vocab`,\n   `--smiles-vocab`). If none are present, ask the user for `--vocab-dir` — or\n   refuse to proceed, because rebuilding a fresh vocab from the new corpus\n   would silently mismatch the ckpt's vocab heads (the ckpt's vocab is\n   authoritative for continue-pretrain).\n\n   Note the **two-layer mount pattern**: pass the host directory to\n   `kermt_container.sh --vocab-dir` (which mounts it at `\u002Fvocab` inside the\n   container), and reference `\u002Fvocab` from the inner `prepare_data.py`\n   command. The same pattern applies to every host path the inner command\n   needs to read (`--data \u003Chost-csv>` → `\u002Fdata\u002F\u003Cbasename>`,\n   `--ckpt \u003Chost-ckpt>` → `\u002Fckpt`).\n\n   ```\n   VOCAB_DIR=$(dirname \u003Cuser-ckpt>)\n   $KERMT_REPO\u002Fagent\u002Fscripts\u002Fkermt_container.sh run \\\n       --data \u003Cuser-csv> --vocab-dir $VOCAB_DIR --run-dir $RUN_DIR -- \\\n       \"python agent\u002Fscripts\u002Fprepare_data.py --mode pretrain \\\\\n            --csv \u002Fdata\u002F\u003Cbasename> --out \u002Fruns\u002Fdata \\\\\n            --vocab-dir \u002Fvocab \\\\\n            [--val-csv \u002Fdata\u002F\u003Cval-basename>] [--val-frac 0.1] [--seed 0]\"\n   ```\n   Outputs land at `$RUN_DIR\u002Fdata\u002Fprepare_data.json` with\n   `vocab_source: \"user_provided\"`. The runner step 7 will verify the vocab\n   files' entry counts match the ckpt's vocab-head sizes and refuse to launch\n   on mismatch.\n\n6. **Estimate runtime + confirm with user.**\n   - Pretrain wall time depends on corpus size × epochs × GPU count.\n   - Tell the user the estimate; ask \"proceed?\" unless `--yes` flag was given\n     (agent-non-interactive case).\n   - Example estimate template:\n     `~N hours on K GPUs for E epochs over M molecules (~steps\u002Fepoch × seconds\u002Fstep)`.\n\n7. **Launch the runner detached.**\n   ```\n   $KERMT_REPO\u002Fagent\u002Fscripts\u002Fkermt_container.sh run_detached \\\\\n       --name kermt-continue-pretrain-\u003Cts> \\\\\n       --ckpt \u003Cuser-ckpt> --run-dir $RUN_DIR -- \\\\\n       \"python agent\u002Fscripts\u002Frun_pretrain_local.py \\\\\n            --ckpt \u002Fckpt \\\\\n            --prepare-manifest \u002Fruns\u002Fdata\u002Fprepare_data.json \\\\\n            --out \u002Fruns \\\\\n            [--epochs N --batch-size N --init-lr F ...]\"\n   ```\n   Returns the container name + id + log file path.\n\n8. **Report to the user.** Output a short summary:\n   - Container name + id\n   - `$RUN_DIR\u002Frun.json` (the manifest with cmd_replay + image digest)\n   - Log file: `$RUN_DIR\u002Flogs\u002Fpretrain_ddp.log`\n   - TensorBoard: `$RUN_DIR\u002Flogs\u002Ftb` (open with `tensorboard --logdir\n     $RUN_DIR\u002Flogs\u002Ftb`)\n   - Suggest invoking `kermt-monitor \u003CRUN_DIR>` to check progress.\n\n## Hard rules\n\n- **Never download the released model without consent.** When `--ckpt` is\n  omitted, download `nvidia\u002FNV-KERMT-70M-v2` only after an explicit user \"yes\"\n  or an explicit `--pretrained-release` flag. `--ckpt` and\n  `--pretrained-release` are mutually exclusive.\n- **Never modify the user's input ckpt.** The runner symlinks it into the\n  save_dir; the symlink is what pretrain_ddp.py auto-resumes from. The\n  source file stays untouched.\n- **Never silently override arch.** If the user passes a `--hidden-size`\n  etc. that doesn't match the ckpt-derived value, the runner aborts loudly.\n  Arch params come from the ckpt, period.\n- **Never block on the long-running pretrain itself.** The runner is invoked\n  via `run_detached`; the skill returns immediately after step 8. Use\n  `kermt-monitor` for progress.\n- **Echo applied defaults back to the user.** The `args_applied` field of\n  `run.json` records every flag's value + source (user \u002F default-config \u002F\n  auto-1gpu \u002F auto-multi-gpu). Skill should surface a summary of any flag\n  not user-specified so the user knows what was assumed.\n\n## Common errors\n\n- `model_type='finetuned'` rejected → the ckpt is a downstream finetune,\n  not a pretrain. The error redirects to the relevant workflow.\n- `grover_base ckpt has no vocab head` → encoder-only ckpt (e.g. the\n  original-grover `grover_base.pt`). The error redirects to\n  `kermt-add-cmim-pretrain`.\n- `prepare_data manifest is missing required outputs` → user passed\n  `--from-prepare` to a directory where prepare was run with `--skip-vocab`\n  or `--skip-split`. Re-run prepare without those flags.\n- `--gpus all` not available → install `nvidia-container-toolkit`; check\n  `kermt_container.sh check_system`.\n\n## Replayability\n\nThe `run.json` `cmd_replay` field is a single-line command that re-runs the\npretrain with the same inputs, hyperparameters, and arch. To replay:\n\n```bash\n# Inside the kermt container:\n$(jq -r .cmd_replay $RUN_DIR\u002Frun.json)\n```\n\nIf `ok_to_replay: false` in the manifest (because the kermt repo working\ntree was dirty at launch time), the replay may not be bit-exact — pin the\nexact commit via the `repo.commit` field and `git checkout` it\nfirst.\n","# kermt-continue-pretrain\n\nContinue pretraining from a user-supplied KERMT checkpoint (grover_base \u002F\ncmim \u002F hybrid). The skill is the workflow orchestrator: it validates inputs,\nprepares the corpus, launches the runner, and returns a run directory.\n\n## Hardware requirements\n\n- **GPUs**: 1–N CUDA-capable NVIDIA GPUs. The runner auto-detects via\n  `torch.cuda.device_count()`; `--gpus 0,2` overrides. On a single GPU the\n  runner falls back to `--batch_size 32 --save_interval 500`; on multi-GPU\n  it uses the `defaults_pretrain.json` values (currently `batch_size 256`).\n  Note: `--gpus N` uses **torch.cuda** indexing, which can differ from\n  `nvidia-smi`'s display order on multi-GPU hosts (PCI bus vs. CUDA\n  enumeration). To target a specific physical GPU, set `CUDA_VISIBLE_DEVICES`\n  before invoking, or run\n  `python -c \"import torch; print([torch.cuda.get_device_name(i) for i in range(torch.cuda.device_count())])\"`\n  to confirm which device you're picking.\n- **VRAM**: the default `--batch-size 256` is sized for A100-class hardware\n  (80 GB VRAM). On smaller GPUs, downscale to avoid OOM:\n\n  | GPU class                | VRAM       | Suggested `--batch-size` |\n  |--------------------------|------------|--------------------------|\n  | L4, T4, V100 16 GB       | 16–24 GB   | 32–64                    |\n  | A100 40 GB, L40, A40     | 40–48 GB   | 128                      |\n  | A100 80 GB, H100, H200   | 80 GB      | 256 (default)            |\n\n  These are rough starting points — pass `--batch-size N` to override.\n- **Disk**: tens of GB depending on corpus size + epochs (each checkpoint\n  is several hundred MB).\n- **Driver \u002F CUDA**: any host supporting CUDA 12.6 (the kermt image base).\n  `kermt-setup` validates this up-front.\n\n## Inputs\n\nRequired:\n\n- `--csv \u003Cpath>` — the pretrain CSV (single column `smiles`). If you have\n  separate train\u002Fval CSVs, pass `--val-csv \u003Cpath>` too.\n\nCheckpoint (optional — defaults to the released model if omitted):\n\n- `--ckpt \u003Cpath>` — the input pretrain checkpoint to continue from. Must be\n  a grover_base (with vocab heads), cmim, or hybrid ckpt; the validator\n  rejects everything else with a redirect to the correct workflow. **If\n  omitted**, the skill offers to download the released pretrained hybrid model\n  **nvidia\u002FNV-KERMT-70M-v2** and continue-pretrain from it — see \"Resolve &\n  validate the checkpoint\" (workflow step 3). The released bundle ships its\n  three vocab files alongside the ckpt, so the authoritative-vocab pass-through\n  (step 5) works automatically.\n- `--pretrained-release` — explicit opt-in to use the released model without\n  the interactive prompt (for non-interactive \u002F agent runs). Mutually\n  exclusive with `--ckpt`.\n- `--model-dir \u003Cdir>` — where to save the downloaded bundle (default\n  `$KERMT_REPO\u002Fmodels\u002FNV-KERMT-70M-v2\u002F`). An already-complete bundle there is\n  reused, not re-downloaded.\n\nOptional:\n\n- `--val-csv \u003Cpath>` — separate validation CSV. Without it, the prep step\n  auto-splits the input by `--val-frac 0.1` (random shuffle with `--seed`).\n- `--epochs N` \u002F `--batch-size N` \u002F `--init-lr F` \u002F `--max-lr F` \u002F\n  `--final-lr F` \u002F `--warmup-epochs F` \u002F `--weight-decay F` \u002F `--dropout F` \u002F\n  `--save-interval N` \u002F `--seed N` — training-hyperparameter overrides.\n  Anything not given is filled from `agent\u002Fconfig\u002Fdefaults_pretrain.json`.\n- `--vocab-loss-weight F` (hybrid only) \u002F `--latent-dim N` \u002F\n  `--contrastive-temperature F` (cmim and hybrid only) — loss \u002F decoder\n  overrides.\n- `--wandb-project NAME` \u002F `--wandb-run-name NAME` — optional Weights & Biases\n  logging. When `--wandb-project` is set, rank 0 logs train\u002Fval losses; the run\n  name is honored only alongside a project. Off by default. (Independent of the\n  ckpt's `wandb_run_id` continuity handling under `--resume`.)\n- `--resume` — see \"Modes\" section below.\n- `--gpus 0,2` — restrict to a GPU subset. Default uses all visible GPUs.\n- `--from-prepare \u003Cdir>` — skip the prepare step and reuse an existing\n  `prepare_data.json` in `\u003Cdir>`. Useful when iterating on hyperparameters.\n\n## Modes\n\nThe runner has two modes for ingesting the input ckpt, dispatched on whether\n`--resume` is set. Pick based on intent:\n\n### Default (fresh-schedule continue-pretrain)\n\n**Use when**: you have a finished pretrain ckpt and want to continue training\nit — on a new corpus, with a different objective, or just for more epochs\nthan its original plan. The previous training's step counter and schedule\nshape are no longer relevant; you want a new learning-rate schedule for the\nnew run.\n\n**What gets loaded from the ckpt**:\n- ✓ Model weights (encoder + vocab heads + contrast head + decoder, whatever\n  is there)\n- ✓ Optimizer state (Adam's running m1\u002Fm2 moments — warm-starts the new\n  schedule so the first few hundred steps aren't dominated by noisy\n  gradient-estimate startup)\n- ✗ Scheduler step counter (reset to 0)\n- ✗ Epoch counter (reset to 0)\n- ✗ Batch counter (reset to 0)\n- ✗ wandb run id (new wandb run, not a continuation)\n\n**Schedule shape** (init\u002Fmax\u002Ffinal LR, warmup epochs, total epochs): from\nyour CLI args or `defaults_pretrain.json`. A fresh NoamLR is constructed\nfrom these values and starts at step 0.\n\n### `--resume` (true resume)\n\n**Use when**: a previous run was interrupted (crash, OOM, Ctrl-C) and you\nwant to pick up exactly where it left off — same dataset, same schedule,\nsame training trajectory.\n\n**What gets loaded from the ckpt**: **everything** in the\n`save_model_for_restart` format. Model weights + optimizer state +\nscheduler_step + epoch + batch_idx + wandb_run_id are all restored. The\nnew run continues from the saved step in the saved schedule (which is\nrecovered from the ckpt's `saved_args`). Mid-epoch resume works too —\n`pretrain_ddp.py`'s sampler skip-count picks up at the saved batch index\nwithin the saved epoch.\n\n**Schedule shape**: inherited from the ckpt's `saved_args`. CLI overrides\nof any schedule flag (`--epochs \u002F --warmup-epochs \u002F --init-lr \u002F --max-lr \u002F\n--final-lr`) are **rejected with a hard error** — pure resume means pure\nresume; if you want to change the schedule, drop `--resume` and start a\nfresh-schedule run.\n\n**Requirements**: the ckpt must have been saved via `save_model_for_restart`\n(i.e., carry `optimizer \u002F scheduler_step \u002F epoch \u002F batch_idx` keys). If\nany of these is missing, the runner errors with a clear message and\nsuggests dropping `--resume`.\n\nThe default mode is the right choice ~90% of the time. Reach for `--resume`\nonly when you genuinely need to continue a single interrupted training\nrun.\n\n## Workflow\n\nLet `$KERMT_REPO` be the path to your kermt repo checkout, and assume\n`kermt-setup` has already built `kermt:latest`. All paths below are on the\nhost; the helper bind-mounts them at known container paths.\n\n1. **Pre-flight: ensure container + system probe.**\n   ```\n   $KERMT_REPO\u002Fagent\u002Fscripts\u002Fkermt_container.sh check_system | python -c \"\n   import json, sys; d = json.load(sys.stdin)\n   if not d['ok']:\n       print('System check failed:', d['gaps']); sys.exit(1)\n   print(f'OK: {len(d[\\\"gpus\\\"])} GPU(s); {d[\\\"disk\\\"][\\\"free_gb\\\"]} GB free; CUDA via container toolkit')\n   \"\n   ```\n   Surface any `gaps` to the user. Refuse to proceed if `ok: false`.\n\n2. **Compute run directory.**\n   ```\n   RUN_DIR=$KERMT_REPO\u002Fruns\u002Fcontinue-pretrain_$(date -u +%Y-%m-%dT%H-%M-%SZ)\n   ```\n\n3. **Resolve & validate the checkpoint.**\n\n   **Resolve — only if `--ckpt` was omitted.** Default to the released\n   pretrained hybrid model **nvidia\u002FNV-KERMT-70M-v2**:\n   - **Consent gate.** Unless `--pretrained-release` was passed, ask the user:\n     \"No checkpoint given — download the released model nvidia\u002FNV-KERMT-70M-v2\n     (NVIDIA Open Model License, https:\u002F\u002Fhuggingface.co\u002Fnvidia\u002FNV-KERMT-70M-v2)\n     and continue-pretrain from it? [y\u002FN]\". **Never download without an\n     explicit yes** (or `--pretrained-release`). If both `--ckpt` and\n     `--pretrained-release` are given, abort — they conflict.\n   - **Save location.** Default `$KERMT_REPO\u002Fmodels\u002FNV-KERMT-70M-v2\u002F`; honor\n     `--model-dir \u003Cdir>` if given. An already-complete bundle is reused.\n   - **Download** (foreground; ~282 MB on first fetch):\n     ```\n     $KERMT_REPO\u002Fagent\u002Fscripts\u002Fkermt_container.sh run --model-dir \u003Csave-dir> -- \\\n         \"python agent\u002Fscripts\u002Ffetch_released_model.py --out \u002Fmodel\"\n     ```\n     Parse the JSON; abort on `ok: false` (surface `errors`). On success set\n     `\u003Cuser-ckpt> = \u003Csave-dir>\u002Fkermt_contrastive_v2.0.pt`. The bundle's three\n     vocab files land in `\u003Csave-dir>` too, so step 5's `--vocab-dir`\n     auto-detection (which looks in the ckpt's parent directory) finds them\n     with no extra work.\n\n   **Validate** the resolved (or user-provided) ckpt:\n   ```\n   $KERMT_REPO\u002Fagent\u002Fscripts\u002Fkermt_container.sh run --ckpt \u003Cuser-ckpt> -- \\\n       \"python agent\u002Fscripts\u002Fcheck_checkpoint.py --mode continue_pretrain --ckpt \u002Fckpt\"\n   ```\n   Parse the JSON. Abort on `ok: false`, showing the error verbatim. The error\n   message redirects the user to `kermt-add-cmim-pretrain` for encoder-only\n   ckpts, or to `kermt-finetune` for finetuned ckpts.\n\n4. **Validate the data.**\n   ```\n   $KERMT_REPO\u002Fagent\u002Fscripts\u002Fkermt_container.sh run --data \u003Cuser-csv> -- \\\n       \"python agent\u002Fscripts\u002Fcheck_data.py --mode pretrain --csv \u002Fdata\u002F\u003Cbasename>\"\n   ```\n   Abort on `ok: false`.\n\n5. **Prepare the data** (skip if `--from-prepare` given).\n   **Pass the ckpt's vocab through.** Look in the ckpt's parent directory for\n   the conventional `pretrain_atom_vocab.{json,pkl}`, `pretrain_bond_vocab.{json,pkl}`,\n   and `pretrain_smiles_vocab.pkl` files (the bundling convention for released\n   models; see `agent\u002FREADME.md` \"Released models\" section). If all three are\n   present, auto-pass via `--vocab-dir \u003Cckpt_parent_dir>`. If only some are\n   present, pass them via explicit flags (`--atom-vocab`, `--bond-vocab`,\n   `--smiles-vocab`). If none are present, ask the user for `--vocab-dir` — or\n   refuse to proceed, because rebuilding a fresh vocab from the new corpus\n   would silently mismatch the ckpt's vocab heads (the ckpt's vocab is\n   authoritative for continue-pretrain).\n\n   Note the **two-layer mount pattern**: pass the host directory to\n   `kermt_container.sh --vocab-dir` (which mounts it at `\u002Fvocab` inside the\n   container), and reference `\u002Fvocab` from the inner `prepare_data.py`\n   command. The same pattern applies to every host path the inner command\n   needs to read (`--data \u003Chost-csv>` → `\u002Fdata\u002F\u003Cbasename>`,\n   `--ckpt \u003Chost-ckpt>` → `\u002Fckpt`).\n\n   ```\n   VOCAB_DIR=$(dirname \u003Cuser-ckpt>)\n   $KERMT_REPO\u002Fagent\u002Fscripts\u002Fkermt_container.sh run \\\n       --data \u003Cuser-csv> --vocab-dir $VOCAB_DIR --run-dir $RUN_DIR -- \\\n       \"python agent\u002Fscripts\u002Fprepare_data.py --mode pretrain \\\\\n            --csv \u002Fdata\u002F\u003Cbasename> --out \u002Fruns\u002Fdata \\\\\n            --vocab-dir \u002Fvocab \\\\\n            [--val-csv \u002Fdata\u002F\u003Cval-basename>] [--val-frac 0.1] [--seed 0]\"\n   ```\n   Outputs land at `$RUN_DIR\u002Fdata\u002Fprepare_data.json` with\n   `vocab_source: \"user_provided\"`. The runner step 7 will verify the vocab\n   files' entry counts match the ckpt's vocab-head sizes and refuse to launch\n   on mismatch.\n\n6. **Estimate runtime + confirm with user.**\n   - Pretrain wall time depends on corpus size × epochs × GPU count.\n   - Tell the user the estimate; ask \"proceed?\" unless `--yes` flag was given\n     (agent-non-interactive case).\n   - Example estimate template:\n     `~N hours on K GPUs for E epochs over M molecules (~steps\u002Fepoch × seconds\u002Fstep)`.\n\n7. **Launch the runner detached.**\n   ```\n   $KERMT_REPO\u002Fagent\u002Fscripts\u002Fkermt_container.sh run_detached \\\\\n       --name kermt-continue-pretrain-\u003Cts> \\\\\n       --ckpt \u003Cuser-ckpt> --run-dir $RUN_DIR -- \\\\\n       \"python agent\u002Fscripts\u002Frun_pretrain_local.py \\\\\n            --ckpt \u002Fckpt \\\\\n            --prepare-manifest \u002Fruns\u002Fdata\u002Fprepare_data.json \\\\\n            --out \u002Fruns \\\\\n            [--epochs N --batch-size N --init-lr F ...]\"\n   ```\n   Returns the container name + id + log file path.\n\n8. **Report to the user.** Output a short summary:\n   - Container name + id\n   - `$RUN_DIR\u002Frun.json` (the manifest with cmd_replay + image digest)\n   - Log file: `$RUN_DIR\u002Flogs\u002Fpretrain_ddp.log`\n   - TensorBoard: `$RUN_DIR\u002Flogs\u002Ftb` (open with `tensorboard --logdir\n     $RUN_DIR\u002Flogs\u002Ftb`)\n   - Suggest invoking `kermt-monitor \u003CRUN_DIR>` to check progress.\n\n## Hard rules\n\n- **Never download the released model without consent.** When `--ckpt` is\n  omitted, download `nvidia\u002FNV-KERMT-70M-v2` only after an explicit user \"yes\"\n  or an explicit `--pretrained-release` flag. `--ckpt` and\n  `--pretrained-release` are mutually exclusive.\n- **Never modify the user's input ckpt.** The runner symlinks it into the\n  save_dir; the symlink is what pretrain_ddp.py auto-resumes from. The\n  source file stays untouched.\n- **Never silently override arch.** If the user passes a `--hidden-size`\n  etc. that doesn't match the ckpt-derived value, the runner aborts loudly.\n  Arch params come from the ckpt, period.\n- **Never block on the long-running pretrain itself.** The runner is invoked\n  via `run_detached`; the skill returns immediately after step 8. Use\n  `kermt-monitor` for progress.\n- **Echo applied defaults back to the user.** The `args_applied` field of\n  `run.json` records every flag's value + source (user \u002F default-config \u002F\n  auto-1gpu \u002F auto-multi-gpu). Skill should surface a summary of any flag\n  not user-specified so the user knows what was assumed.\n\n## Common errors\n\n- `model_type='finetuned'` rejected → the ckpt is a downstream finetune,\n  not a pretrain. The error redirects to the relevant workflow.\n- `grover_base ckpt has no vocab head` → encoder-only ckpt (e.g. the\n  original-grover `grover_base.pt`). The error redirects to\n  `kermt-add-cmim-pretrain`.\n- `prepare_data manifest is missing required outputs` → user passed\n  `--from-prepare` to a directory where prepare was run with `--skip-vocab`\n  or `--skip-split`. Re-run prepare without those flags.\n- `--gpus all` not available → install `nvidia-container-toolkit`; check\n  `kermt_container.sh check_system`.\n\n## Replayability\n\nThe `run.json` `cmd_replay` field is a single-line command that re-runs the\npretrain with the same inputs, hyperparameters, and arch. To replay:\n\n```bash\n# Inside the kermt container:\n$(jq -r .cmd_replay $RUN_DIR\u002Frun.json)\n```\n\nIf `ok_to_replay: false` in the manifest (because the kermt repo working\ntree was dirty at launch time), the replay may not be bit-exact — pin the\nexact commit via the `repo.commit` field and `git checkout` it\nfirst.\n",{"name":5,"license":15,"metadata":24,"description":7,"compatibility":28},{"owner":25,"risk_tier":26,"classification":27},"evax@nvidia.com","skill","workflow-skill","Requires docker, nvidia-container-toolkit, and a CUDA-capable NVIDIA GPU. Designed for Claude Code, Codex, and Nemotron.","master","49e49abf2f670d429e0b7e2c0349739c3fdd5cbe","skills\u002Fkermt-continue-pretrain",5,"https:\u002F\u002Fgithub.com\u002Fstanfish06\u002Fskillquarium","https:\u002F\u002Fgithub.com\u002Fstanfish06","https:\u002F\u002Fwww.stanfish.dev\u002F","2026-11-14 21:31:37",{"source":8,"skillName":5,"commands":38,"prompt":48,"zipAvailable":49},[39,42,45],{"tool":40,"command":41},"npx","npx skills add stanfish06\u002Fskillquarium --skill kermt-continue-pretrain",{"tool":43,"command":44},"bunx","bunx skills add stanfish06\u002Fskillquarium --skill kermt-continue-pretrain",{"tool":46,"command":47},"pnpm","pnpm dlx skills add stanfish06\u002Fskillquarium --skill kermt-continue-pretrain","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fstanfish06\u002Fskillquarium\u002Fblob\u002Fmaster\u002Fskills\u002Fkermt-continue-pretrain\u002FSKILL.md",true,[51],{"path":52,"resourceType":53,"resourceTypeDesc":54,"sizeBytes":55,"sourceHtmlUrl":56},"evals\u002Fevals.json","REFERENCE","参考文档",7722,"https:\u002F\u002Fgithub.com\u002Fstanfish06\u002Fskillquarium\u002Fblob\u002Fmaster\u002Fskills\u002Fkermt-continue-pretrain\u002Fevals\u002Fevals.json",[58,66,74,81,88,95],{"skillNo":59,"name":60,"nameZh":6,"description":61,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":62,"licenseFileUrl":6,"sourceHtmlUrl":63,"categories":64,"keywords":65,"publishedAt":19},"01M76ZJTRVA456GD7K8Y3EP82H","zarr-python","Stores and queries chunked N-D scientific arrays with Zarr-Python 3, including codecs, sharding, S3\u002FGCS storage, and NumPy\u002FDask\u002FXarray integration. Use for array layout, bounded I\u002FO, format migration, or scientific metadata preservation.","MIT","https:\u002F\u002Fgithub.com\u002Fstanfish06\u002Fskillquarium\u002Fblob\u002Fmaster\u002Fskills\u002Fzarr-python\u002FSKILL.md",[],[],{"skillNo":67,"name":68,"nameZh":6,"description":69,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":70,"sourceHtmlUrl":71,"categories":72,"keywords":73,"publishedAt":19},"01M76ZJQZ15Q1HKC5V85NHE325","yeet","Publish local changes to GitHub by confirming scope, committing intentionally, pushing the branch, and opening a draft PR through the GitHub app from this plugin, with `gh` used only as a fallback where connector coverage is insufficient.","https:\u002F\u002Fgithub.com\u002Fstanfish06\u002Fskillquarium\u002Fblob\u002Fmaster\u002Fskills\u002Fyeet\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Fstanfish06\u002Fskillquarium\u002Fblob\u002Fmaster\u002Fskills\u002Fyeet\u002FSKILL.md",[],[],{"skillNo":75,"name":76,"nameZh":6,"description":77,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":78,"categories":79,"keywords":80,"publishedAt":19},"01M76ZJ8ASZ4QYQ70Y6SBA4Z4S","xarray","N-dimensional labeled arrays and datasets in Python. Built on top of NumPy and Dask. It introduces labels in the form of dimensions, coordinates, and attributes on top of raw NumPy-like arrays, making data analysis in physical sciences more intuitive and less error-prone. Use for working with multi-dimensional scientific data, NetCDF\u002FGRIB\u002FZarr files, climate\u002Fweather\u002Foceanographic datasets, remote sensing, geospatial imaging, large out-of-memory datasets with Dask, and labeled array operations.","https:\u002F\u002Fgithub.com\u002Fstanfish06\u002Fskillquarium\u002Fblob\u002Fmaster\u002Fskills\u002Fxarray\u002FSKILL.md",[],[],{"skillNo":82,"name":83,"nameZh":6,"description":84,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":62,"licenseFileUrl":6,"sourceHtmlUrl":85,"categories":86,"keywords":87,"publishedAt":19},"01M76ZHFBXD6VXPQ02VCEG202B","writing-mstest-tests","Use when asked to fix, rewrite, update, improve, modernize, show corrected code for, or explain existing MSTest tests or MSTest-specific configuration. Use for \"review\" when corrected code or edits are wanted, even for one pasted assertion or passing tests with bad failure output. Covers expected\u002Factual labels; generic Boolean, collection, string, numeric, null, identity, exception, hard-cast, and object[] checks; TestContext\u002Flifecycle; timeout\u002Fcancellation; OS\u002FCI conditions, retry, cleanup, parallelization, MSTest.Sdk project setup, and MSTESTxxxx. Honor the installed MSTest version. DO NOT USE to design new test cases (code-testing), perform report-only audits, create project files rather than explain MSTest setup, run tests, migrate frameworks, or handle non-MSTest\u002Fnon-.NET code.","https:\u002F\u002Fgithub.com\u002Fstanfish06\u002Fskillquarium\u002Fblob\u002Fmaster\u002Fskills\u002Fwriting-mstest-tests\u002FSKILL.md",[],[],{"skillNo":89,"name":90,"nameZh":6,"description":91,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":62,"licenseFileUrl":6,"sourceHtmlUrl":92,"categories":93,"keywords":94,"publishedAt":19},"01M76ZDFDF4N88RH6HSYGS7GP2","wgs-prs","End-to-end WGS to polygenic risk score pipeline. Takes paired-end FASTQ files (or a pre-existing VCF) through nf-core\u002Fsarek for variant calling, applies VCF QC (normalisation, hard filtering, Ti\u002FTv and Het\u002FHom checks), then computes polygenic risk scores via the PGS Catalog. Fills the FASTQ to VCF gap upstream of the gwas-prs skill.","https:\u002F\u002Fgithub.com\u002Fstanfish06\u002Fskillquarium\u002Fblob\u002Fmaster\u002Fskills\u002Fwgs-prs\u002FSKILL.md",[],[],{"skillNo":96,"name":97,"nameZh":6,"description":98,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":99,"sourceHtmlUrl":100,"categories":101,"keywords":102,"publishedAt":19},"01M76ZCC1SC68J42QK4MDC8RCG","webapp-testing","Toolkit for interacting with and testing local web applications using Playwright. Supports verifying frontend functionality, debugging UI behavior, capturing browser screenshots, and viewing browser logs.","https:\u002F\u002Fgithub.com\u002Fstanfish06\u002Fskillquarium\u002Fblob\u002Fmaster\u002Fskills\u002Fwebapp-testing\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002Fstanfish06\u002Fskillquarium\u002Fblob\u002Fmaster\u002Fskills\u002Fwebapp-testing\u002FSKILL.md",[],[]]