[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M79FT3F81ECQN15EE1DAWX2F":3,"skill-same:01M79FT3F81ECQN15EE1DAWX2F":92},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":32,"commitSha":33,"skillDirectory":34,"forksCount":14,"repositoryHtmlUrl":35,"repositoryHomepageUrl":6,"ownerHtmlUrl":36,"ownerBlogUrl":6,"ownerDescription":6,"updatedAt":37,"install":38,"resources":51},"01M79FT3F81ECQN15EE1DAWX2F","tao-run-on-slurm",null,"Remote SLURM GPU cluster execution over SSH with sbatch\u002Fsrun, Pyxis\u002FEnroot containers, and Lustre-backed results. Use when running TAO training\u002Feval\u002Finference jobs on an on-prem or DGX SLURM cluster. Trigger phrases include \"run on SLURM\", \"submit sbatch\", \"DGX SLURM cluster\", \"Pyxis\u002FEnroot container\", \"Lustre dataset\".","Elogic360\u002Fsoftware-factory","Elogic360","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F156535519?v=4","01M74NH3F0RB13PWB7JZDE2V80","elogic360","01M74NH3FPC65082MDZVMFHDM3",0,"Apache-2.0","https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Ftao-run-on-slurm\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: tao-run-on-slurm\ndescription: Remote SLURM GPU cluster execution over SSH with sbatch\u002Fsrun, Pyxis\u002FEnroot containers, and Lustre-backed\n  results. Use when running TAO training\u002Feval\u002Finference jobs on an on-prem or DGX SLURM cluster. Trigger phrases include\n  \"run on SLURM\", \"submit sbatch\", \"DGX SLURM cluster\", \"Pyxis\u002FEnroot container\", \"Lustre dataset\".\nlicense: Apache-2.0\ncompatibility: Requires SSH access to a SLURM login node (passwordless via key auth) and SLURM_USER + SLURM_HOSTNAME env vars.\n  The TAO SDK with the slurm extra (pip install 'nvidia-tao-sdk[slurm]') is needed only if you want Job handles, S3 I\u002FO wrapping,\n  or run-folder durability via ActionWorkflow.\nmetadata:\n  author: NVIDIA Corporation\n  version: \"0.2.0\"\nallowed-tools: Read Bash\ntags:\n- platform\n- slurm\n---\n\n# SLURM\n\nRemote GPU compute platform for clusters managed by SLURM. Jobs are submitted\nfrom the TAO service or SDK host to a login node over SSH, staged on a shared\nfilesystem, submitted with `sbatch`, and executed with `srun` container support.\n\nUse SLURM when the user has access to a managed GPU cluster, shared Lustre\nstorage, and scheduler-owned GPU allocation. Do not use SLURM for local files\nthat exist only on the agent machine; data and outputs must be reachable from\nthe cluster.\n\n## Preflight\n\n```bash\n# 1. SSH to the login node works without a password prompt\nSLURM_HOST=\"${SLURM_HOSTNAME%%,*}\"\n[ -n \"$SLURM_USER\" ] && [ -n \"$SLURM_HOST\" ] || {\n  echo \"MISSING: set SLURM_USER and SLURM_HOSTNAME (comma-separated for failover) in your env (~\u002F.config\u002Ftao\u002F.env).\"\n  exit 1\n}\nssh -o BatchMode=yes -o ConnectTimeout=10 \"${SLURM_USER}@${SLURM_HOST}\" \"true\" 2>\u002Fdev\u002Fnull || {\n  echo \"MISSING: passwordless SSH to ${SLURM_USER}@${SLURM_HOST} not working. See references\u002Fssh-setup.md.\"\n  exit 1\n}\n\n# 2. Optional: TAO SDK wrapper for Job handles + S3 wrapping.\n# nvidia-tao-sdk is on public PyPI; pin lives in versions.yaml (wheels.tao_sdk_slurm).\nPIN=$(\"${TAO_SKILL_BANK_PATH:?}\u002Fscripts\u002Fresolve_versions_key.py\" wheels.tao_sdk_slurm)\npython -c \"import tao_sdk\" 2>\u002Fdev\u002Fnull || {\n  echo \"MISSING: nvidia-tao-sdk not installed. Run:\"\n  echo \"  pip install \\\"$PIN\\\"\"\n  exit 1\n}\n```\n\nIf a check fails, the agent prompts the user to authorize the install\u002Ffix via Bash.\n\nA third preflight step applies only for **private `nvcr.io` images**: Pyxis on\nthe compute nodes needs persistent enroot credentials in\n`~\u002F.config\u002Fenroot\u002F.credentials` on the cluster (it does NOT read `NGC_KEY` from\nthe job env). Without them, auth-gated pulls fail with \"Could not process JSON\ninput\" at job startup. This runs once per (cluster, user). See\n`references\u002Fssh-setup.md` for the full check and the `printf | ssh` install\npattern that keeps `NGC_KEY` out of history, files, and chat output. Skip it for\npublic images.\n\n## Prerequisites\n\nBefore any job is submitted, the host running the TAO service or SDK must log in\nto at least one host from `SLURM_HOSTNAME` over SSH **without an interactive\npassword prompt**. The handler runs `sbatch`, `squeue`, `sacct`, `scancel`, and\nlog tails non-interactively, so password or 2FA prompts will fail the job at\nsubmit or status time.\n\nSet this up once per (host, login node, user) tuple: create an SSH keypair,\ninstall the public key on each login host, trust the host key, lock private-key\npermissions to `chmod 600`, and verify with `ssh -o BatchMode=yes ...`. See\n`references\u002Fssh-setup.md` for the full step-by-step (including the `~\u002F.ssh\u002Fconfig`\nalias, the container key-mount note, and the 2FA \u002F `SSH_AUTH_SOCK` fallback). The\nsame file holds the **SSH failure remediation prompt** to show the user when\npasswordless SSH fails.\n\n## Credentials\n\n- **SLURM_USER** (required): SSH username for the login node. In microservices\n  workspace metadata this is `cloud_specific_details.slurm_user`.\n- **SLURM_HOSTNAME** (required): Comma-separated login hostnames for failover.\n  Microservices schema stores this as the list field\n  `cloud_specific_details.slurm_hostname`.\n- **SLURM_PARTITION** (required): Partition list for GPU job submission. Ask\n  for this in the mandatory SLURM intake list. The packaged default is\n  `polar,polar3,polar4,grizzly`, which are treated as 4-hour queues.\n- **SSH_KEY_PATH** (preferred and expected before launch): private key path for\n  non-interactive public-key auth to the login node. If passwordless SSH fails,\n  ask the user for `SSH_KEY_PATH=\u002Fpath\u002Fto\u002Fprivate_key` and show the setup steps\n  in `references\u002Fssh-setup.md`; do not bury this behind several alternate choices.\n- **SSH_AUTH_SOCK** (advanced fallback): SSH agent socket with an accepted key\n  already loaded. Prefer `SSH_KEY_PATH` in user-facing remediation prompts.\n- **SLURM_BASE_RESULTS_DIR** (optional): Base shared filesystem path. Default\n  convention from `tao-core` is `\u002Flustre\u002Ffsw\u002Fportfolios\u002Fedgeai\u002F\u003Cyour-dir>`,\n  where `\u003Cyour-dir>` is your per-user directory on the cluster.\n- **SLURM_ACCOUNT** (usually required by site policy): Account charged by\n  `#SBATCH --account`.\n\nDo not ask for `SLURM_ACCOUNT` or `SLURM_BASE_RESULTS_DIR` in the initial\nintake unless the user says their site requires an account, wants a custom\nresults root, or the workflow cannot proceed without overriding defaults.\n\n## Backend Details\n\nUse `backend_details.backend_type = \"slurm\"` when routing a job to this\nplatform. Supported backend details from the microservices schema:\n\n```json\n{\n  \"backend_type\": \"slurm\",\n  \"partition\": \"polar,polar3,polar4,grizzly\",\n  \"cluster_name\": \"optional-name\"\n}\n```\n\nRuntime metadata is stored under `backend_details.slurm_metadata`, especially\n`slurm_job_id` and `job_dir`. Do not invent these values. They are written\nafter `sbatch` returns a scheduler job id.\n\n## Storage\n\nSLURM jobs run on the cluster, so local paths from the API host are not valid\ndataset paths. Prefer shared filesystem URIs:\n\n- Use `lustre:\u002F\u002F\u002Fabsolute\u002Fpath` for user-provided datasets on Lustre.\n- `slurm:\u002F\u002F` paths may appear in microservices metadata and are converted to\n  actual Lustre paths before the container starts.\n- Avoid bare `\u002Flocal\u002Fpath` and `file:\u002F\u002F` dataset URIs for SLURM. Validation in\n  `tao-core` rejects local and file paths for remote backends.\n\nAccept either dataset roots or direct spec-key paths:\n\n- Root mode: `\u002Flustre\u002F...\u002F\u003Cmodel>\u002Ftrain`, which model skills map to required\n  files such as `\u003Croot>\u002Fannotations.json` and `\u003Croot>` as media path.\n- Direct spec mode: exact fields such as\n  `custom.train_dataset.annotation_path=\u002Flustre\u002F...\u002Ftrain.json` and\n  `custom.train_dataset.media_path=\u002Flustre\u002F...\u002Fvideos.tar.gz`.\n\nAfter passwordless SSH succeeds and before generating scripts, validate each\nrequired dataset file\u002Fpath from the login host:\n\n```bash\nssh -o BatchMode=yes \u003CSLURM_USER>@\u003Cworking-login-host> \\\n  'test -e \u002Flustre\u002F...\u002Fannotations.json && test -e \u002Flustre\u002F...\u002Fmedia_or_archive'\n```\n\nIf the remote `test -e` fails, stop and ask for corrected paths or for the data\nto be staged onto shared cluster storage. Do not create runner scripts that will\nfail inside the first training job.\n\nResults default to:\n\n```text\n\u002Flustre\u002Ffsw\u002Fportfolios\u002Fedgeai\u002F\u003Cyour-dir>\u002Fresults\u002F\u003Cjob_id>\n```\n\n`\u003Cyour-dir>` is your per-user directory on the cluster.\n\nThe runner sets `TAO_API_RESULTS_DIR` to the parent results directory because\ncontainer code appends the job id when writing status and artifacts.\n\n> **Use Lustre, not S3, for SLURM job inputs.** SLURM's scheduler enforces a\n> GPU-idle timeout — a long `s3:\u002F\u002F` download at the top of the script can burn\n> the allocation before training begins, and the scheduler may kill the job.\n> Stage training data onto Lustre first; S3 \u002F HF \u002F NGC pre-fetch is fine only\n> for small auxiliary inputs (checkpoints, configs). See `references\u002Fsdk-usage.md`\n> for the full rationale.\n\n## Container Execution\n\n`tao-core` uses the SLURM handler to run TAO containers through Pyxis\u002FEnroot:\n\n1. Stage compact JSON files for specs, environment, and cloud metadata under\n   `\u003Cjob_dir>\u002Fspecs`, `\u003Cjob_dir>\u002Fenv`, and `\u003Cjob_dir>\u002Fmeta`.\n2. Optionally convert the Docker image to a cached SQSH image with\n   `srun -n1 -p \u003Cconversion_partition> enroot import`.\n3. Write an sbatch script under `\u003Cjob_dir>\u002Fsbatch\u002Fjob_\u003Cjob_id>.sbatch`.\n4. Submit `sbatch --export=ALL \u003Cscript>`.\n5. Run the container with `srun --container-image=\u003Cimage> --container-mounts=\u002Flustre`.\n\nImage formats accepted by the handler:\n\n- `\u002Fpath\u002Fto\u002Fimage.sqsh`\n- `registry#image:tag`\n- `docker:\u002F\u002Fregistry#image:tag`\n- ordinary `registry\u002Fimage:tag`, which is converted to Pyxis form when needed\n\nSQSH conversion is cached by image name. For `:latest` images, cached SQSH is\nused unless `force_reconvert_latest` is enabled.\n\n## Resource Mapping\n\nDefaults from `tao-core`:\n\n- `num_nodes`: 1\n- `num_gpus`: 4\n- `max_num_gpus_per_node`: 8\n- `cpus_per_task`: 16\n- `time_hours`: 4\n- `timeout_hours`: 3.8\n- `max_time_hours`: 4\n- `container_mounts`: `\u002Flustre`\n- `use_requeue`: true\n- `use_sqsh`: true\n\nWhen generating launchers or wrapper scripts for SLURM, set the wall-time\ndefaults explicitly from the packaged platform resource defaults:\n\n```bash\nexport SLURM_TIME_HOURS=\"${SLURM_TIME_HOURS:-4}\"\nexport SLURM_TIMEOUT_HOURS=\"${SLURM_TIMEOUT_HOURS:-3.8}\"\n```\n\nDo not default to 12 hours on SLURM. If the user supplies a longer\n`SLURM_TIME_HOURS`, verify that the selected partition supports it before\nsubmitting. For the packaged default partition list\n`polar,polar3,polar4,grizzly`, reject requests above 4 hours and ask for a\ndifferent partition only if the user actually wants a longer wall time.\n\nWhen `num_gpus` is greater than or equal to `max_num_gpus_per_node`, the\nhandler treats the request as exclusive per node and computes additional nodes\nfrom total GPU count when necessary.\n\nFor multi-node jobs (`num_nodes > 1`), the sbatch script exports `WORLD_SIZE`,\n`MASTER_ADDR`, `MASTER_PORT`, `NODE_RANK`, and `NUM_GPU_PER_NODE`, and Cosmos-RL\nhas special multi-node role handling for controller, policy, and rollout\nworkers. See `references\u002Fmulti-node.md` for the full sbatch directives, the\nrendezvous env-var table and contract, and cluster requirements.\n\n## Monitoring\n\n- Scheduler status comes from the stored SLURM job id via `squeue` or `sacct`.\n- TAO terminal status comes from `status.json` in the shared results folder.\n- If the user enabled chat monitoring, continue polling at the requested\n  interval while the job is `PENDING`, `RUNNING`, or otherwise non-terminal.\n  Do not stop after a fixed elapsed time such as 30 minutes; long queue waits\n  are normal on shared GPU partitions.\n- Do not send a final response for a non-terminal SLURM job when chat\n  monitoring is enabled. A final response is a detach action; use it only if\n  the user asked to detach\u002Fstop or the job reached terminal state.\n- Logs are read over SSH from:\n\n```text\n\u003Cjob_dir>\u002Fslurm-logs\u002F\u003Cslurm_job_name>-\u003Cslurm_job_id>\u002Fmain.out\n\u003Cjob_dir>\u002Fslurm-logs\u002F\u003Cslurm_job_name>-\u003Cslurm_job_id>\u002Fmain.err\n```\n\nStatus mapping:\n\n- `PENDING` -> `Pending`\n- `RUNNING` or `COMPLETING` -> `Running`\n- `COMPLETED` -> check `status.json`\n- `FAILED`, `BOOT_FAIL`, `DEADLINE`, `OUT_OF_MEMORY`, `NODE_FAIL` -> retry if\n  logs match retriable infrastructure patterns, otherwise `Error`\n- `CANCELLED`, `PREEMPTED`, `REVOKED` -> `Canceled`\n- `TIMEOUT` -> `Error`\n- `SUSPENDED`, `STOPPED` -> `Paused`\n\n## Cancellation\n\nCancel by looking up `backend_details.slurm_metadata.slurm_job_id` and running\n`scancel \u003Cslurm_job_id>` over SSH. Treat missing or already terminated SLURM\njobs as successful cancellation.\n\n## Multi-node training (distributed)\n\nSLURM is the platform of choice for large multi-node runs — pass `num_nodes > 1`\nand the SDK handles the sbatch directives and PyTorch-distributed env vars\nautomatically. See `references\u002Fmulti-node.md` for a worked `create_job` example,\nthe generated sbatch directives, the rendezvous env-var table (`WORLD_SIZE`,\n`NUM_GPU_PER_NODE`, `NODE_RANK`, `MASTER_ADDR`, `MASTER_PORT`), the Cosmos-RL\nrole note, cluster requirements (Pyxis\u002FEnroot, InfiniBand\u002FNVLink, Lustre), and\nupstream reference links.\n\n## Running via the TAO SDK\n\nThe SDK install is covered in Preflight — `pip install 'nvidia-tao-sdk[slurm]'`.\nUse it when you want Job handles, the sbatch\u002F`squeue`\u002F`sacct` plumbing handled\nfor you, run-folder durability via `ActionWorkflow`, or convenient cloud-storage\nI\u002FO (`s3:\u002F\u002F`, `hf_model:\u002F\u002F`, `ngc:\u002F\u002F`). Without the SDK, drive `sbatch` and\n`srun` yourself.\n\nAuto-retry is **fully automatic**: a background monitor polls `squeue`\u002F`sacct`\nand re-`sbatch`'s the staged script on infrastructure-looking failures up to\n`MAX_JOB_RETRIES = 10`, while plain training failures surface immediately. In\naddition, `#SBATCH --requeue` is set by default (`SLURM_USE_REQUEUE`, defaults\nto `true`). See `references\u002Fsdk-usage.md` for the `SlurmSDK` \u002F `build_entrypoint`\ncode example, the Lustre-not-S3 rule, the retriable-failure classification, and\nthe full auto-retry and requeue behavior.\n\n## Failure Modes\n\nCommon failures: SSH auth failure, local dataset path rejected, SQSH conversion\ntimeout, Pyxis\u002FEnroot unavailable, and bad-node \u002F transient GPU failures (which\nthe handler retries up to the configured limit). See\n`references\u002Ftroubleshooting.md` for the diagnosis and remediation of each.\n","# SLURM\n\nRemote GPU compute platform for clusters managed by SLURM. Jobs are submitted\nfrom the TAO service or SDK host to a login node over SSH, staged on a shared\nfilesystem, submitted with `sbatch`, and executed with `srun` container support.\n\nUse SLURM when the user has access to a managed GPU cluster, shared Lustre\nstorage, and scheduler-owned GPU allocation. Do not use SLURM for local files\nthat exist only on the agent machine; data and outputs must be reachable from\nthe cluster.\n\n## Preflight\n\n```bash\n# 1. SSH to the login node works without a password prompt\nSLURM_HOST=\"${SLURM_HOSTNAME%%,*}\"\n[ -n \"$SLURM_USER\" ] && [ -n \"$SLURM_HOST\" ] || {\n  echo \"MISSING: set SLURM_USER and SLURM_HOSTNAME (comma-separated for failover) in your env (~\u002F.config\u002Ftao\u002F.env).\"\n  exit 1\n}\nssh -o BatchMode=yes -o ConnectTimeout=10 \"${SLURM_USER}@${SLURM_HOST}\" \"true\" 2>\u002Fdev\u002Fnull || {\n  echo \"MISSING: passwordless SSH to ${SLURM_USER}@${SLURM_HOST} not working. See references\u002Fssh-setup.md.\"\n  exit 1\n}\n\n# 2. Optional: TAO SDK wrapper for Job handles + S3 wrapping.\n# nvidia-tao-sdk is on public PyPI; pin lives in versions.yaml (wheels.tao_sdk_slurm).\nPIN=$(\"${TAO_SKILL_BANK_PATH:?}\u002Fscripts\u002Fresolve_versions_key.py\" wheels.tao_sdk_slurm)\npython -c \"import tao_sdk\" 2>\u002Fdev\u002Fnull || {\n  echo \"MISSING: nvidia-tao-sdk not installed. Run:\"\n  echo \"  pip install \\\"$PIN\\\"\"\n  exit 1\n}\n```\n\nIf a check fails, the agent prompts the user to authorize the install\u002Ffix via Bash.\n\nA third preflight step applies only for **private `nvcr.io` images**: Pyxis on\nthe compute nodes needs persistent enroot credentials in\n`~\u002F.config\u002Fenroot\u002F.credentials` on the cluster (it does NOT read `NGC_KEY` from\nthe job env). Without them, auth-gated pulls fail with \"Could not process JSON\ninput\" at job startup. This runs once per (cluster, user). See\n`references\u002Fssh-setup.md` for the full check and the `printf | ssh` install\npattern that keeps `NGC_KEY` out of history, files, and chat output. Skip it for\npublic images.\n\n## Prerequisites\n\nBefore any job is submitted, the host running the TAO service or SDK must log in\nto at least one host from `SLURM_HOSTNAME` over SSH **without an interactive\npassword prompt**. The handler runs `sbatch`, `squeue`, `sacct`, `scancel`, and\nlog tails non-interactively, so password or 2FA prompts will fail the job at\nsubmit or status time.\n\nSet this up once per (host, login node, user) tuple: create an SSH keypair,\ninstall the public key on each login host, trust the host key, lock private-key\npermissions to `chmod 600`, and verify with `ssh -o BatchMode=yes ...`. See\n`references\u002Fssh-setup.md` for the full step-by-step (including the `~\u002F.ssh\u002Fconfig`\nalias, the container key-mount note, and the 2FA \u002F `SSH_AUTH_SOCK` fallback). The\nsame file holds the **SSH failure remediation prompt** to show the user when\npasswordless SSH fails.\n\n## Credentials\n\n- **SLURM_USER** (required): SSH username for the login node. In microservices\n  workspace metadata this is `cloud_specific_details.slurm_user`.\n- **SLURM_HOSTNAME** (required): Comma-separated login hostnames for failover.\n  Microservices schema stores this as the list field\n  `cloud_specific_details.slurm_hostname`.\n- **SLURM_PARTITION** (required): Partition list for GPU job submission. Ask\n  for this in the mandatory SLURM intake list. The packaged default is\n  `polar,polar3,polar4,grizzly`, which are treated as 4-hour queues.\n- **SSH_KEY_PATH** (preferred and expected before launch): private key path for\n  non-interactive public-key auth to the login node. If passwordless SSH fails,\n  ask the user for `SSH_KEY_PATH=\u002Fpath\u002Fto\u002Fprivate_key` and show the setup steps\n  in `references\u002Fssh-setup.md`; do not bury this behind several alternate choices.\n- **SSH_AUTH_SOCK** (advanced fallback): SSH agent socket with an accepted key\n  already loaded. Prefer `SSH_KEY_PATH` in user-facing remediation prompts.\n- **SLURM_BASE_RESULTS_DIR** (optional): Base shared filesystem path. Default\n  convention from `tao-core` is `\u002Flustre\u002Ffsw\u002Fportfolios\u002Fedgeai\u002F\u003Cyour-dir>`,\n  where `\u003Cyour-dir>` is your per-user directory on the cluster.\n- **SLURM_ACCOUNT** (usually required by site policy): Account charged by\n  `#SBATCH --account`.\n\nDo not ask for `SLURM_ACCOUNT` or `SLURM_BASE_RESULTS_DIR` in the initial\nintake unless the user says their site requires an account, wants a custom\nresults root, or the workflow cannot proceed without overriding defaults.\n\n## Backend Details\n\nUse `backend_details.backend_type = \"slurm\"` when routing a job to this\nplatform. Supported backend details from the microservices schema:\n\n```json\n{\n  \"backend_type\": \"slurm\",\n  \"partition\": \"polar,polar3,polar4,grizzly\",\n  \"cluster_name\": \"optional-name\"\n}\n```\n\nRuntime metadata is stored under `backend_details.slurm_metadata`, especially\n`slurm_job_id` and `job_dir`. Do not invent these values. They are written\nafter `sbatch` returns a scheduler job id.\n\n## Storage\n\nSLURM jobs run on the cluster, so local paths from the API host are not valid\ndataset paths. Prefer shared filesystem URIs:\n\n- Use `lustre:\u002F\u002F\u002Fabsolute\u002Fpath` for user-provided datasets on Lustre.\n- `slurm:\u002F\u002F` paths may appear in microservices metadata and are converted to\n  actual Lustre paths before the container starts.\n- Avoid bare `\u002Flocal\u002Fpath` and `file:\u002F\u002F` dataset URIs for SLURM. Validation in\n  `tao-core` rejects local and file paths for remote backends.\n\nAccept either dataset roots or direct spec-key paths:\n\n- Root mode: `\u002Flustre\u002F...\u002F\u003Cmodel>\u002Ftrain`, which model skills map to required\n  files such as `\u003Croot>\u002Fannotations.json` and `\u003Croot>` as media path.\n- Direct spec mode: exact fields such as\n  `custom.train_dataset.annotation_path=\u002Flustre\u002F...\u002Ftrain.json` and\n  `custom.train_dataset.media_path=\u002Flustre\u002F...\u002Fvideos.tar.gz`.\n\nAfter passwordless SSH succeeds and before generating scripts, validate each\nrequired dataset file\u002Fpath from the login host:\n\n```bash\nssh -o BatchMode=yes \u003CSLURM_USER>@\u003Cworking-login-host> \\\n  'test -e \u002Flustre\u002F...\u002Fannotations.json && test -e \u002Flustre\u002F...\u002Fmedia_or_archive'\n```\n\nIf the remote `test -e` fails, stop and ask for corrected paths or for the data\nto be staged onto shared cluster storage. Do not create runner scripts that will\nfail inside the first training job.\n\nResults default to:\n\n```text\n\u002Flustre\u002Ffsw\u002Fportfolios\u002Fedgeai\u002F\u003Cyour-dir>\u002Fresults\u002F\u003Cjob_id>\n```\n\n`\u003Cyour-dir>` is your per-user directory on the cluster.\n\nThe runner sets `TAO_API_RESULTS_DIR` to the parent results directory because\ncontainer code appends the job id when writing status and artifacts.\n\n> **Use Lustre, not S3, for SLURM job inputs.** SLURM's scheduler enforces a\n> GPU-idle timeout — a long `s3:\u002F\u002F` download at the top of the script can burn\n> the allocation before training begins, and the scheduler may kill the job.\n> Stage training data onto Lustre first; S3 \u002F HF \u002F NGC pre-fetch is fine only\n> for small auxiliary inputs (checkpoints, configs). See `references\u002Fsdk-usage.md`\n> for the full rationale.\n\n## Container Execution\n\n`tao-core` uses the SLURM handler to run TAO containers through Pyxis\u002FEnroot:\n\n1. Stage compact JSON files for specs, environment, and cloud metadata under\n   `\u003Cjob_dir>\u002Fspecs`, `\u003Cjob_dir>\u002Fenv`, and `\u003Cjob_dir>\u002Fmeta`.\n2. Optionally convert the Docker image to a cached SQSH image with\n   `srun -n1 -p \u003Cconversion_partition> enroot import`.\n3. Write an sbatch script under `\u003Cjob_dir>\u002Fsbatch\u002Fjob_\u003Cjob_id>.sbatch`.\n4. Submit `sbatch --export=ALL \u003Cscript>`.\n5. Run the container with `srun --container-image=\u003Cimage> --container-mounts=\u002Flustre`.\n\nImage formats accepted by the handler:\n\n- `\u002Fpath\u002Fto\u002Fimage.sqsh`\n- `registry#image:tag`\n- `docker:\u002F\u002Fregistry#image:tag`\n- ordinary `registry\u002Fimage:tag`, which is converted to Pyxis form when needed\n\nSQSH conversion is cached by image name. For `:latest` images, cached SQSH is\nused unless `force_reconvert_latest` is enabled.\n\n## Resource Mapping\n\nDefaults from `tao-core`:\n\n- `num_nodes`: 1\n- `num_gpus`: 4\n- `max_num_gpus_per_node`: 8\n- `cpus_per_task`: 16\n- `time_hours`: 4\n- `timeout_hours`: 3.8\n- `max_time_hours`: 4\n- `container_mounts`: `\u002Flustre`\n- `use_requeue`: true\n- `use_sqsh`: true\n\nWhen generating launchers or wrapper scripts for SLURM, set the wall-time\ndefaults explicitly from the packaged platform resource defaults:\n\n```bash\nexport SLURM_TIME_HOURS=\"${SLURM_TIME_HOURS:-4}\"\nexport SLURM_TIMEOUT_HOURS=\"${SLURM_TIMEOUT_HOURS:-3.8}\"\n```\n\nDo not default to 12 hours on SLURM. If the user supplies a longer\n`SLURM_TIME_HOURS`, verify that the selected partition supports it before\nsubmitting. For the packaged default partition list\n`polar,polar3,polar4,grizzly`, reject requests above 4 hours and ask for a\ndifferent partition only if the user actually wants a longer wall time.\n\nWhen `num_gpus` is greater than or equal to `max_num_gpus_per_node`, the\nhandler treats the request as exclusive per node and computes additional nodes\nfrom total GPU count when necessary.\n\nFor multi-node jobs (`num_nodes > 1`), the sbatch script exports `WORLD_SIZE`,\n`MASTER_ADDR`, `MASTER_PORT`, `NODE_RANK`, and `NUM_GPU_PER_NODE`, and Cosmos-RL\nhas special multi-node role handling for controller, policy, and rollout\nworkers. See `references\u002Fmulti-node.md` for the full sbatch directives, the\nrendezvous env-var table and contract, and cluster requirements.\n\n## Monitoring\n\n- Scheduler status comes from the stored SLURM job id via `squeue` or `sacct`.\n- TAO terminal status comes from `status.json` in the shared results folder.\n- If the user enabled chat monitoring, continue polling at the requested\n  interval while the job is `PENDING`, `RUNNING`, or otherwise non-terminal.\n  Do not stop after a fixed elapsed time such as 30 minutes; long queue waits\n  are normal on shared GPU partitions.\n- Do not send a final response for a non-terminal SLURM job when chat\n  monitoring is enabled. A final response is a detach action; use it only if\n  the user asked to detach\u002Fstop or the job reached terminal state.\n- Logs are read over SSH from:\n\n```text\n\u003Cjob_dir>\u002Fslurm-logs\u002F\u003Cslurm_job_name>-\u003Cslurm_job_id>\u002Fmain.out\n\u003Cjob_dir>\u002Fslurm-logs\u002F\u003Cslurm_job_name>-\u003Cslurm_job_id>\u002Fmain.err\n```\n\nStatus mapping:\n\n- `PENDING` -> `Pending`\n- `RUNNING` or `COMPLETING` -> `Running`\n- `COMPLETED` -> check `status.json`\n- `FAILED`, `BOOT_FAIL`, `DEADLINE`, `OUT_OF_MEMORY`, `NODE_FAIL` -> retry if\n  logs match retriable infrastructure patterns, otherwise `Error`\n- `CANCELLED`, `PREEMPTED`, `REVOKED` -> `Canceled`\n- `TIMEOUT` -> `Error`\n- `SUSPENDED`, `STOPPED` -> `Paused`\n\n## Cancellation\n\nCancel by looking up `backend_details.slurm_metadata.slurm_job_id` and running\n`scancel \u003Cslurm_job_id>` over SSH. Treat missing or already terminated SLURM\njobs as successful cancellation.\n\n## Multi-node training (distributed)\n\nSLURM is the platform of choice for large multi-node runs — pass `num_nodes > 1`\nand the SDK handles the sbatch directives and PyTorch-distributed env vars\nautomatically. See `references\u002Fmulti-node.md` for a worked `create_job` example,\nthe generated sbatch directives, the rendezvous env-var table (`WORLD_SIZE`,\n`NUM_GPU_PER_NODE`, `NODE_RANK`, `MASTER_ADDR`, `MASTER_PORT`), the Cosmos-RL\nrole note, cluster requirements (Pyxis\u002FEnroot, InfiniBand\u002FNVLink, Lustre), and\nupstream reference links.\n\n## Running via the TAO SDK\n\nThe SDK install is covered in Preflight — `pip install 'nvidia-tao-sdk[slurm]'`.\nUse it when you want Job handles, the sbatch\u002F`squeue`\u002F`sacct` plumbing handled\nfor you, run-folder durability via `ActionWorkflow`, or convenient cloud-storage\nI\u002FO (`s3:\u002F\u002F`, `hf_model:\u002F\u002F`, `ngc:\u002F\u002F`). Without the SDK, drive `sbatch` and\n`srun` yourself.\n\nAuto-retry is **fully automatic**: a background monitor polls `squeue`\u002F`sacct`\nand re-`sbatch`'s the staged script on infrastructure-looking failures up to\n`MAX_JOB_RETRIES = 10`, while plain training failures surface immediately. In\naddition, `#SBATCH --requeue` is set by default (`SLURM_USE_REQUEUE`, defaults\nto `true`). See `references\u002Fsdk-usage.md` for the `SlurmSDK` \u002F `build_entrypoint`\ncode example, the Lustre-not-S3 rule, the retriable-failure classification, and\nthe full auto-retry and requeue behavior.\n\n## Failure Modes\n\nCommon failures: SSH auth failure, local dataset path rejected, SQSH conversion\ntimeout, Pyxis\u002FEnroot unavailable, and bad-node \u002F transient GPU failures (which\nthe handler retries up to the configured limit). See\n`references\u002Ftroubleshooting.md` for the diagnosis and remediation of each.\n",{"name":5,"tags":24,"license":15,"metadata":27,"description":7,"allowed-tools":30,"compatibility":31},[25,26],"platform","slurm",{"author":28,"version":29},"NVIDIA Corporation","0.2.0","Read Bash","Requires SSH access to a SLURM login node (passwordless via key auth) and SLURM_USER + SLURM_HOSTNAME env vars. The TAO SDK with the slurm extra (pip install 'nvidia-tao-sdk[slurm]') is needed only if you want Job handles, S3 I\u002FO wrapping, or run-folder durability via ActionWorkflow.","main","c475fc64e94e803b50d14a791dfee8853fbb2c5c","skills\u002Fnvidia\u002Ftao-run-on-slurm","https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory","https:\u002F\u002Fgithub.com\u002FElogic360","2026-11-08 22:21:22",{"source":8,"skillName":5,"commands":39,"prompt":49,"zipAvailable":50},[40,43,46],{"tool":41,"command":42},"npx","npx skills add Elogic360\u002Fsoftware-factory --skill tao-run-on-slurm",{"tool":44,"command":45},"bunx","bunx skills add Elogic360\u002Fsoftware-factory --skill tao-run-on-slurm",{"tool":47,"command":48},"pnpm","pnpm dlx skills add Elogic360\u002Fsoftware-factory --skill tao-run-on-slurm","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Ftao-run-on-slurm\u002FSKILL.md",true,[52,58,62,66,70,74,78,82,86],{"path":53,"resourceType":54,"resourceTypeDesc":55,"sizeBytes":56,"sourceHtmlUrl":57},"BENCHMARK.md","REFERENCE","参考文档",4284,"https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Ftao-run-on-slurm\u002FBENCHMARK.md",{"path":59,"resourceType":54,"resourceTypeDesc":55,"sizeBytes":60,"sourceHtmlUrl":61},"evals\u002Fevals.json",762,"https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Ftao-run-on-slurm\u002Fevals\u002Fevals.json",{"path":63,"resourceType":54,"resourceTypeDesc":55,"sizeBytes":64,"sourceHtmlUrl":65},"references\u002Fmulti-node.md",3705,"https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Ftao-run-on-slurm\u002Freferences\u002Fmulti-node.md",{"path":67,"resourceType":54,"resourceTypeDesc":55,"sizeBytes":68,"sourceHtmlUrl":69},"references\u002Fsdk-usage.md",4156,"https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Ftao-run-on-slurm\u002Freferences\u002Fsdk-usage.md",{"path":71,"resourceType":54,"resourceTypeDesc":55,"sizeBytes":72,"sourceHtmlUrl":73},"references\u002Fskill_info.yaml",1282,"https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Ftao-run-on-slurm\u002Freferences\u002Fskill_info.yaml",{"path":75,"resourceType":54,"resourceTypeDesc":55,"sizeBytes":76,"sourceHtmlUrl":77},"references\u002Fssh-setup.md",5253,"https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Ftao-run-on-slurm\u002Freferences\u002Fssh-setup.md",{"path":79,"resourceType":54,"resourceTypeDesc":55,"sizeBytes":80,"sourceHtmlUrl":81},"references\u002Ftroubleshooting.md",1092,"https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Ftao-run-on-slurm\u002Freferences\u002Ftroubleshooting.md",{"path":83,"resourceType":54,"resourceTypeDesc":55,"sizeBytes":84,"sourceHtmlUrl":85},"skill-card.md",3823,"https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Ftao-run-on-slurm\u002Fskill-card.md",{"path":87,"resourceType":88,"resourceTypeDesc":89,"sizeBytes":90,"sourceHtmlUrl":91},"skill.oms.sig","OTHER","其他",5777,"https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Ftao-run-on-slurm\u002Fskill.oms.sig",[93,100,107,114,121,128],{"skillNo":94,"name":95,"nameZh":6,"description":96,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":97,"categories":98,"keywords":99,"publishedAt":19},"01M79G206MT3J6S0GY3C45P2KV","vss-summarize-video","Use to summarize a recorded video via the LVS summarization microservice (HITL-gated) with a VLM fallback. Not for report generation or live RTSP captioning.","https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fvss-summarize-video\u002FSKILL.md",[],[],{"skillNo":101,"name":102,"nameZh":6,"description":103,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":104,"categories":105,"keywords":106,"publishedAt":19},"01M79G1WN63PGRE8CKD47WEBQ5","vss-setup-video-analytics-api","Use to deploy the vss-video-analytics-api REST service standalone (config-source, data-log bind, Elasticsearch, optional Kafka). Not for full warehouse deploy.","https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fvss-setup-video-analytics-api\u002FSKILL.md",[],[],{"skillNo":108,"name":109,"nameZh":6,"description":110,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":111,"categories":112,"keywords":113,"publishedAt":19},"01M79G1S9F7VY3EF6H53PRP5AN","vss-setup-behavior-analytics","Use to deploy the vss-behavior-analytics service standalone (entrypoint, config-source, optional calibration). Not for the full warehouse deploy.","https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fvss-setup-behavior-analytics\u002FSKILL.md",[],[],{"skillNo":115,"name":116,"nameZh":6,"description":117,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":118,"categories":119,"keywords":120,"publishedAt":19},"01M79G1N1S1Y27PFZPYMNK1MTQ","vss-search-archive","Use this skill to run top-level VSS fusion search on archived video, or to ingest video files \u002F RTSP streams for search. Do NOT use for ad-hoc visual Q&A (use vss-ask-video), live captioning (use vss-deploy-dense-captioning), or video summarization and reports (use vss-summarize-video).","https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fvss-search-archive\u002FSKILL.md",[],[],{"skillNo":122,"name":123,"nameZh":6,"description":124,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":125,"categories":126,"keywords":127,"publishedAt":19},"01M79G1H6WN0Q6C0VCAXCZX6B7","vss-query-analytics","Use this skill when reading video-analytics metrics, incidents, alerts, and sensor data via the VA-MCP server (port 9901). Not for live VLM or incident-range narrative reports.","https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fvss-query-analytics\u002FSKILL.md",[],[],{"skillNo":129,"name":130,"nameZh":6,"description":131,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":132,"categories":133,"keywords":134,"publishedAt":19},"01M79G1DYPNXQS7QHZ0AAF5FX5","vss-manage-video-io-storage","Use to call the VIOS REST API (sensor list, timelines, clip extraction, snapshots, add\u002Fdelete sensors and streams). Not for VLM inference or search.","https:\u002F\u002Fgithub.com\u002FElogic360\u002Fsoftware-factory\u002Fblob\u002Fmain\u002Fskills\u002Fnvidia\u002Fvss-manage-video-io-storage\u002FSKILL.md",[],[]]