[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7A72R4NWRY3SWHCCMMX3BCA":3,"skill-same:01M7A72R4NWRY3SWHCCMMX3BCA":104},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":38,"commitSha":39,"skillDirectory":40,"forksCount":41,"repositoryHtmlUrl":42,"repositoryHomepageUrl":43,"ownerHtmlUrl":44,"ownerBlogUrl":45,"ownerDescription":46,"updatedAt":47,"install":48,"resources":61},"01M7A72R4NWRY3SWHCCMMX3BCA","tao-finetune-cosmos-embed",null,"Cosmos-Embed1 video-text embedding for text-to-video retrieval, video-to-video search, semantic deduplication, and fine-tuning. Use when the user asks to \"fine-tune Cosmos-Embed1\", \"run cosmos-embed inference\", \"export Cosmos-Embed1\", \"embed videos\", or \"search videos with text\".","themusebook\u002Fskills","themusebook","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F349927247?v=4","01M7A5C1RV5S3V5FF6ZACVDNRN","musebook","01M7A5C1SFC3KRZAXNN60XJ7KM",0,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fpublic\u002Fapi\u002Fskills\u002Fnvidia\u002Ftao-finetune-cosmos-embed\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: tao-finetune-cosmos-embed\ndescription: >-\n  Cosmos-Embed1 video-text embedding for text-to-video retrieval, video-to-video search, semantic deduplication, and\n  fine-tuning. Use when the user asks to \"fine-tune Cosmos-Embed1\", \"run cosmos-embed inference\", \"export Cosmos-Embed1\",\n  \"embed videos\", or \"search videos with text\".\nlicense: Apache-2.0\ncompatibility: Requires docker + nvidia-container-toolkit, the published Cosmos-Embed TAO container from versions.yaml, and a HuggingFace token when downloading pretrained `nvidia\u002FCosmos-Embed1-*` weights.\nmetadata:\n  author: NVIDIA Corporation\n  version: \"0.1.0\"\nallowed-tools: Read Bash\ntags:\n- video\n- vision-language\n- vlm\n- multimodal\n- retrieval\n- embedding\n- cosmos\n- fine-tuning\n---\n\n# Cosmos-Embed\n\nCosmos-Embed1 is a joint video-text embedder for text-to-video retrieval, video-to-video search, zero-shot\u002FkNN classification, and semantic deduplication. The packaged CLI is `cosmos-embed1` and supports `train`, `evaluate`, `inference`, and `export`.\n\nContainer image and per-action commands are in `references\u002Fskill_info.yaml`. Compact starting specs are in `references\u002Fspec_template_*.yaml`.\n\n## Train Action Policy\n\nAutoML is not packaged for this model skill because there are no Cosmos-Embed schemas under `schemas\u002F`. Always use the direct model skill actions for `train`, `evaluate`, `inference`, and `export`, even when a higher-level request includes `automl_policy: on`. Do not route Cosmos-Embed through workflow or AutoML skills until model-specific train schemas and templates are added.\n\nNon-train actions such as `evaluate`, `inference`, `export`, and deploy flows stay in this model skill. The per-run `automl_policy` override does not change model metadata.\n\n## Quick Start\n\nUse the published Cosmos-Embed container declared by `references\u002Fskill_info.yaml`\nand resolved through `versions.yaml`. Do not build from the private\nCosmos-Embed1 source tree for normal skill use; build from source only when\ndeveloping the container itself.\n\n```bash\nTAO_SKILL_BANK_PATH=\"${TAO_SKILL_BANK_PATH:-$PWD}\"\nCOSMOS_EMBED_IMAGE=\"${COSMOS_EMBED_IMAGE:-$(\n  python \"$TAO_SKILL_BANK_PATH\u002Fscripts\u002Fresolve_tao_image.py\" \\\n    --skill-bank \"$TAO_SKILL_BANK_PATH\" \\\n    --model tao-finetune-cosmos-embed \\\n    --action train \\\n    --format json |\n  python -c 'import json,sys; print(json.load(sys.stdin)[\"image\"])'\n)}\"\ndocker pull \"$COSMOS_EMBED_IMAGE\"\n```\n\nExpected local workspace layout:\n\n```text\nworkspace\u002F\n├── data\u002F\n│   ├── msrvtt_test_1k.json\n│   └── video\u002F\n│       ├── video7020.mp4\n│       └── ...\n├── model\u002F\n│   └── Cosmos-Embed1-224p\u002F        # optional if using HF repo id\n├── specs\u002F\n│   ├── train.yaml\n│   ├── evaluate.yaml\n│   ├── inference.yaml\n│   ├── export_onnx.yaml\n│   └── export_hf.yaml\n└── results\u002F\n```\n\nUse these Docker options for all actions unless the local Docker\u002Fplatform skill gives a stricter environment-specific command:\n\n```bash\nTAO_SKILL_BANK_PATH=\"${TAO_SKILL_BANK_PATH:-$PWD}\"\nCOSMOS_EMBED_IMAGE=\"${COSMOS_EMBED_IMAGE:-$(\n  python \"$TAO_SKILL_BANK_PATH\u002Fscripts\u002Fresolve_tao_image.py\" \\\n    --skill-bank \"$TAO_SKILL_BANK_PATH\" \\\n    --model tao-finetune-cosmos-embed \\\n    --action train \\\n    --format json |\n  python -c 'import json,sys; print(json.load(sys.stdin)[\"image\"])'\n)}\"\nRUN_ROOT=\"${RUN_ROOT:-$PWD}\"\nDOCKER_COMMON=(\n  --rm --gpus all --ipc=host --network=host\n  --shm-size=64g\n  --ulimit memlock=-1\n  --ulimit stack=67108864\n  -e HF_TOKEN\n  -e WANDB_DISABLED=true\n  -e WANDB_MODE=disabled\n  -e HUGGINGFACE_HUB_CACHE=\u002Fhf_cache\n  -v \"$RUN_ROOT\u002Fdata:\u002Fdata:ro\"\n  -v \"$RUN_ROOT\u002Fmodel:\u002Fmodel\"\n  -v \"$RUN_ROOT\u002Fspecs:\u002Fspecs:ro\"\n  -v \"$RUN_ROOT\u002Fresults:\u002Fresults\"\n  -v \"$RUN_ROOT\u002Fhf_cache:\u002Fhf_cache\"\n)\n```\n\nFor Cosmos-Embed images that ship `protobuf==7.x`, run a small startup\npreamble before every action:\n\n```bash\npython -m pip install \"protobuf\u003C7\"\n```\n\nThe image contains `wandb==0.21.0` with `protobuf==7.x`; importing W&B fails before training\u002Fevaluation unless protobuf is pinned below 7. Use `WANDB_DISABLED=true` and `WANDB_MODE=disabled` for smoke or offline runs. Cosmos-Embed may still download the public `google-bert\u002Fbert-base-uncased` Q-Former component even when the model checkpoint is disabled, so pass `HF_TOKEN` as an environment variable or mount a persistent HuggingFace cache. Do not write the token into specs, logs, or reports.\n\nTrain:\n\n```bash\ndocker run \"${DOCKER_COMMON[@]}\" \"$COSMOS_EMBED_IMAGE\" \\\n  bash -lc \"python -m pip install 'protobuf\u003C7' && cosmos-embed1 train -e \u002Fspecs\u002Ftrain.yaml results_dir=\u002Fresults\"\n```\n\nEvaluate:\n\n```bash\ndocker run \"${DOCKER_COMMON[@]}\" \"$COSMOS_EMBED_IMAGE\" \\\n  bash -lc \"python -m pip install 'protobuf\u003C7' && cosmos-embed1 evaluate -e \u002Fspecs\u002Fevaluate.yaml results_dir=\u002Fresults\"\n```\n\nInference:\n\n```bash\ndocker run \"${DOCKER_COMMON[@]}\" \"$COSMOS_EMBED_IMAGE\" \\\n  bash -lc \"python -m pip install 'protobuf\u003C7' && cosmos-embed1 inference -e \u002Fspecs\u002Finference.yaml \\\n  'inference.query.input_texts=[\\\"a man is singing on stage\\\"]' \\\n  inference.k=5 \\\n  results_dir=\u002Fresults\"\n```\n\nExport ONNX:\n\n```bash\ndocker run \"${DOCKER_COMMON[@]}\" \"$COSMOS_EMBED_IMAGE\" \\\n  bash -lc \"python -m pip install 'protobuf\u003C7' && cosmos-embed1 export -e \u002Fspecs\u002Fexport_onnx.yaml \\\n  export.checkpoint=\u002Fresults\u002Ftrain\u002Fcheckpoints\u002Fiter_000000001.pt \\\n  export.onnx_file=\u002Fresults\u002Fexport\u002Fcosmos_embed1_combined.onnx \\\n  results_dir=\u002Fresults\"\n```\n\nExport HuggingFace format:\n\n```bash\ndocker run \"${DOCKER_COMMON[@]}\" \"$COSMOS_EMBED_IMAGE\" \\\n  bash -lc \"python -m pip install 'protobuf\u003C7' && cosmos-embed1 export -e \u002Fspecs\u002Fexport_hf.yaml \\\n  export.checkpoint=\u002Fresults\u002Ftrain\u002Fcheckpoints\u002Fiter_000000001.pt \\\n  export.hf_output_dir=\u002Fresults\u002Fexport_hf\u002Fcosmos_embed1_hf \\\n  results_dir=\u002Fresults\"\n```\n\n## Smoke Overrides\n\nFor a small functional check, keep the same specs and override the expensive knobs:\n\n```bash\ntrain.max_iter=1\ntrain.validation_iter=2\ntrain.checkpoint_iter=1\ntrain.optim.optim=adamw\ntrain.optim.warmup_steps=0\ntrain.optim.lr_decay_iters=1\ndataset.train_dataset.batch_size=1\ndataset.val_dataset.batch_size=1\ndataset.train_dataset.workers=0\ndataset.val_dataset.workers=0\n```\n\nWhen shortening the cosine scheduler for smoke runs, keep\n`train.optim.lr_decay_iters` greater than `train.optim.warmup_steps`, or set\n`train.optim.warmup_steps=0` as shown above. The scheduler divides by\n`lr_decay_iters - warmup_steps`, so equal values fail before the checkpoint is\nwritten.\n\nIf no local Cosmos-Embed1 pretrained checkpoint is available, set `model.pretrained_model_path=null` for a plumbing-only smoke train. The model quality is meaningless in that mode, but the train\u002Fevaluate\u002Finference\u002Fexport action paths can still be exercised. In the current container, the Q-Former path can still fetch `google-bert\u002Fbert-base-uncased`; provide `HF_TOKEN` or a mounted HuggingFace cache for fresh ephemeral containers.\n\nFor evaluation and inference smoke tests on a tiny subset:\n\n```bash\nevaluate.callbacks.embedding_visualization=false\nevaluate.callbacks.max_eval_samples=8\ndataset.test_dataset.batch_size=1\ndataset.test_dataset.workers=0\ninference.k=2\ndataset.inference_dataset.batch_size=1\ndataset.inference_dataset.workers=0\n```\n\n## Data Format\n\nThe MSR-VTT path expects a local video glob and a JSON metadata file:\n\n```yaml\ndataset:\n  train_dataset:\n    dataset_type: msrvtt\n    mp4_urls: \u002Fdata\u002Fvideo\u002F*.mp4\n    metadata: \u002Fdata\u002Fmsrvtt_test_1k.json\n```\n\nList-format metadata rows must include at least `video` and `caption`:\n\n```json\n{\"video_id\": \"video7020\", \"video\": \"video7020.mp4\", \"caption\": \"a woman creating a fondant baby and flower\"}\n```\n\nThe dataset loader derives the video id from the local `.mp4` filename and filters to videos present in the metadata. If a run finds zero videos, check that `mp4_urls` points to a container-local glob and that metadata `video` names match the filenames.\n\n## Model Weights\n\n- Local HF directory: mount it under `\u002Fmodel` and set `model.pretrained_model_path=\u002Fmodel\u002FCosmos-Embed1-224p`.\n- HuggingFace repo: set `model.pretrained_model_path=nvidia\u002FCosmos-Embed1-224p` and pass `HF_TOKEN` if access is gated.\n- Fine-tuned checkpoint: set downstream actions to the resolver-selected `\u002Fresults\u002Ftrain\u002Fcheckpoints\u002Fiter_#########.pt` file.\n\nTraining writes full checkpoints under `results\u002Ftrain\u002Fcheckpoints\u002Fiter_#########.pt`, updates `results\u002Ftrain\u002Fcheckpoints\u002Flatest_checkpoint.txt`, and creates a `cosmos_embed1_model_latest.pth` symlink. For `evaluate.checkpoint`, `inference.checkpoint`, `export.checkpoint`, and `train.resume_training_checkpoint_path`, resolve and pass the exact `iter_#########.pt` file for the intended iteration. The action spec templates intentionally leave these checkpoint fields null so the model-skill runner or the user must provide the resolver-selected checkpoint. Use the latest symlink only when the user explicitly asks for latest.\n\nFor single-GPU resume\u002Fretrain from a consolidated checkpoint, set `model.fsdp_shard_size: 1`. The container default is 8, which sends resumed training through an FSDP apply path that Cosmos-Embed1 does not implement for this model class.\n\nVariants:\n\n| Variant | Resolution | Frames | Embedding dim |\n|---|---:|---:|---:|\n| `Cosmos-Embed1-224p` | 224 x 224 | 8 | 256 |\n| `Cosmos-Embed1-336p` | 336 x 336 | 8 | 768 |\n| `Cosmos-Embed1-448p` | 448 x 448 | 8 | 768 |\n\nKeep `model.network.embed_dim`, `model.input_hw`, and `model.network.spatial_resolution` aligned with the selected variant.\n\n## Important Parameters\n\n| Parameter | Notes |\n|---|---|\n| `train.num_gpus` | `1` for single GPU, `>1` auto-launches `torchrun`, `-1` auto-detects visible GPUs. |\n| `train.max_iter` | Main training length. Use `1` only for smoke testing. |\n| `train.optim.optim` | `fused_adamw` is faster when available; `adamw` is safer for smoke and portability. |\n| `model.lora.enabled` | Enables LoRA. Set `model.network.visual_encoder.transformer_engine=false` when LoRA is on. |\n| `model.lora.lora_rank` | LoRA rank. Start with `8`; try `4`, `8`, or `16` for manual or AutoML-style sweeps. |\n| `model.lora.lora_alpha` | LoRA scaling factor. Start with `16`; keep near `2 * lora_rank` unless experiments show otherwise. |\n| `model.lora.lora_dropout` | LoRA dropout. Start with `0.1`; sweep `0.0`, `0.05`, and `0.1` for small datasets. |\n| `model.lora.bias` | Bias policy: `none`, `all`, or `lora_only`. Keep `none` unless intentionally training biases. |\n| `model.lora.use_rslora` \u002F `use_dora` | Optional LoRA variants. Enable one at a time and record the setting with the checkpoint. |\n| `model.lora.target_modules` | Optional module-name patterns for LoRA injection. Leave empty for the default ViT + Q-Former attention\u002FMLP targets. |\n| `model.lora.modules_to_save` | Optional modules to keep fully trainable alongside LoRA. Leave empty unless preserving a task-specific head. |\n| `evaluate.load_dataset_pkl` \u002F `save_dataset_pkl` | Cache evaluation embeddings. |\n| `inference.load_dataset_pkl` \u002F `save_dataset_pkl` | Cache the search database for repeated retrieval. |\n| `export.mode` | `video`, `text`, `combined`, or `huggingface`. |\n| `export.on_cpu` | Recommended for export to avoid device mismatch issues. |\n\n### LoRA and AutoML Notes\n\nFor parameter-efficient fine-tuning, set `model.lora.enabled=true` and keep\n`model.network.visual_encoder.transformer_engine=false`; TAO Core's\nCosmos-Embed1 config notes that PEFT cannot inject adapters into Transformer\nEngine layers. Treat the LoRA fields above as the first candidate parameters\nfor manual tuning or AutoML-style search before unfreezing larger model blocks.\nAvoid changing `target_modules` or `modules_to_save` unless the user explicitly\nneeds custom adapter placement.\n\n## S3 Staging\n\nThe Cosmos-Embed1 CLI consumes local paths and Python globs, not raw `s3:\u002F\u002F...\u002F*.mp4` URIs. For S3-backed runs, first stage a subset or full dataset to the execution host\u002Fcontainer filesystem, then use local paths such as `\u002Fdata\u002Fvideo\u002F*.mp4` in the spec.\n\nRecommended S3 layout for staged MSR-VTT data:\n\n```text\ns3:\u002F\u002Fbucket\u002Fpath\u002Fcosmos-embed\u002Fmsrvtt-subset\u002F\n├── msrvtt_test_1k.json\n└── video\u002F\n    ├── video7020.mp4\n    └── ...\n```\n\nAfter downloading\u002Fsyncing that prefix into the mounted `data\u002F` directory, use the same Docker commands above.\n\n## Outputs\n\n```text\nresults\u002F\n├── train\u002F\n│   ├── cosmos_embed1_model_latest.pth\n│   ├── cosmos_embed1_model_\u003Citer>.pth\n│   └── experiment.yaml\n├── evaluate\u002F\n│   ├── metrics.json\n│   └── experiment.yaml\n├── inference\u002F\n│   ├── results.json\n│   └── experiment.yaml\n├── export\u002F\n│   ├── cosmos_embed1_combined.onnx\n│   └── export_config.yaml\n└── export_hf\u002F\n    └── cosmos_embed1_hf\u002F\n```\n\n## Known Pitfalls\n\n| Symptom | Cause | Fix |\n|---|---|---|\n| `MSRVTTDataset: 0 videos found` | `mp4_urls` is not a local glob or metadata filenames do not match videos. | Mount data into the container and set `mp4_urls=\u002Fdata\u002Fvideo\u002F*.mp4`. |\n| HF download\u002Fauth failure | Missing or invalid `HF_TOKEN`, or model agreement not accepted. | Accept the model terms and pass `-e HF_TOKEN`. |\n| `cannot import name 'Imports' from 'wandb.proto.wandb_telemetry_pb2'` | `wandb==0.21.0` in the container is incompatible with `protobuf==7.x`. | Run `python -m pip install \"protobuf\u003C7\"` in the container before invoking `cosmos-embed1`. |\n| Resume fails with `Model does not implement 'apply_fsdp'` | Single-GPU resume loaded a consolidated checkpoint while `model.fsdp_shard_size` stayed at the default 8. | Set `model.fsdp_shard_size=1` for local single-GPU resume\u002Fretrain. |\n| LoRA injection failure | Transformer Engine visual encoder is enabled. | Set `model.network.visual_encoder.transformer_engine=false`. |\n| ONNX\u002FHF export complains about missing components | Export checkpoint is partial or adapter-only. | Use a full checkpoint or configure pretrained visual\u002Ftext sources before export. |\n| CUDA OOM | Batch\u002Fresolution too high for the GPU. | Reduce batch size, use 224p, enable LoRA, or use more GPUs. |\n","# Cosmos-Embed\n\nCosmos-Embed1 is a joint video-text embedder for text-to-video retrieval, video-to-video search, zero-shot\u002FkNN classification, and semantic deduplication. The packaged CLI is `cosmos-embed1` and supports `train`, `evaluate`, `inference`, and `export`.\n\nContainer image and per-action commands are in `references\u002Fskill_info.yaml`. Compact starting specs are in `references\u002Fspec_template_*.yaml`.\n\n## Train Action Policy\n\nAutoML is not packaged for this model skill because there are no Cosmos-Embed schemas under `schemas\u002F`. Always use the direct model skill actions for `train`, `evaluate`, `inference`, and `export`, even when a higher-level request includes `automl_policy: on`. Do not route Cosmos-Embed through workflow or AutoML skills until model-specific train schemas and templates are added.\n\nNon-train actions such as `evaluate`, `inference`, `export`, and deploy flows stay in this model skill. The per-run `automl_policy` override does not change model metadata.\n\n## Quick Start\n\nUse the published Cosmos-Embed container declared by `references\u002Fskill_info.yaml`\nand resolved through `versions.yaml`. Do not build from the private\nCosmos-Embed1 source tree for normal skill use; build from source only when\ndeveloping the container itself.\n\n```bash\nTAO_SKILL_BANK_PATH=\"${TAO_SKILL_BANK_PATH:-$PWD}\"\nCOSMOS_EMBED_IMAGE=\"${COSMOS_EMBED_IMAGE:-$(\n  python \"$TAO_SKILL_BANK_PATH\u002Fscripts\u002Fresolve_tao_image.py\" \\\n    --skill-bank \"$TAO_SKILL_BANK_PATH\" \\\n    --model tao-finetune-cosmos-embed \\\n    --action train \\\n    --format json |\n  python -c 'import json,sys; print(json.load(sys.stdin)[\"image\"])'\n)}\"\ndocker pull \"$COSMOS_EMBED_IMAGE\"\n```\n\nExpected local workspace layout:\n\n```text\nworkspace\u002F\n├── data\u002F\n│   ├── msrvtt_test_1k.json\n│   └── video\u002F\n│       ├── video7020.mp4\n│       └── ...\n├── model\u002F\n│   └── Cosmos-Embed1-224p\u002F        # optional if using HF repo id\n├── specs\u002F\n│   ├── train.yaml\n│   ├── evaluate.yaml\n│   ├── inference.yaml\n│   ├── export_onnx.yaml\n│   └── export_hf.yaml\n└── results\u002F\n```\n\nUse these Docker options for all actions unless the local Docker\u002Fplatform skill gives a stricter environment-specific command:\n\n```bash\nTAO_SKILL_BANK_PATH=\"${TAO_SKILL_BANK_PATH:-$PWD}\"\nCOSMOS_EMBED_IMAGE=\"${COSMOS_EMBED_IMAGE:-$(\n  python \"$TAO_SKILL_BANK_PATH\u002Fscripts\u002Fresolve_tao_image.py\" \\\n    --skill-bank \"$TAO_SKILL_BANK_PATH\" \\\n    --model tao-finetune-cosmos-embed \\\n    --action train \\\n    --format json |\n  python -c 'import json,sys; print(json.load(sys.stdin)[\"image\"])'\n)}\"\nRUN_ROOT=\"${RUN_ROOT:-$PWD}\"\nDOCKER_COMMON=(\n  --rm --gpus all --ipc=host --network=host\n  --shm-size=64g\n  --ulimit memlock=-1\n  --ulimit stack=67108864\n  -e HF_TOKEN\n  -e WANDB_DISABLED=true\n  -e WANDB_MODE=disabled\n  -e HUGGINGFACE_HUB_CACHE=\u002Fhf_cache\n  -v \"$RUN_ROOT\u002Fdata:\u002Fdata:ro\"\n  -v \"$RUN_ROOT\u002Fmodel:\u002Fmodel\"\n  -v \"$RUN_ROOT\u002Fspecs:\u002Fspecs:ro\"\n  -v \"$RUN_ROOT\u002Fresults:\u002Fresults\"\n  -v \"$RUN_ROOT\u002Fhf_cache:\u002Fhf_cache\"\n)\n```\n\nFor Cosmos-Embed images that ship `protobuf==7.x`, run a small startup\npreamble before every action:\n\n```bash\npython -m pip install \"protobuf\u003C7\"\n```\n\nThe image contains `wandb==0.21.0` with `protobuf==7.x`; importing W&B fails before training\u002Fevaluation unless protobuf is pinned below 7. Use `WANDB_DISABLED=true` and `WANDB_MODE=disabled` for smoke or offline runs. Cosmos-Embed may still download the public `google-bert\u002Fbert-base-uncased` Q-Former component even when the model checkpoint is disabled, so pass `HF_TOKEN` as an environment variable or mount a persistent HuggingFace cache. Do not write the token into specs, logs, or reports.\n\nTrain:\n\n```bash\ndocker run \"${DOCKER_COMMON[@]}\" \"$COSMOS_EMBED_IMAGE\" \\\n  bash -lc \"python -m pip install 'protobuf\u003C7' && cosmos-embed1 train -e \u002Fspecs\u002Ftrain.yaml results_dir=\u002Fresults\"\n```\n\nEvaluate:\n\n```bash\ndocker run \"${DOCKER_COMMON[@]}\" \"$COSMOS_EMBED_IMAGE\" \\\n  bash -lc \"python -m pip install 'protobuf\u003C7' && cosmos-embed1 evaluate -e \u002Fspecs\u002Fevaluate.yaml results_dir=\u002Fresults\"\n```\n\nInference:\n\n```bash\ndocker run \"${DOCKER_COMMON[@]}\" \"$COSMOS_EMBED_IMAGE\" \\\n  bash -lc \"python -m pip install 'protobuf\u003C7' && cosmos-embed1 inference -e \u002Fspecs\u002Finference.yaml \\\n  'inference.query.input_texts=[\\\"a man is singing on stage\\\"]' \\\n  inference.k=5 \\\n  results_dir=\u002Fresults\"\n```\n\nExport ONNX:\n\n```bash\ndocker run \"${DOCKER_COMMON[@]}\" \"$COSMOS_EMBED_IMAGE\" \\\n  bash -lc \"python -m pip install 'protobuf\u003C7' && cosmos-embed1 export -e \u002Fspecs\u002Fexport_onnx.yaml \\\n  export.checkpoint=\u002Fresults\u002Ftrain\u002Fcheckpoints\u002Fiter_000000001.pt \\\n  export.onnx_file=\u002Fresults\u002Fexport\u002Fcosmos_embed1_combined.onnx \\\n  results_dir=\u002Fresults\"\n```\n\nExport HuggingFace format:\n\n```bash\ndocker run \"${DOCKER_COMMON[@]}\" \"$COSMOS_EMBED_IMAGE\" \\\n  bash -lc \"python -m pip install 'protobuf\u003C7' && cosmos-embed1 export -e \u002Fspecs\u002Fexport_hf.yaml \\\n  export.checkpoint=\u002Fresults\u002Ftrain\u002Fcheckpoints\u002Fiter_000000001.pt \\\n  export.hf_output_dir=\u002Fresults\u002Fexport_hf\u002Fcosmos_embed1_hf \\\n  results_dir=\u002Fresults\"\n```\n\n## Smoke Overrides\n\nFor a small functional check, keep the same specs and override the expensive knobs:\n\n```bash\ntrain.max_iter=1\ntrain.validation_iter=2\ntrain.checkpoint_iter=1\ntrain.optim.optim=adamw\ntrain.optim.warmup_steps=0\ntrain.optim.lr_decay_iters=1\ndataset.train_dataset.batch_size=1\ndataset.val_dataset.batch_size=1\ndataset.train_dataset.workers=0\ndataset.val_dataset.workers=0\n```\n\nWhen shortening the cosine scheduler for smoke runs, keep\n`train.optim.lr_decay_iters` greater than `train.optim.warmup_steps`, or set\n`train.optim.warmup_steps=0` as shown above. The scheduler divides by\n`lr_decay_iters - warmup_steps`, so equal values fail before the checkpoint is\nwritten.\n\nIf no local Cosmos-Embed1 pretrained checkpoint is available, set `model.pretrained_model_path=null` for a plumbing-only smoke train. The model quality is meaningless in that mode, but the train\u002Fevaluate\u002Finference\u002Fexport action paths can still be exercised. In the current container, the Q-Former path can still fetch `google-bert\u002Fbert-base-uncased`; provide `HF_TOKEN` or a mounted HuggingFace cache for fresh ephemeral containers.\n\nFor evaluation and inference smoke tests on a tiny subset:\n\n```bash\nevaluate.callbacks.embedding_visualization=false\nevaluate.callbacks.max_eval_samples=8\ndataset.test_dataset.batch_size=1\ndataset.test_dataset.workers=0\ninference.k=2\ndataset.inference_dataset.batch_size=1\ndataset.inference_dataset.workers=0\n```\n\n## Data Format\n\nThe MSR-VTT path expects a local video glob and a JSON metadata file:\n\n```yaml\ndataset:\n  train_dataset:\n    dataset_type: msrvtt\n    mp4_urls: \u002Fdata\u002Fvideo\u002F*.mp4\n    metadata: \u002Fdata\u002Fmsrvtt_test_1k.json\n```\n\nList-format metadata rows must include at least `video` and `caption`:\n\n```json\n{\"video_id\": \"video7020\", \"video\": \"video7020.mp4\", \"caption\": \"a woman creating a fondant baby and flower\"}\n```\n\nThe dataset loader derives the video id from the local `.mp4` filename and filters to videos present in the metadata. If a run finds zero videos, check that `mp4_urls` points to a container-local glob and that metadata `video` names match the filenames.\n\n## Model Weights\n\n- Local HF directory: mount it under `\u002Fmodel` and set `model.pretrained_model_path=\u002Fmodel\u002FCosmos-Embed1-224p`.\n- HuggingFace repo: set `model.pretrained_model_path=nvidia\u002FCosmos-Embed1-224p` and pass `HF_TOKEN` if access is gated.\n- Fine-tuned checkpoint: set downstream actions to the resolver-selected `\u002Fresults\u002Ftrain\u002Fcheckpoints\u002Fiter_#########.pt` file.\n\nTraining writes full checkpoints under `results\u002Ftrain\u002Fcheckpoints\u002Fiter_#########.pt`, updates `results\u002Ftrain\u002Fcheckpoints\u002Flatest_checkpoint.txt`, and creates a `cosmos_embed1_model_latest.pth` symlink. For `evaluate.checkpoint`, `inference.checkpoint`, `export.checkpoint`, and `train.resume_training_checkpoint_path`, resolve and pass the exact `iter_#########.pt` file for the intended iteration. The action spec templates intentionally leave these checkpoint fields null so the model-skill runner or the user must provide the resolver-selected checkpoint. Use the latest symlink only when the user explicitly asks for latest.\n\nFor single-GPU resume\u002Fretrain from a consolidated checkpoint, set `model.fsdp_shard_size: 1`. The container default is 8, which sends resumed training through an FSDP apply path that Cosmos-Embed1 does not implement for this model class.\n\nVariants:\n\n| Variant | Resolution | Frames | Embedding dim |\n|---|---:|---:|---:|\n| `Cosmos-Embed1-224p` | 224 x 224 | 8 | 256 |\n| `Cosmos-Embed1-336p` | 336 x 336 | 8 | 768 |\n| `Cosmos-Embed1-448p` | 448 x 448 | 8 | 768 |\n\nKeep `model.network.embed_dim`, `model.input_hw`, and `model.network.spatial_resolution` aligned with the selected variant.\n\n## Important Parameters\n\n| Parameter | Notes |\n|---|---|\n| `train.num_gpus` | `1` for single GPU, `>1` auto-launches `torchrun`, `-1` auto-detects visible GPUs. |\n| `train.max_iter` | Main training length. Use `1` only for smoke testing. |\n| `train.optim.optim` | `fused_adamw` is faster when available; `adamw` is safer for smoke and portability. |\n| `model.lora.enabled` | Enables LoRA. Set `model.network.visual_encoder.transformer_engine=false` when LoRA is on. |\n| `model.lora.lora_rank` | LoRA rank. Start with `8`; try `4`, `8`, or `16` for manual or AutoML-style sweeps. |\n| `model.lora.lora_alpha` | LoRA scaling factor. Start with `16`; keep near `2 * lora_rank` unless experiments show otherwise. |\n| `model.lora.lora_dropout` | LoRA dropout. Start with `0.1`; sweep `0.0`, `0.05`, and `0.1` for small datasets. |\n| `model.lora.bias` | Bias policy: `none`, `all`, or `lora_only`. Keep `none` unless intentionally training biases. |\n| `model.lora.use_rslora` \u002F `use_dora` | Optional LoRA variants. Enable one at a time and record the setting with the checkpoint. |\n| `model.lora.target_modules` | Optional module-name patterns for LoRA injection. Leave empty for the default ViT + Q-Former attention\u002FMLP targets. |\n| `model.lora.modules_to_save` | Optional modules to keep fully trainable alongside LoRA. Leave empty unless preserving a task-specific head. |\n| `evaluate.load_dataset_pkl` \u002F `save_dataset_pkl` | Cache evaluation embeddings. |\n| `inference.load_dataset_pkl` \u002F `save_dataset_pkl` | Cache the search database for repeated retrieval. |\n| `export.mode` | `video`, `text`, `combined`, or `huggingface`. |\n| `export.on_cpu` | Recommended for export to avoid device mismatch issues. |\n\n### LoRA and AutoML Notes\n\nFor parameter-efficient fine-tuning, set `model.lora.enabled=true` and keep\n`model.network.visual_encoder.transformer_engine=false`; TAO Core's\nCosmos-Embed1 config notes that PEFT cannot inject adapters into Transformer\nEngine layers. Treat the LoRA fields above as the first candidate parameters\nfor manual tuning or AutoML-style search before unfreezing larger model blocks.\nAvoid changing `target_modules` or `modules_to_save` unless the user explicitly\nneeds custom adapter placement.\n\n## S3 Staging\n\nThe Cosmos-Embed1 CLI consumes local paths and Python globs, not raw `s3:\u002F\u002F...\u002F*.mp4` URIs. For S3-backed runs, first stage a subset or full dataset to the execution host\u002Fcontainer filesystem, then use local paths such as `\u002Fdata\u002Fvideo\u002F*.mp4` in the spec.\n\nRecommended S3 layout for staged MSR-VTT data:\n\n```text\ns3:\u002F\u002Fbucket\u002Fpath\u002Fcosmos-embed\u002Fmsrvtt-subset\u002F\n├── msrvtt_test_1k.json\n└── video\u002F\n    ├── video7020.mp4\n    └── ...\n```\n\nAfter downloading\u002Fsyncing that prefix into the mounted `data\u002F` directory, use the same Docker commands above.\n\n## Outputs\n\n```text\nresults\u002F\n├── train\u002F\n│   ├── cosmos_embed1_model_latest.pth\n│   ├── cosmos_embed1_model_\u003Citer>.pth\n│   └── experiment.yaml\n├── evaluate\u002F\n│   ├── metrics.json\n│   └── experiment.yaml\n├── inference\u002F\n│   ├── results.json\n│   └── experiment.yaml\n├── export\u002F\n│   ├── cosmos_embed1_combined.onnx\n│   └── export_config.yaml\n└── export_hf\u002F\n    └── cosmos_embed1_hf\u002F\n```\n\n## Known Pitfalls\n\n| Symptom | Cause | Fix |\n|---|---|---|\n| `MSRVTTDataset: 0 videos found` | `mp4_urls` is not a local glob or metadata filenames do not match videos. | Mount data into the container and set `mp4_urls=\u002Fdata\u002Fvideo\u002F*.mp4`. |\n| HF download\u002Fauth failure | Missing or invalid `HF_TOKEN`, or model agreement not accepted. | Accept the model terms and pass `-e HF_TOKEN`. |\n| `cannot import name 'Imports' from 'wandb.proto.wandb_telemetry_pb2'` | `wandb==0.21.0` in the container is incompatible with `protobuf==7.x`. | Run `python -m pip install \"protobuf\u003C7\"` in the container before invoking `cosmos-embed1`. |\n| Resume fails with `Model does not implement 'apply_fsdp'` | Single-GPU resume loaded a consolidated checkpoint while `model.fsdp_shard_size` stayed at the default 8. | Set `model.fsdp_shard_size=1` for local single-GPU resume\u002Fretrain. |\n| LoRA injection failure | Transformer Engine visual encoder is enabled. | Set `model.network.visual_encoder.transformer_engine=false`. |\n| ONNX\u002FHF export complains about missing components | Export checkpoint is partial or adapter-only. | Use a full checkpoint or configure pretrained visual\u002Ftext sources before export. |\n| CUDA OOM | Batch\u002Fresolution too high for the GPU. | Reduce batch size, use 224p, enable LoRA, or use more GPUs. |\n",{"name":5,"tags":24,"license":15,"metadata":33,"description":7,"allowed-tools":36,"compatibility":37},[25,26,27,28,29,30,31,32],"video","vision-language","vlm","multimodal","retrieval","embedding","cosmos","fine-tuning",{"author":34,"version":35},"NVIDIA Corporation","0.1.0","Read Bash","Requires docker + nvidia-container-toolkit, the published Cosmos-Embed TAO container from versions.yaml, and a HuggingFace token when downloading pretrained `nvidia\u002FCosmos-Embed1-*` weights.","main","1c523aba5cdf867a79f078847075492d7778092a","public\u002Fapi\u002Fskills\u002Fnvidia\u002Ftao-finetune-cosmos-embed",1,"https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills","https:\u002F\u002Fskills.x402.wtf","https:\u002F\u002Fgithub.com\u002Fthemusebook","https:\u002F\u002Fmusebook.trade","The official Github of musebook","2026-11-02 23:42:26",{"source":8,"skillName":5,"commands":49,"prompt":59,"zipAvailable":60},[50,53,56],{"tool":51,"command":52},"npx","npx skills add themusebook\u002Fskills --skill tao-finetune-cosmos-embed",{"tool":54,"command":55},"bunx","bunx skills add themusebook\u002Fskills --skill tao-finetune-cosmos-embed",{"tool":57,"command":58},"pnpm","pnpm dlx skills add themusebook\u002Fskills --skill tao-finetune-cosmos-embed","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fpublic\u002Fapi\u002Fskills\u002Fnvidia\u002Ftao-finetune-cosmos-embed\u002FSKILL.md",true,[62,68,72,76,80,84,88,92,96,100],{"path":63,"resourceType":64,"resourceTypeDesc":65,"sizeBytes":66,"sourceHtmlUrl":67},"BENCHMARK.md","REFERENCE","参考文档",3996,"https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fpublic\u002Fapi\u002Fskills\u002Fnvidia\u002Ftao-finetune-cosmos-embed\u002FBENCHMARK.md",{"path":69,"resourceType":64,"resourceTypeDesc":65,"sizeBytes":70,"sourceHtmlUrl":71},"metadata.json",515,"https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fpublic\u002Fapi\u002Fskills\u002Fnvidia\u002Ftao-finetune-cosmos-embed\u002Fmetadata.json",{"path":73,"resourceType":64,"resourceTypeDesc":65,"sizeBytes":74,"sourceHtmlUrl":75},"references\u002Fskill_info.yaml",2780,"https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fpublic\u002Fapi\u002Fskills\u002Fnvidia\u002Ftao-finetune-cosmos-embed\u002Freferences\u002Fskill_info.yaml",{"path":77,"resourceType":64,"resourceTypeDesc":65,"sizeBytes":78,"sourceHtmlUrl":79},"references\u002Fspec_template_evaluate.yaml",1175,"https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fpublic\u002Fapi\u002Fskills\u002Fnvidia\u002Ftao-finetune-cosmos-embed\u002Freferences\u002Fspec_template_evaluate.yaml",{"path":81,"resourceType":64,"resourceTypeDesc":65,"sizeBytes":82,"sourceHtmlUrl":83},"references\u002Fspec_template_export_hf.yaml",755,"https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fpublic\u002Fapi\u002Fskills\u002Fnvidia\u002Ftao-finetune-cosmos-embed\u002Freferences\u002Fspec_template_export_hf.yaml",{"path":85,"resourceType":64,"resourceTypeDesc":65,"sizeBytes":86,"sourceHtmlUrl":87},"references\u002Fspec_template_export_onnx.yaml",827,"https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fpublic\u002Fapi\u002Fskills\u002Fnvidia\u002Ftao-finetune-cosmos-embed\u002Freferences\u002Fspec_template_export_onnx.yaml",{"path":89,"resourceType":64,"resourceTypeDesc":65,"sizeBytes":90,"sourceHtmlUrl":91},"references\u002Fspec_template_inference.yaml",1130,"https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fpublic\u002Fapi\u002Fskills\u002Fnvidia\u002Ftao-finetune-cosmos-embed\u002Freferences\u002Fspec_template_inference.yaml",{"path":93,"resourceType":64,"resourceTypeDesc":65,"sizeBytes":94,"sourceHtmlUrl":95},"references\u002Fspec_template_train.yaml",2282,"https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fpublic\u002Fapi\u002Fskills\u002Fnvidia\u002Ftao-finetune-cosmos-embed\u002Freferences\u002Fspec_template_train.yaml",{"path":97,"resourceType":64,"resourceTypeDesc":65,"sizeBytes":98,"sourceHtmlUrl":99},"skill-card.md",4102,"https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fpublic\u002Fapi\u002Fskills\u002Fnvidia\u002Ftao-finetune-cosmos-embed\u002Fskill-card.md",{"path":101,"resourceType":64,"resourceTypeDesc":65,"sizeBytes":102,"sourceHtmlUrl":103},"verification.json",2589,"https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fpublic\u002Fapi\u002Fskills\u002Fnvidia\u002Ftao-finetune-cosmos-embed\u002Fverification.json",[105,114,121,128,135,142],{"skillNo":106,"name":107,"nameZh":6,"description":108,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":109,"licenseFileUrl":110,"sourceHtmlUrl":111,"categories":112,"keywords":113,"publishedAt":19},"01M7AB34SPKVSDRRG009PM1CZH","youtube-clipper","YouTube 视频智能剪辑工具。下载视频和字幕，AI 分析生成精细章节（几分钟级别）， 用户选择片段后自动剪辑、翻译字幕为中英双语、烧录字幕到视频，并生成总结文案。 使用场景：当用户需要剪辑 YouTube 视频、生成短视频片段、制作双语字幕版本时。 关键词：视频剪辑、YouTube、字幕翻译、双语字幕、视频下载、clip video","MIT","https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fyoutube-clipper\u002FLICENSE","https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fyoutube-clipper\u002FSKILL.md",[],[],{"skillNo":115,"name":116,"nameZh":6,"description":117,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":109,"licenseFileUrl":6,"sourceHtmlUrl":118,"categories":119,"keywords":120,"publishedAt":19},"01M7AB18BH6N7KR7939HRNWVMQ","viem-integration","Integrate EVM blockchains using viem. Use when user says \"read blockchain data\", \"send transaction\", \"interact with smart contract\", \"connect to Ethereum\", \"use viem\", \"use wagmi\", \"wallet integration\", \"viem setup\", or mentions blockchain\u002FEVM development with TypeScript.","https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fviem-integration\u002FSKILL.md",[],[],{"skillNo":122,"name":123,"nameZh":6,"description":124,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":109,"licenseFileUrl":6,"sourceHtmlUrl":125,"categories":126,"keywords":127,"publishedAt":19},"01M7AB12GHGP48WCNXDQQ9222H","v4-security-foundations","Security-first Uniswap v4 hook development. Use when user mentions \"v4 hooks\", \"hook security\", \"PoolManager\", \"beforeSwap\", \"afterSwap\", or asks about V4 hook best practices, vulnerabilities, or audit requirements.","https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fv4-security-foundations\u002FSKILL.md",[],[],{"skillNo":129,"name":130,"nameZh":6,"description":131,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":109,"licenseFileUrl":6,"sourceHtmlUrl":132,"categories":133,"keywords":134,"publishedAt":19},"01M7AB0XM3PSR7V3Q2ZRR0Q0FB","v4-sdk-integration","App-layer SDK guide for building swap and liquidity experiences directly with the Uniswap v4 SDK. Use when user asks about \"v4 sdk\", \"uniswap v4\", \"v4 swap\", \"v4 liquidity\", \"PoolManager\", \"V4Planner\", \"StateView\", \"PositionManager\", \"pool state\", \"v4 position\", \"uniswap sdk\", or when building swap\u002Fliquidity UX directly with SDKs rather than via the Trading API.","https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fv4-sdk-integration\u002FSKILL.md",[],[],{"skillNo":136,"name":137,"nameZh":6,"description":138,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":109,"licenseFileUrl":6,"sourceHtmlUrl":139,"categories":140,"keywords":141,"publishedAt":19},"01M7AB0SGA93ASA588DJW4P9WT","v4-hook-generator","Generate Uniswap v4 hook contracts via OpenZeppelin MCP. Use when building custom swap logic, async swaps, hook-owned liquidity, custom curves, dynamic fees, MEV protection, limit orders, or oracle hooks.","https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Fv4-hook-generator\u002FSKILL.md",[],[],{"skillNo":143,"name":144,"nameZh":6,"description":145,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":109,"licenseFileUrl":6,"sourceHtmlUrl":146,"categories":147,"keywords":148,"publishedAt":19},"01M7AAZYH75H4CS03WSW2EZ8DM","testing","For testing with Light Protocol programs and clients on localnet, devnet, and mainnet validation.","https:\u002F\u002Fgithub.com\u002Fthemusebook\u002Fskills\u002Fblob\u002Fmain\u002Fskills\u002Ftesting\u002FSKILL.md",[],[]]