[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6MB9QBZPMBSGAJGVPDQ90WK":3,"skill-same:01M6MB9QBZPMBSGAJGVPDQ90WK":85},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":40,"linkOnly":41,"markdownContent":42,"markdownBody":43,"frontMatter":44,"refName":51,"commitSha":52,"skillDirectory":53,"forksCount":54,"repositoryHtmlUrl":55,"repositoryHomepageUrl":56,"ownerHtmlUrl":57,"ownerBlogUrl":58,"ownerDescription":59,"updatedAt":60,"install":61,"resources":74},"01M6MB9QBZPMBSGAJGVPDQ90WK","torchforge-rl-training",null,"Provides guidance for PyTorch-native agentic RL using torchforge, Meta's library separating infra from algorithms. Use when you want clean RL abstractions, easy algorithm experimentation, or scalable training with Monarch and TorchTitan.","hashgraph-online\u002Fawesome-codex-plugins","hashgraph-online","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F160371962?v=4","01M6CNB6NRJVNNGN14TR01Q6D3","HOL (Hashgraph Online)","01M6JT1HSKKG9V68KQAQGBA7XP",1559,"MIT","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Ftorchforge-rl-training\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39],"training","torchforge rl training","torchforge","rl","clean","when","for","you","native","Scalable","experimentation","algorithms","infra","agentic","use","learning","Meta","vLLM","library","ml","pytorch","2026-11-10 10:12:30",false,"---\nname: torchforge-rl-training\ndescription: Provides guidance for PyTorch-native agentic RL using torchforge, Meta's library separating infra from algorithms. Use when you want clean RL abstractions, easy algorithm experimentation, or scalable training with Monarch and TorchTitan.\nlicense: MIT\nmetadata:\n  version: 1.0.0\n  category: ml-training\n  maintainer: Kalaris Labs\n  tags: Reinforcement Learning, PyTorch, GRPO, SFT, Monarch, TorchTitan, Meta\n  dependencies: torch>=2.9.0, torchtitan>=0.2.0, vllm, monarch\n---\n\n# torchforge: PyTorch-Native Agentic RL Library\n\ntorchforge is Meta's PyTorch-native RL library that separates infrastructure concerns from algorithm concerns. It enables rapid RL research by letting you focus on algorithms while handling distributed training, inference, and weight sync automatically.\n\n## When to Use torchforge\n\n**Choose torchforge when you need:**\n- Clean separation between RL algorithms and infrastructure\n- PyTorch-native abstractions (no Ray dependency)\n- Easy algorithm experimentation (GRPO, DAPO, SAPO in ~100 lines)\n- Scalable training with Monarch actor system\n- Integration with TorchTitan for model parallelism\n\n**Consider alternatives when:**\n- You need production-ready stability → use **miles** or **verl**\n- You want Megatron-native training → use **slime**\n- torchforge is experimental and APIs may change\n\n## Key Features\n\n- **Algorithm isolation**: Implement RL algorithms without touching infrastructure\n- **Scalability**: From single GPU to thousands via Monarch\n- **Modern stack**: TorchTitan (training), vLLM (inference), TorchStore (sync)\n- **Loss functions**: GRPO, DAPO, CISPO, GSPO, SAPO built-in\n\n## Architecture Overview\n\n```\n┌─────────────────────────────────────────────────────────┐\n│ Application Layer (Your Code)                           │\n│ - Define reward models, loss functions, sampling        │\n└─────────────────────┬───────────────────────────────────┘\n                      │\n┌─────────────────────▼───────────────────────────────────┐\n│ Forge API Layer                                         │\n│ - Episode, Group dataclasses                           │\n│ - Service interfaces (async\u002Fawait)                      │\n└─────────────────────┬───────────────────────────────────┘\n                      │\n┌─────────────────────▼───────────────────────────────────┐\n│ Distributed Services (Monarch)                          │\n│ ├── Trainer (TorchTitan FSDP)                          │\n│ ├── Generator (vLLM inference)                          │\n│ ├── Reference Model (frozen KL baseline)               │\n│ └── Reward Actors (compute rewards)                    │\n└─────────────────────────────────────────────────────────┘\n```\n\n## Installation\n\n```bash\n# Create environment\nconda create -n forge python=3.12\nconda activate forge\n\n# Install (handles PyTorch nightly + dependencies)\n.\u002Fscripts\u002Finstall.sh\n\n# Verify\npython -c \"import torch, forge, vllm; print('OK')\"\n```\n\n### ROCm Installation\n\n```bash\n.\u002Fscripts\u002Finstall_rocm.sh\n```\n\n## Quick Start\n\n### SFT Training (2+ GPUs)\n\n```bash\npython -m apps.sft.main --config apps\u002Fsft\u002Fllama3_8b.yaml\n```\n\n### GRPO Training (3+ GPUs)\n\n```bash\npython -m apps.grpo.main --config apps\u002Fgrpo\u002Fqwen3_1_7b.yaml\n```\n\n---\n\n## Workflow 1: GRPO Training for Math Reasoning\n\nUse this workflow for training reasoning models with group-relative advantages.\n\n### Prerequisites Checklist\n- [ ] 3+ GPUs (GPU0: trainer, GPU1: ref_model, GPU2: generator)\n- [ ] Model from HuggingFace Hub\n- [ ] Training dataset (GSM8K, MATH, etc.)\n\n### Step 1: Create Configuration\n\n```yaml\n# config\u002Fgrpo_math.yaml\nmodel: \"Qwen\u002FQwen2.5-7B-Instruct\"\n\ndataset:\n  path: \"openai\u002Fgsm8k\"\n  split: \"train\"\n  streaming: true\n\ntraining:\n  batch_size: 4\n  learning_rate: 1e-6\n  seq_len: 4096\n  dtype: bfloat16\n  gradient_accumulation_steps: 4\n\ngrpo:\n  n_samples: 8           # Responses per prompt\n  clip_low: 0.2\n  clip_high: 0.28\n  beta: 0.1              # KL penalty coefficient\n  temperature: 0.7\n\nservices:\n  generator:\n    procs: 1\n    num_replicas: 1\n    with_gpus: true\n  trainer:\n    procs: 1\n    num_replicas: 1\n    with_gpus: true\n  ref_model:\n    procs: 1\n    num_replicas: 1\n    with_gpus: true\n```\n\n### Step 2: Define Reward Function\n\n```python\n# rewards.py\n# Reward functions are in forge.data.rewards\nfrom forge.data.rewards import MathReward, ThinkingReward\nimport re\n\n# Or define your own reward function\nclass CustomMathReward:\n    def __call__(self, prompt: str, response: str, target: str) -> float:\n        # Extract answer from response\n        match = re.search(r'\\\\boxed{([^}]+)}', response)\n        if not match:\n            return 0.0\n\n        answer = match.group(1).strip()\n        return 1.0 if answer == target else 0.0\n```\n\n### Step 3: Launch Training\n\n```bash\npython -m apps.grpo.main --config config\u002Fgrpo_math.yaml\n```\n\n### Step 4: Monitor Progress\n- [ ] Check W&B dashboard for loss curves\n- [ ] Verify entropy is decreasing (policy becoming more deterministic)\n- [ ] Monitor KL divergence (should stay bounded)\n\n---\n\n## Workflow 2: Custom Loss Function\n\nUse this workflow to implement new RL algorithms.\n\n### Step 1: Create Loss Class\n\n```python\n# src\u002Fforge\u002Flosses\u002Fcustom_loss.py\nimport torch\nimport torch.nn as nn\n\nclass CustomLoss(nn.Module):\n    def __init__(self, clip_range: float = 0.2, beta: float = 0.1):\n        super().__init__()\n        self.clip_range = clip_range\n        self.beta = beta\n\n    def forward(\n        self,\n        logprobs: torch.Tensor,\n        ref_logprobs: torch.Tensor,\n        advantages: torch.Tensor,\n        padding_mask: torch.Tensor,\n    ) -> torch.Tensor:\n        # Compute importance ratio\n        ratio = torch.exp(logprobs - ref_logprobs)\n\n        # Clipped policy gradient\n        clipped_ratio = torch.clamp(\n            ratio,\n            1 - self.clip_range,\n            1 + self.clip_range\n        )\n        pg_loss = -torch.min(ratio * advantages, clipped_ratio * advantages)\n\n        # KL penalty\n        kl = ref_logprobs - logprobs\n\n        # Apply mask and aggregate\n        masked_loss = (pg_loss + self.beta * kl) * padding_mask\n        loss = masked_loss.sum() \u002F padding_mask.sum()\n\n        return loss\n```\n\n### Step 2: Integrate into Application\n\n```python\n# apps\u002Fcustom\u002Fmain.py\nfrom forge.losses.custom_loss import CustomLoss\n\nloss_fn = CustomLoss(clip_range=0.2, beta=0.1)\n\n# In training loop\nloss = loss_fn(\n    logprobs=logprobs,\n    ref_logprobs=ref_logprobs,\n    advantages=advantages,\n    padding_mask=padding_mask,\n)\n```\n\n---\n\n## Workflow 3: Multi-GPU Distributed Training\n\nUse this workflow for scaling to multiple GPUs or nodes.\n\n### Configuration for Distributed\n\n```yaml\n# config\u002Fdistributed.yaml\nmodel: \"meta-llama\u002FMeta-Llama-3.1-8B-Instruct\"\n\nparallelism:\n  tensor_parallel_degree: 2    # Split model across GPUs\n  pipeline_parallel_degree: 1\n  data_parallel_shard_degree: 2\n\nservices:\n  generator:\n    procs: 2                   # 2 processes for TP=2\n    num_replicas: 1\n    with_gpus: true\n  trainer:\n    procs: 2\n    num_replicas: 1\n    with_gpus: true\n```\n\n### Launch with SLURM\n\n```bash\n# Submit job\nsbatch --nodes=2 --gpus-per-node=8 run_grpo.sh\n```\n\n### Launch Locally (Multi-GPU)\n\n```bash\n# 8 GPU setup\npython -m apps.grpo.main \\\n    --config config\u002Fdistributed.yaml \\\n    --trainer.procs 4 \\\n    --generator.procs 4\n```\n\n---\n\n## Core API Reference\n\n### Training Batch Format\n\ntorchforge uses dictionary-based batches for training:\n\n```python\n# inputs: list of dicts with torch.Tensor values\ninputs = [{\"tokens\": torch.Tensor}]\n\n# targets: list of dicts with training signals\ntargets = [{\n    \"response\": torch.Tensor,\n    \"ref_logprobs\": torch.Tensor,\n    \"advantages\": torch.Tensor,\n    \"padding_mask\": torch.Tensor\n}]\n\n# train_step returns loss as float\nloss = trainer.train_step(inputs, targets)\n```\n\n### Completion\n\nGenerated output from vLLM:\n\n```python\n@dataclass\nclass Completion:\n    text: str              # Generated text\n    token_ids: list[int]   # Token IDs\n    logprobs: list[float]  # Log probabilities\n    metadata: dict         # Custom metadata\n```\n\n---\n\n## Built-in Loss Functions\n\n### Loss Functions\n\nLoss functions are in the `forge.losses` module:\n\n```python\nfrom forge.losses import SimpleGRPOLoss, ReinforceLoss\n\n# SimpleGRPOLoss for GRPO training\nloss_fn = SimpleGRPOLoss(beta=0.1)\n\n# Forward pass\nloss = loss_fn(\n    logprobs=logprobs,\n    ref_logprobs=ref_logprobs,\n    advantages=advantages,\n    padding_mask=padding_mask\n)\n```\n\n### ReinforceLoss\n\n```python\nfrom forge.losses.reinforce_loss import ReinforceLoss\n\n# With optional importance ratio clipping\nloss_fn = ReinforceLoss(clip_ratio=0.2)\n```\n\n---\n\n## Common Issues and Solutions\n\n### Issue: Not Enough GPUs\n\n**Symptoms**: \"Insufficient GPU resources\" error\n\n**Solutions**:\n```yaml\n# Reduce service requirements\nservices:\n  generator:\n    procs: 1\n    with_gpus: true\n  trainer:\n    procs: 1\n    with_gpus: true\n  # Remove ref_model (uses generator weights)\n```\n\nOr use CPU for reference model:\n```yaml\nref_model:\n  with_gpus: false\n```\n\n### Issue: OOM During Generation\n\n**Symptoms**: CUDA OOM in vLLM\n\n**Solutions**:\n```yaml\n# Reduce batch size\ngrpo:\n  n_samples: 4  # Reduce from 8\n\n# Or reduce sequence length\ntraining:\n  seq_len: 2048\n```\n\n### Issue: Slow Weight Sync\n\n**Symptoms**: Long pauses between training and generation\n\n**Solutions**:\n```bash\n# Enable RDMA (if available)\nexport TORCHSTORE_USE_RDMA=1\n\n# Or reduce sync frequency\ntraining:\n  sync_interval: 10  # Sync every 10 steps\n```\n\n### Issue: Policy Collapse\n\n**Symptoms**: Entropy drops to zero, reward stops improving\n\n**Solutions**:\n```yaml\n# Increase KL penalty\ngrpo:\n  beta: 0.2  # Increase from 0.1\n\n# Or add entropy bonus\ntraining:\n  entropy_coef: 0.01\n```\n\n---\n\n## Resources\n\n- **Documentation**: https:\u002F\u002Fmeta-pytorch.org\u002Ftorchforge\n- **GitHub**: https:\u002F\u002Fgithub.com\u002Fmeta-pytorch\u002Ftorchforge\n- **Discord**: https:\u002F\u002Fdiscord.gg\u002FYsTYBh6PD9\n- **TorchTitan**: https:\u002F\u002Fgithub.com\u002Fpytorch\u002Ftorchtitan\n- **Monarch**: https:\u002F\u002Fgithub.com\u002Fmeta-pytorch\u002Fmonarch\n\n## Agent operating procedure\n\n1. **Check the environment.** Check GPU type, memory and driver\u002FCUDA versions (`nvidia-smi`), framework versions, and dataset location and size.\n2. **Pin down the inputs.** Confirm formats, identifiers and parameters from the data or the user. Ask rather than guess any value that changes the result.\n3. **Run a small version first.** Do a smoke run: tiny model or subset, few steps, and confirm loss decreases and checkpoints save.\n4. **Execute the full task** using the instructions and references above.\n5. **Validate the result.** Track metrics on held-out data, compare against a baseline, and record seeds, configs and hardware.\n6. **Report.** State what was run (versions, commands, parameters), what was checked, and what is still uncertain.\n\n| If this happens | Do this |\n|---|---|\n| CUDA out-of-memory | Reduce batch size, enable gradient accumulation\u002Fcheckpointing or mixed precision, or shard the model. |\n| Loss is NaN or diverges | Lower the learning rate, check data for invalid values, and enable gradient clipping. |\n| A function, flag or endpoint in these instructions is missing in the installed version | Check the installed version's own documentation (`help()`, `--help`, official docs), adapt, and tell the user. Never invent an API. |\n| A required input, identifier or parameter is ambiguous | Ask the user, or state the assumption explicitly before running. |\n\n**Integrity rules**\n\n- Never fabricate results, parameters, identifiers, citations or statistics. If something cannot be run or verified, say so plainly.\n- Never claim training results without logs; estimate compute cost before launching large jobs.\n- Treat version-specific details here as possibly outdated: confirm them against the official documentation for the installed version.\n- Ask before actions that cost money, consume shared GPUs or cloud quota, touch personal or patient data, or cannot be undone.\n\n## Related skills\n\n- `distributed-llm-pretraining-torchtitan`: Provides PyTorch-native distributed LLM pretraining using torchtitan with 4D parallelism (FSDP2, TP, PP, CP).\n- `pytorch-lightning-distributed`: High-level PyTorch framework with Trainer class, automatic distributed training (DDP\u002FFSDP\u002FDeepSpeed), callbacks system, and minimal boilerp…\n","# torchforge: PyTorch-Native Agentic RL Library\n\ntorchforge is Meta's PyTorch-native RL library that separates infrastructure concerns from algorithm concerns. It enables rapid RL research by letting you focus on algorithms while handling distributed training, inference, and weight sync automatically.\n\n## When to Use torchforge\n\n**Choose torchforge when you need:**\n- Clean separation between RL algorithms and infrastructure\n- PyTorch-native abstractions (no Ray dependency)\n- Easy algorithm experimentation (GRPO, DAPO, SAPO in ~100 lines)\n- Scalable training with Monarch actor system\n- Integration with TorchTitan for model parallelism\n\n**Consider alternatives when:**\n- You need production-ready stability → use **miles** or **verl**\n- You want Megatron-native training → use **slime**\n- torchforge is experimental and APIs may change\n\n## Key Features\n\n- **Algorithm isolation**: Implement RL algorithms without touching infrastructure\n- **Scalability**: From single GPU to thousands via Monarch\n- **Modern stack**: TorchTitan (training), vLLM (inference), TorchStore (sync)\n- **Loss functions**: GRPO, DAPO, CISPO, GSPO, SAPO built-in\n\n## Architecture Overview\n\n```\n┌─────────────────────────────────────────────────────────┐\n│ Application Layer (Your Code)                           │\n│ - Define reward models, loss functions, sampling        │\n└─────────────────────┬───────────────────────────────────┘\n                      │\n┌─────────────────────▼───────────────────────────────────┐\n│ Forge API Layer                                         │\n│ - Episode, Group dataclasses                           │\n│ - Service interfaces (async\u002Fawait)                      │\n└─────────────────────┬───────────────────────────────────┘\n                      │\n┌─────────────────────▼───────────────────────────────────┐\n│ Distributed Services (Monarch)                          │\n│ ├── Trainer (TorchTitan FSDP)                          │\n│ ├── Generator (vLLM inference)                          │\n│ ├── Reference Model (frozen KL baseline)               │\n│ └── Reward Actors (compute rewards)                    │\n└─────────────────────────────────────────────────────────┘\n```\n\n## Installation\n\n```bash\n# Create environment\nconda create -n forge python=3.12\nconda activate forge\n\n# Install (handles PyTorch nightly + dependencies)\n.\u002Fscripts\u002Finstall.sh\n\n# Verify\npython -c \"import torch, forge, vllm; print('OK')\"\n```\n\n### ROCm Installation\n\n```bash\n.\u002Fscripts\u002Finstall_rocm.sh\n```\n\n## Quick Start\n\n### SFT Training (2+ GPUs)\n\n```bash\npython -m apps.sft.main --config apps\u002Fsft\u002Fllama3_8b.yaml\n```\n\n### GRPO Training (3+ GPUs)\n\n```bash\npython -m apps.grpo.main --config apps\u002Fgrpo\u002Fqwen3_1_7b.yaml\n```\n\n---\n\n## Workflow 1: GRPO Training for Math Reasoning\n\nUse this workflow for training reasoning models with group-relative advantages.\n\n### Prerequisites Checklist\n- [ ] 3+ GPUs (GPU0: trainer, GPU1: ref_model, GPU2: generator)\n- [ ] Model from HuggingFace Hub\n- [ ] Training dataset (GSM8K, MATH, etc.)\n\n### Step 1: Create Configuration\n\n```yaml\n# config\u002Fgrpo_math.yaml\nmodel: \"Qwen\u002FQwen2.5-7B-Instruct\"\n\ndataset:\n  path: \"openai\u002Fgsm8k\"\n  split: \"train\"\n  streaming: true\n\ntraining:\n  batch_size: 4\n  learning_rate: 1e-6\n  seq_len: 4096\n  dtype: bfloat16\n  gradient_accumulation_steps: 4\n\ngrpo:\n  n_samples: 8           # Responses per prompt\n  clip_low: 0.2\n  clip_high: 0.28\n  beta: 0.1              # KL penalty coefficient\n  temperature: 0.7\n\nservices:\n  generator:\n    procs: 1\n    num_replicas: 1\n    with_gpus: true\n  trainer:\n    procs: 1\n    num_replicas: 1\n    with_gpus: true\n  ref_model:\n    procs: 1\n    num_replicas: 1\n    with_gpus: true\n```\n\n### Step 2: Define Reward Function\n\n```python\n# rewards.py\n# Reward functions are in forge.data.rewards\nfrom forge.data.rewards import MathReward, ThinkingReward\nimport re\n\n# Or define your own reward function\nclass CustomMathReward:\n    def __call__(self, prompt: str, response: str, target: str) -> float:\n        # Extract answer from response\n        match = re.search(r'\\\\boxed{([^}]+)}', response)\n        if not match:\n            return 0.0\n\n        answer = match.group(1).strip()\n        return 1.0 if answer == target else 0.0\n```\n\n### Step 3: Launch Training\n\n```bash\npython -m apps.grpo.main --config config\u002Fgrpo_math.yaml\n```\n\n### Step 4: Monitor Progress\n- [ ] Check W&B dashboard for loss curves\n- [ ] Verify entropy is decreasing (policy becoming more deterministic)\n- [ ] Monitor KL divergence (should stay bounded)\n\n---\n\n## Workflow 2: Custom Loss Function\n\nUse this workflow to implement new RL algorithms.\n\n### Step 1: Create Loss Class\n\n```python\n# src\u002Fforge\u002Flosses\u002Fcustom_loss.py\nimport torch\nimport torch.nn as nn\n\nclass CustomLoss(nn.Module):\n    def __init__(self, clip_range: float = 0.2, beta: float = 0.1):\n        super().__init__()\n        self.clip_range = clip_range\n        self.beta = beta\n\n    def forward(\n        self,\n        logprobs: torch.Tensor,\n        ref_logprobs: torch.Tensor,\n        advantages: torch.Tensor,\n        padding_mask: torch.Tensor,\n    ) -> torch.Tensor:\n        # Compute importance ratio\n        ratio = torch.exp(logprobs - ref_logprobs)\n\n        # Clipped policy gradient\n        clipped_ratio = torch.clamp(\n            ratio,\n            1 - self.clip_range,\n            1 + self.clip_range\n        )\n        pg_loss = -torch.min(ratio * advantages, clipped_ratio * advantages)\n\n        # KL penalty\n        kl = ref_logprobs - logprobs\n\n        # Apply mask and aggregate\n        masked_loss = (pg_loss + self.beta * kl) * padding_mask\n        loss = masked_loss.sum() \u002F padding_mask.sum()\n\n        return loss\n```\n\n### Step 2: Integrate into Application\n\n```python\n# apps\u002Fcustom\u002Fmain.py\nfrom forge.losses.custom_loss import CustomLoss\n\nloss_fn = CustomLoss(clip_range=0.2, beta=0.1)\n\n# In training loop\nloss = loss_fn(\n    logprobs=logprobs,\n    ref_logprobs=ref_logprobs,\n    advantages=advantages,\n    padding_mask=padding_mask,\n)\n```\n\n---\n\n## Workflow 3: Multi-GPU Distributed Training\n\nUse this workflow for scaling to multiple GPUs or nodes.\n\n### Configuration for Distributed\n\n```yaml\n# config\u002Fdistributed.yaml\nmodel: \"meta-llama\u002FMeta-Llama-3.1-8B-Instruct\"\n\nparallelism:\n  tensor_parallel_degree: 2    # Split model across GPUs\n  pipeline_parallel_degree: 1\n  data_parallel_shard_degree: 2\n\nservices:\n  generator:\n    procs: 2                   # 2 processes for TP=2\n    num_replicas: 1\n    with_gpus: true\n  trainer:\n    procs: 2\n    num_replicas: 1\n    with_gpus: true\n```\n\n### Launch with SLURM\n\n```bash\n# Submit job\nsbatch --nodes=2 --gpus-per-node=8 run_grpo.sh\n```\n\n### Launch Locally (Multi-GPU)\n\n```bash\n# 8 GPU setup\npython -m apps.grpo.main \\\n    --config config\u002Fdistributed.yaml \\\n    --trainer.procs 4 \\\n    --generator.procs 4\n```\n\n---\n\n## Core API Reference\n\n### Training Batch Format\n\ntorchforge uses dictionary-based batches for training:\n\n```python\n# inputs: list of dicts with torch.Tensor values\ninputs = [{\"tokens\": torch.Tensor}]\n\n# targets: list of dicts with training signals\ntargets = [{\n    \"response\": torch.Tensor,\n    \"ref_logprobs\": torch.Tensor,\n    \"advantages\": torch.Tensor,\n    \"padding_mask\": torch.Tensor\n}]\n\n# train_step returns loss as float\nloss = trainer.train_step(inputs, targets)\n```\n\n### Completion\n\nGenerated output from vLLM:\n\n```python\n@dataclass\nclass Completion:\n    text: str              # Generated text\n    token_ids: list[int]   # Token IDs\n    logprobs: list[float]  # Log probabilities\n    metadata: dict         # Custom metadata\n```\n\n---\n\n## Built-in Loss Functions\n\n### Loss Functions\n\nLoss functions are in the `forge.losses` module:\n\n```python\nfrom forge.losses import SimpleGRPOLoss, ReinforceLoss\n\n# SimpleGRPOLoss for GRPO training\nloss_fn = SimpleGRPOLoss(beta=0.1)\n\n# Forward pass\nloss = loss_fn(\n    logprobs=logprobs,\n    ref_logprobs=ref_logprobs,\n    advantages=advantages,\n    padding_mask=padding_mask\n)\n```\n\n### ReinforceLoss\n\n```python\nfrom forge.losses.reinforce_loss import ReinforceLoss\n\n# With optional importance ratio clipping\nloss_fn = ReinforceLoss(clip_ratio=0.2)\n```\n\n---\n\n## Common Issues and Solutions\n\n### Issue: Not Enough GPUs\n\n**Symptoms**: \"Insufficient GPU resources\" error\n\n**Solutions**:\n```yaml\n# Reduce service requirements\nservices:\n  generator:\n    procs: 1\n    with_gpus: true\n  trainer:\n    procs: 1\n    with_gpus: true\n  # Remove ref_model (uses generator weights)\n```\n\nOr use CPU for reference model:\n```yaml\nref_model:\n  with_gpus: false\n```\n\n### Issue: OOM During Generation\n\n**Symptoms**: CUDA OOM in vLLM\n\n**Solutions**:\n```yaml\n# Reduce batch size\ngrpo:\n  n_samples: 4  # Reduce from 8\n\n# Or reduce sequence length\ntraining:\n  seq_len: 2048\n```\n\n### Issue: Slow Weight Sync\n\n**Symptoms**: Long pauses between training and generation\n\n**Solutions**:\n```bash\n# Enable RDMA (if available)\nexport TORCHSTORE_USE_RDMA=1\n\n# Or reduce sync frequency\ntraining:\n  sync_interval: 10  # Sync every 10 steps\n```\n\n### Issue: Policy Collapse\n\n**Symptoms**: Entropy drops to zero, reward stops improving\n\n**Solutions**:\n```yaml\n# Increase KL penalty\ngrpo:\n  beta: 0.2  # Increase from 0.1\n\n# Or add entropy bonus\ntraining:\n  entropy_coef: 0.01\n```\n\n---\n\n## Resources\n\n- **Documentation**: https:\u002F\u002Fmeta-pytorch.org\u002Ftorchforge\n- **GitHub**: https:\u002F\u002Fgithub.com\u002Fmeta-pytorch\u002Ftorchforge\n- **Discord**: https:\u002F\u002Fdiscord.gg\u002FYsTYBh6PD9\n- **TorchTitan**: https:\u002F\u002Fgithub.com\u002Fpytorch\u002Ftorchtitan\n- **Monarch**: https:\u002F\u002Fgithub.com\u002Fmeta-pytorch\u002Fmonarch\n\n## Agent operating procedure\n\n1. **Check the environment.** Check GPU type, memory and driver\u002FCUDA versions (`nvidia-smi`), framework versions, and dataset location and size.\n2. **Pin down the inputs.** Confirm formats, identifiers and parameters from the data or the user. Ask rather than guess any value that changes the result.\n3. **Run a small version first.** Do a smoke run: tiny model or subset, few steps, and confirm loss decreases and checkpoints save.\n4. **Execute the full task** using the instructions and references above.\n5. **Validate the result.** Track metrics on held-out data, compare against a baseline, and record seeds, configs and hardware.\n6. **Report.** State what was run (versions, commands, parameters), what was checked, and what is still uncertain.\n\n| If this happens | Do this |\n|---|---|\n| CUDA out-of-memory | Reduce batch size, enable gradient accumulation\u002Fcheckpointing or mixed precision, or shard the model. |\n| Loss is NaN or diverges | Lower the learning rate, check data for invalid values, and enable gradient clipping. |\n| A function, flag or endpoint in these instructions is missing in the installed version | Check the installed version's own documentation (`help()`, `--help`, official docs), adapt, and tell the user. Never invent an API. |\n| A required input, identifier or parameter is ambiguous | Ask the user, or state the assumption explicitly before running. |\n\n**Integrity rules**\n\n- Never fabricate results, parameters, identifiers, citations or statistics. If something cannot be run or verified, say so plainly.\n- Never claim training results without logs; estimate compute cost before launching large jobs.\n- Treat version-specific details here as possibly outdated: confirm them against the official documentation for the installed version.\n- Ask before actions that cost money, consume shared GPUs or cloud quota, touch personal or patient data, or cannot be undone.\n\n## Related skills\n\n- `distributed-llm-pretraining-torchtitan`: Provides PyTorch-native distributed LLM pretraining using torchtitan with 4D parallelism (FSDP2, TP, PP, CP).\n- `pytorch-lightning-distributed`: High-level PyTorch framework with Trainer class, automatic distributed training (DDP\u002FFSDP\u002FDeepSpeed), callbacks system, and minimal boilerp…\n",{"name":5,"license":15,"metadata":45,"description":7},{"tags":46,"version":47,"category":48,"maintainer":49,"dependencies":50},"Reinforcement Learning, PyTorch, GRPO, SFT, Monarch, TorchTitan, Meta","1.0.0","ml-training","Kalaris Labs","torch>=2.9.0, torchtitan>=0.2.0, vllm, monarch","main","2b63f856ce19c2a797929cacabd9e0fc936812fa","plugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Ftorchforge-rl-training",474,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins","https:\u002F\u002Fhol.org\u002Fplugins\u002Fbest-codex-plugins","https:\u002F\u002Fgithub.com\u002Fhashgraph-online","https:\u002F\u002Fhol.org","Open-source standards and developer tools for interoperable AI agents: discovery, identity, communication, payments, privacy and security.","2026-11-11 14:40:14",{"source":8,"skillName":5,"commands":62,"prompt":72,"zipAvailable":73},[63,66,69],{"tool":64,"command":65},"npx","npx skills add hashgraph-online\u002Fawesome-codex-plugins --skill torchforge-rl-training",{"tool":67,"command":68},"bunx","bunx skills add hashgraph-online\u002Fawesome-codex-plugins --skill torchforge-rl-training",{"tool":70,"command":71},"pnpm","pnpm dlx skills add hashgraph-online\u002Fawesome-codex-plugins --skill torchforge-rl-training","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Ftorchforge-rl-training\u002FSKILL.md",true,[75,81],{"path":76,"resourceType":77,"resourceTypeDesc":78,"sizeBytes":79,"sourceHtmlUrl":80},"references\u002Fapi-reference.md","REFERENCE","参考文档",8685,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Ftorchforge-rl-training\u002Freferences\u002Fapi-reference.md",{"path":82,"resourceType":77,"resourceTypeDesc":78,"sizeBytes":83,"sourceHtmlUrl":84},"references\u002Ftroubleshooting.md",6709,"https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002FKalarisLabs\u002Fresearch-agent-skills\u002Fskills\u002Ftorchforge-rl-training\u002Freferences\u002Ftroubleshooting.md",[86,95,102,109,134,159],{"skillNo":87,"name":88,"nameZh":6,"description":89,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":90,"licenseFileUrl":6,"sourceHtmlUrl":91,"categories":92,"keywords":93,"publishedAt":94},"01M746ZWACVKMZ1BJXJZHYJSSX","migrate-to-e2e","Moves an existing Playwright, Cypress, Selenium, Appium, Detox, Maestro, or TesterArmy suite to e2e one slice at a time, with the old suite running beside the new one until each ported test is proven. Keeps a MIGRATION.md progress file so any later session resumes where the last stopped. Optionally plants a defect to show both suites catch it. Use for \u002Fmigrate-to-e2e, \"migrate our tests to e2e\", \"port these Maestro flows\", \"move our TesterArmy tests\", or when a project has an e2e config and an older suite still running.","Apache-2.0","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Ftester-army\u002Fe2e\u002Fskills\u002Fmigrate-to-e2e\u002FSKILL.md",[],[],"2026-11-11 15:11:14",{"skillNo":96,"name":97,"nameZh":6,"description":98,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":90,"licenseFileUrl":6,"sourceHtmlUrl":99,"categories":100,"keywords":101,"publishedAt":94},"01M746ZPTAX1Q0FBKAB1CYND7E","e2e","Writes, runs, and debugs end-to-end tests for web and mobile apps with e2e, the agentic test runner, which mixes agent.act goals with exact locator assertions on a Playwright browser or an iOS or Android device. Also drives a project's running app live over the e2e MCP server to look at a screen, check a UI change, take a screenshot or record a video, and try a locator; and runs bug bashes, parallel e2e explore runs with each bug proven by a repro test. Use when a project depends on e2e or has an e2e.config.ts, when asked for end-to-end, browser, mobile, or agentic UI tests, to check a change in the running app, to QA or bug bash an app or branch, or when an e2e run fails.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Ftester-army\u002Fe2e\u002Fskills\u002Fe2e\u002FSKILL.md",[],[],{"skillNo":103,"name":104,"nameZh":6,"description":105,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":90,"licenseFileUrl":6,"sourceHtmlUrl":106,"categories":107,"keywords":108,"publishedAt":94},"01M746ZJN784HVHMDP9MQGCSGG","create-verification-skill","Generate a project-local verify-\u003Capp> skill on top of e2e, so any coding agent can launch the app, check it is worth driving, exercise a feature the way a user does, keep evidence, and bug bash it. Writes the skill, a feature map with one explore charter per feature, and proves it once. Use for \u002Fcreate-verification-skill, \"make a verification skill for this repo\", or when a project has no scripted way to prove UI behavior.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Ftester-army\u002Fe2e\u002Fskills\u002Fcreate-verification-skill\u002FSKILL.md",[],[],{"skillNo":110,"name":111,"nameZh":6,"description":112,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":90,"licenseFileUrl":6,"sourceHtmlUrl":113,"categories":114,"keywords":115,"publishedAt":40},"01M716A6ETNB9B1SQ3GXZC75VR","deja-search","deja-vu memory — search the user's past AI coding sessions with the deja CLI. Holds what they already settled and never wrote into the repo (naming rules, banned or already-tried tools, required checks before merge, preferred values, who owns what). Use before your first edit in a task, before you call a change done or ready to merge, before you recommend a tool, library, design or reviewer, before re-debugging an error that may already be solved, when they say 'didn't we fix this before', 'what did we decide about X' or 'I already have that', and before telling them something on this machine does not exist.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fvshulcz\u002Fdeja-vu\u002Fskills\u002Fdeja-search\u002FSKILL.md",[],[116,117,118,119,120,121,122,123,124,125,126,24,26,127,128,129,130,131,132,133],"search","deja search","never","error","ready","memory","re","decide","fix","design","debugging","done","reviewer","io","change","cli","merge","past",{"skillNo":135,"name":136,"nameZh":6,"description":137,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":90,"licenseFileUrl":6,"sourceHtmlUrl":138,"categories":139,"keywords":140,"publishedAt":40},"01M716A3J2W5SENSE9VGMXB375","deja-reflect","End-of-session reflection and memory curation with deja-vu. Review what was decided, what commands broke and what repaired them, and record durable project rules or decisions with deja remember so the next session starts with them.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Fvshulcz\u002Fdeja-vu\u002Fskills\u002Fdeja-reflect\u002FSKILL.md",[],[141,142,143,129,144,145,146,147,148,149,150,151,152,153,121,154,155,156,157,158],"deja reflect","reflect","review","with","record","github","next","https","node","Rules","decisions","brew","remember","project rules","project","session","end","reflection",{"skillNo":160,"name":161,"nameZh":6,"description":162,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":90,"licenseFileUrl":6,"sourceHtmlUrl":163,"categories":164,"keywords":176,"publishedAt":40},"01M715T3RR4BAYSR57HPPTJ0PF","revid","Make, edit, export and post AI videos with Revid (revid.ai) through the Revid MCP server. Use whenever the user wants a short-form or long-form video made (from an idea, a script, a URL, an article, a PDF, a tweet, a product page, their own clips, or a song), wants to find what is going viral on TikTok, YouTube, Instagram or X before making something, wants to edit or re-export an existing Revid project, generate a standalone AI image or clip, make a series episode, or publish and schedule a finished video to connected social accounts. Covers setup (hosted MCP with OAuth, or an API key), the 36 tools, credits, polling and the rules for publishing. Routes to the more specific revid-* skills when one fits.","https:\u002F\u002Fgithub.com\u002Fhashgraph-online\u002Fawesome-codex-plugins\u002Fblob\u002Fmain\u002Fplugins\u002Ftmakersas\u002Frevid-skills\u002Fskills\u002Frevid\u002FSKILL.md",[165,170,173],{"categoryNo":166,"categoryName":167,"taxonomyType":168,"taxonomyTypeDesc":169},"01JBSEEDCAT000000000000016","图像与多媒体","SCENARIO","使用场景",{"categoryNo":171,"categoryName":172,"taxonomyType":168,"taxonomyTypeDesc":169},"01JBSEEDCAT000000000000023","Skill与智能体开发",{"categoryNo":174,"categoryName":175,"taxonomyType":168,"taxonomyTypeDesc":169},"01JBSEEDCAT000000000000010","后端与接口",[161,177,178,179,180,181,182,155,183,122,184,185,186,187,24,188,189,190,191,192],"URL","Video","short-form","standalone","PDF","viral","tiktok","short","social","MCP","song","image","mcp server","schedule","api","publish"]