[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7EPPZDPC1SCNYE4DQAZAPZS":3,"skill-same:01M7EPPZDPC1SCNYE4DQAZAPZS":71},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":39,"commitSha":40,"skillDirectory":41,"forksCount":42,"repositoryHtmlUrl":43,"repositoryHomepageUrl":6,"ownerHtmlUrl":44,"ownerBlogUrl":45,"ownerDescription":6,"updatedAt":46,"install":47,"resources":60},"01M7EPPZDPC1SCNYE4DQAZAPZS","post-training-torchforge",null,"Provides guidance for PyTorch-native agentic RL using torchforge, Meta's library separating infra from algorithms. Use when you want clean RL abstractions, easy algorithm experimentation, or scalable training with Monarch and TorchTitan.","TanNhatCMS\u002Fjarvis","TanNhatCMS","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F113796420?v=4","01M753MV17KG5CD3152G3WQEQY","KietNT","01M7EKEDYVYQ6PWVJJMZ911VMX",0,"MIT","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fams-imp-post-training-torchforge\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: post-training-torchforge\ndescription: Provides guidance for PyTorch-native agentic RL using torchforge, Meta's library separating infra from algorithms. Use when you want clean RL abstractions, easy algorithm experimentation, or scalable training with Monarch and TorchTitan.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Reinforcement Learning, PyTorch, GRPO, SFT, Monarch, TorchTitan, Meta]\ndependencies: [torch>=2.9.0, torchtitan>=0.2.0, vllm, monarch]\n---\n\n# torchforge: PyTorch-Native Agentic RL Library\n\ntorchforge is Meta's PyTorch-native RL library that separates infrastructure concerns from algorithm concerns. It enables rapid RL research by letting you focus on algorithms while handling distributed training, inference, and weight sync automatically.\n\n## When to Use torchforge\n\n**Choose torchforge when you need:**\n- Clean separation between RL algorithms and infrastructure\n- PyTorch-native abstractions (no Ray dependency)\n- Easy algorithm experimentation (GRPO, DAPO, SAPO in ~100 lines)\n- Scalable training with Monarch actor system\n- Integration with TorchTitan for model parallelism\n\n**Consider alternatives when:**\n- You need production-ready stability → use **miles** or **verl**\n- You want Megatron-native training → use **slime**\n- torchforge is experimental and APIs may change\n\n## Key Features\n\n- **Algorithm isolation**: Implement RL algorithms without touching infrastructure\n- **Scalability**: From single GPU to thousands via Monarch\n- **Modern stack**: TorchTitan (training), vLLM (inference), TorchStore (sync)\n- **Loss functions**: GRPO, DAPO, CISPO, GSPO, SAPO built-in\n\n## Architecture Overview\n\n```\n┌─────────────────────────────────────────────────────────┐\n│ Application Layer (Your Code)                           │\n│ - Define reward models, loss functions, sampling        │\n└─────────────────────┬───────────────────────────────────┘\n                      │\n┌─────────────────────▼───────────────────────────────────┐\n│ Forge API Layer                                         │\n│ - Episode, Group dataclasses                           │\n│ - Service interfaces (async\u002Fawait)                      │\n└─────────────────────┬───────────────────────────────────┘\n                      │\n┌─────────────────────▼───────────────────────────────────┐\n│ Distributed Services (Monarch)                          │\n│ ├── Trainer (TorchTitan FSDP)                          │\n│ ├── Generator (vLLM inference)                          │\n│ ├── Reference Model (frozen KL baseline)               │\n│ └── Reward Actors (compute rewards)                    │\n└─────────────────────────────────────────────────────────┘\n```\n\n## Installation\n\n```bash\n# Create environment\nconda create -n forge python=3.12\nconda activate forge\n\n# Install (handles PyTorch nightly + dependencies)\n.\u002Fscripts\u002Finstall.sh\n\n# Verify\npython -c \"import torch, forge, vllm; print('OK')\"\n```\n\n### ROCm Installation\n\n```bash\n.\u002Fscripts\u002Finstall_rocm.sh\n```\n\n## Quick Start\n\n### SFT Training (2+ GPUs)\n\n```bash\npython -m apps.sft.main --config apps\u002Fsft\u002Fllama3_8b.yaml\n```\n\n### GRPO Training (3+ GPUs)\n\n```bash\npython -m apps.grpo.main --config apps\u002Fgrpo\u002Fqwen3_1_7b.yaml\n```\n\n---\n\n## Workflow 1: GRPO Training for Math Reasoning\n\nUse this workflow for training reasoning models with group-relative advantages.\n\n### Prerequisites Checklist\n- [ ] 3+ GPUs (GPU0: trainer, GPU1: ref_model, GPU2: generator)\n- [ ] Model from HuggingFace Hub\n- [ ] Training dataset (GSM8K, MATH, etc.)\n\n### Step 1: Create Configuration\n\n```yaml\n# config\u002Fgrpo_math.yaml\nmodel: \"Qwen\u002FQwen2.5-7B-Instruct\"\n\ndataset:\n  path: \"openai\u002Fgsm8k\"\n  split: \"train\"\n  streaming: true\n\ntraining:\n  batch_size: 4\n  learning_rate: 1e-6\n  seq_len: 4096\n  dtype: bfloat16\n  gradient_accumulation_steps: 4\n\ngrpo:\n  n_samples: 8           # Responses per prompt\n  clip_low: 0.2\n  clip_high: 0.28\n  beta: 0.1              # KL penalty coefficient\n  temperature: 0.7\n\nservices:\n  generator:\n    procs: 1\n    num_replicas: 1\n    with_gpus: true\n  trainer:\n    procs: 1\n    num_replicas: 1\n    with_gpus: true\n  ref_model:\n    procs: 1\n    num_replicas: 1\n    with_gpus: true\n```\n\n### Step 2: Define Reward Function\n\n```python\n# rewards.py\n# Reward functions are in forge.data.rewards\nfrom forge.data.rewards import MathReward, ThinkingReward\nimport re\n\n# Or define your own reward function\nclass CustomMathReward:\n    def __call__(self, prompt: str, response: str, target: str) -> float:\n        # Extract answer from response\n        match = re.search(r'\\\\boxed{([^}]+)}', response)\n        if not match:\n            return 0.0\n\n        answer = match.group(1).strip()\n        return 1.0 if answer == target else 0.0\n```\n\n### Step 3: Launch Training\n\n```bash\npython -m apps.grpo.main --config config\u002Fgrpo_math.yaml\n```\n\n### Step 4: Monitor Progress\n- [ ] Check W&B dashboard for loss curves\n- [ ] Verify entropy is decreasing (policy becoming more deterministic)\n- [ ] Monitor KL divergence (should stay bounded)\n\n---\n\n## Workflow 2: Custom Loss Function\n\nUse this workflow to implement new RL algorithms.\n\n### Step 1: Create Loss Class\n\n```python\n# src\u002Fforge\u002Flosses\u002Fcustom_loss.py\nimport torch\nimport torch.nn as nn\n\nclass CustomLoss(nn.Module):\n    def __init__(self, clip_range: float = 0.2, beta: float = 0.1):\n        super().__init__()\n        self.clip_range = clip_range\n        self.beta = beta\n\n    def forward(\n        self,\n        logprobs: torch.Tensor,\n        ref_logprobs: torch.Tensor,\n        advantages: torch.Tensor,\n        padding_mask: torch.Tensor,\n    ) -> torch.Tensor:\n        # Compute importance ratio\n        ratio = torch.exp(logprobs - ref_logprobs)\n\n        # Clipped policy gradient\n        clipped_ratio = torch.clamp(\n            ratio,\n            1 - self.clip_range,\n            1 + self.clip_range\n        )\n        pg_loss = -torch.min(ratio * advantages, clipped_ratio * advantages)\n\n        # KL penalty\n        kl = ref_logprobs - logprobs\n\n        # Apply mask and aggregate\n        masked_loss = (pg_loss + self.beta * kl) * padding_mask\n        loss = masked_loss.sum() \u002F padding_mask.sum()\n\n        return loss\n```\n\n### Step 2: Integrate into Application\n\n```python\n# apps\u002Fcustom\u002Fmain.py\nfrom forge.losses.custom_loss import CustomLoss\n\nloss_fn = CustomLoss(clip_range=0.2, beta=0.1)\n\n# In training loop\nloss = loss_fn(\n    logprobs=logprobs,\n    ref_logprobs=ref_logprobs,\n    advantages=advantages,\n    padding_mask=padding_mask,\n)\n```\n\n---\n\n## Workflow 3: Multi-GPU Distributed Training\n\nUse this workflow for scaling to multiple GPUs or nodes.\n\n### Configuration for Distributed\n\n```yaml\n# config\u002Fdistributed.yaml\nmodel: \"meta-llama\u002FMeta-Llama-3.1-8B-Instruct\"\n\nparallelism:\n  tensor_parallel_degree: 2    # Split model across GPUs\n  pipeline_parallel_degree: 1\n  data_parallel_shard_degree: 2\n\nservices:\n  generator:\n    procs: 2                   # 2 processes for TP=2\n    num_replicas: 1\n    with_gpus: true\n  trainer:\n    procs: 2\n    num_replicas: 1\n    with_gpus: true\n```\n\n### Launch with SLURM\n\n```bash\n# Submit job\nsbatch --nodes=2 --gpus-per-node=8 run_grpo.sh\n```\n\n### Launch Locally (Multi-GPU)\n\n```bash\n# 8 GPU setup\npython -m apps.grpo.main \\\n    --config config\u002Fdistributed.yaml \\\n    --trainer.procs 4 \\\n    --generator.procs 4\n```\n\n---\n\n## Core API Reference\n\n### Training Batch Format\n\ntorchforge uses dictionary-based batches for training:\n\n```python\n# inputs: list of dicts with torch.Tensor values\ninputs = [{\"tokens\": torch.Tensor}]\n\n# targets: list of dicts with training signals\ntargets = [{\n    \"response\": torch.Tensor,\n    \"ref_logprobs\": torch.Tensor,\n    \"advantages\": torch.Tensor,\n    \"padding_mask\": torch.Tensor\n}]\n\n# train_step returns loss as float\nloss = trainer.train_step(inputs, targets)\n```\n\n### Completion\n\nGenerated output from vLLM:\n\n```python\n@dataclass\nclass Completion:\n    text: str              # Generated text\n    token_ids: list[int]   # Token IDs\n    logprobs: list[float]  # Log probabilities\n    metadata: dict         # Custom metadata\n```\n\n---\n\n## Built-in Loss Functions\n\n### Loss Functions\n\nLoss functions are in the `forge.losses` module:\n\n```python\nfrom forge.losses import SimpleGRPOLoss, ReinforceLoss\n\n# SimpleGRPOLoss for GRPO training\nloss_fn = SimpleGRPOLoss(beta=0.1)\n\n# Forward pass\nloss = loss_fn(\n    logprobs=logprobs,\n    ref_logprobs=ref_logprobs,\n    advantages=advantages,\n    padding_mask=padding_mask\n)\n```\n\n### ReinforceLoss\n\n```python\nfrom forge.losses.reinforce_loss import ReinforceLoss\n\n# With optional importance ratio clipping\nloss_fn = ReinforceLoss(clip_ratio=0.2)\n```\n\n---\n\n## Common Issues and Solutions\n\n### Issue: Not Enough GPUs\n\n**Symptoms**: \"Insufficient GPU resources\" error\n\n**Solutions**:\n```yaml\n# Reduce service requirements\nservices:\n  generator:\n    procs: 1\n    with_gpus: true\n  trainer:\n    procs: 1\n    with_gpus: true\n  # Remove ref_model (uses generator weights)\n```\n\nOr use CPU for reference model:\n```yaml\nref_model:\n  with_gpus: false\n```\n\n### Issue: OOM During Generation\n\n**Symptoms**: CUDA OOM in vLLM\n\n**Solutions**:\n```yaml\n# Reduce batch size\ngrpo:\n  n_samples: 4  # Reduce from 8\n\n# Or reduce sequence length\ntraining:\n  seq_len: 2048\n```\n\n### Issue: Slow Weight Sync\n\n**Symptoms**: Long pauses between training and generation\n\n**Solutions**:\n```bash\n# Enable RDMA (if available)\nexport TORCHSTORE_USE_RDMA=1\n\n# Or reduce sync frequency\ntraining:\n  sync_interval: 10  # Sync every 10 steps\n```\n\n### Issue: Policy Collapse\n\n**Symptoms**: Entropy drops to zero, reward stops improving\n\n**Solutions**:\n```yaml\n# Increase KL penalty\ngrpo:\n  beta: 0.2  # Increase from 0.1\n\n# Or add entropy bonus\ntraining:\n  entropy_coef: 0.01\n```\n\n---\n\n## Resources\n\n- **Documentation**: https:\u002F\u002Fmeta-pytorch.org\u002Ftorchforge\n- **GitHub**: https:\u002F\u002Fgithub.com\u002Fmeta-pytorch\u002Ftorchforge\n- **Discord**: https:\u002F\u002Fdiscord.gg\u002FYsTYBh6PD9\n- **TorchTitan**: https:\u002F\u002Fgithub.com\u002Fpytorch\u002Ftorchtitan\n- **Monarch**: https:\u002F\u002Fgithub.com\u002Fmeta-pytorch\u002Fmonarch\n\n","# torchforge: PyTorch-Native Agentic RL Library\n\ntorchforge is Meta's PyTorch-native RL library that separates infrastructure concerns from algorithm concerns. It enables rapid RL research by letting you focus on algorithms while handling distributed training, inference, and weight sync automatically.\n\n## When to Use torchforge\n\n**Choose torchforge when you need:**\n- Clean separation between RL algorithms and infrastructure\n- PyTorch-native abstractions (no Ray dependency)\n- Easy algorithm experimentation (GRPO, DAPO, SAPO in ~100 lines)\n- Scalable training with Monarch actor system\n- Integration with TorchTitan for model parallelism\n\n**Consider alternatives when:**\n- You need production-ready stability → use **miles** or **verl**\n- You want Megatron-native training → use **slime**\n- torchforge is experimental and APIs may change\n\n## Key Features\n\n- **Algorithm isolation**: Implement RL algorithms without touching infrastructure\n- **Scalability**: From single GPU to thousands via Monarch\n- **Modern stack**: TorchTitan (training), vLLM (inference), TorchStore (sync)\n- **Loss functions**: GRPO, DAPO, CISPO, GSPO, SAPO built-in\n\n## Architecture Overview\n\n```\n┌─────────────────────────────────────────────────────────┐\n│ Application Layer (Your Code)                           │\n│ - Define reward models, loss functions, sampling        │\n└─────────────────────┬───────────────────────────────────┘\n                      │\n┌─────────────────────▼───────────────────────────────────┐\n│ Forge API Layer                                         │\n│ - Episode, Group dataclasses                           │\n│ - Service interfaces (async\u002Fawait)                      │\n└─────────────────────┬───────────────────────────────────┘\n                      │\n┌─────────────────────▼───────────────────────────────────┐\n│ Distributed Services (Monarch)                          │\n│ ├── Trainer (TorchTitan FSDP)                          │\n│ ├── Generator (vLLM inference)                          │\n│ ├── Reference Model (frozen KL baseline)               │\n│ └── Reward Actors (compute rewards)                    │\n└─────────────────────────────────────────────────────────┘\n```\n\n## Installation\n\n```bash\n# Create environment\nconda create -n forge python=3.12\nconda activate forge\n\n# Install (handles PyTorch nightly + dependencies)\n.\u002Fscripts\u002Finstall.sh\n\n# Verify\npython -c \"import torch, forge, vllm; print('OK')\"\n```\n\n### ROCm Installation\n\n```bash\n.\u002Fscripts\u002Finstall_rocm.sh\n```\n\n## Quick Start\n\n### SFT Training (2+ GPUs)\n\n```bash\npython -m apps.sft.main --config apps\u002Fsft\u002Fllama3_8b.yaml\n```\n\n### GRPO Training (3+ GPUs)\n\n```bash\npython -m apps.grpo.main --config apps\u002Fgrpo\u002Fqwen3_1_7b.yaml\n```\n\n---\n\n## Workflow 1: GRPO Training for Math Reasoning\n\nUse this workflow for training reasoning models with group-relative advantages.\n\n### Prerequisites Checklist\n- [ ] 3+ GPUs (GPU0: trainer, GPU1: ref_model, GPU2: generator)\n- [ ] Model from HuggingFace Hub\n- [ ] Training dataset (GSM8K, MATH, etc.)\n\n### Step 1: Create Configuration\n\n```yaml\n# config\u002Fgrpo_math.yaml\nmodel: \"Qwen\u002FQwen2.5-7B-Instruct\"\n\ndataset:\n  path: \"openai\u002Fgsm8k\"\n  split: \"train\"\n  streaming: true\n\ntraining:\n  batch_size: 4\n  learning_rate: 1e-6\n  seq_len: 4096\n  dtype: bfloat16\n  gradient_accumulation_steps: 4\n\ngrpo:\n  n_samples: 8           # Responses per prompt\n  clip_low: 0.2\n  clip_high: 0.28\n  beta: 0.1              # KL penalty coefficient\n  temperature: 0.7\n\nservices:\n  generator:\n    procs: 1\n    num_replicas: 1\n    with_gpus: true\n  trainer:\n    procs: 1\n    num_replicas: 1\n    with_gpus: true\n  ref_model:\n    procs: 1\n    num_replicas: 1\n    with_gpus: true\n```\n\n### Step 2: Define Reward Function\n\n```python\n# rewards.py\n# Reward functions are in forge.data.rewards\nfrom forge.data.rewards import MathReward, ThinkingReward\nimport re\n\n# Or define your own reward function\nclass CustomMathReward:\n    def __call__(self, prompt: str, response: str, target: str) -> float:\n        # Extract answer from response\n        match = re.search(r'\\\\boxed{([^}]+)}', response)\n        if not match:\n            return 0.0\n\n        answer = match.group(1).strip()\n        return 1.0 if answer == target else 0.0\n```\n\n### Step 3: Launch Training\n\n```bash\npython -m apps.grpo.main --config config\u002Fgrpo_math.yaml\n```\n\n### Step 4: Monitor Progress\n- [ ] Check W&B dashboard for loss curves\n- [ ] Verify entropy is decreasing (policy becoming more deterministic)\n- [ ] Monitor KL divergence (should stay bounded)\n\n---\n\n## Workflow 2: Custom Loss Function\n\nUse this workflow to implement new RL algorithms.\n\n### Step 1: Create Loss Class\n\n```python\n# src\u002Fforge\u002Flosses\u002Fcustom_loss.py\nimport torch\nimport torch.nn as nn\n\nclass CustomLoss(nn.Module):\n    def __init__(self, clip_range: float = 0.2, beta: float = 0.1):\n        super().__init__()\n        self.clip_range = clip_range\n        self.beta = beta\n\n    def forward(\n        self,\n        logprobs: torch.Tensor,\n        ref_logprobs: torch.Tensor,\n        advantages: torch.Tensor,\n        padding_mask: torch.Tensor,\n    ) -> torch.Tensor:\n        # Compute importance ratio\n        ratio = torch.exp(logprobs - ref_logprobs)\n\n        # Clipped policy gradient\n        clipped_ratio = torch.clamp(\n            ratio,\n            1 - self.clip_range,\n            1 + self.clip_range\n        )\n        pg_loss = -torch.min(ratio * advantages, clipped_ratio * advantages)\n\n        # KL penalty\n        kl = ref_logprobs - logprobs\n\n        # Apply mask and aggregate\n        masked_loss = (pg_loss + self.beta * kl) * padding_mask\n        loss = masked_loss.sum() \u002F padding_mask.sum()\n\n        return loss\n```\n\n### Step 2: Integrate into Application\n\n```python\n# apps\u002Fcustom\u002Fmain.py\nfrom forge.losses.custom_loss import CustomLoss\n\nloss_fn = CustomLoss(clip_range=0.2, beta=0.1)\n\n# In training loop\nloss = loss_fn(\n    logprobs=logprobs,\n    ref_logprobs=ref_logprobs,\n    advantages=advantages,\n    padding_mask=padding_mask,\n)\n```\n\n---\n\n## Workflow 3: Multi-GPU Distributed Training\n\nUse this workflow for scaling to multiple GPUs or nodes.\n\n### Configuration for Distributed\n\n```yaml\n# config\u002Fdistributed.yaml\nmodel: \"meta-llama\u002FMeta-Llama-3.1-8B-Instruct\"\n\nparallelism:\n  tensor_parallel_degree: 2    # Split model across GPUs\n  pipeline_parallel_degree: 1\n  data_parallel_shard_degree: 2\n\nservices:\n  generator:\n    procs: 2                   # 2 processes for TP=2\n    num_replicas: 1\n    with_gpus: true\n  trainer:\n    procs: 2\n    num_replicas: 1\n    with_gpus: true\n```\n\n### Launch with SLURM\n\n```bash\n# Submit job\nsbatch --nodes=2 --gpus-per-node=8 run_grpo.sh\n```\n\n### Launch Locally (Multi-GPU)\n\n```bash\n# 8 GPU setup\npython -m apps.grpo.main \\\n    --config config\u002Fdistributed.yaml \\\n    --trainer.procs 4 \\\n    --generator.procs 4\n```\n\n---\n\n## Core API Reference\n\n### Training Batch Format\n\ntorchforge uses dictionary-based batches for training:\n\n```python\n# inputs: list of dicts with torch.Tensor values\ninputs = [{\"tokens\": torch.Tensor}]\n\n# targets: list of dicts with training signals\ntargets = [{\n    \"response\": torch.Tensor,\n    \"ref_logprobs\": torch.Tensor,\n    \"advantages\": torch.Tensor,\n    \"padding_mask\": torch.Tensor\n}]\n\n# train_step returns loss as float\nloss = trainer.train_step(inputs, targets)\n```\n\n### Completion\n\nGenerated output from vLLM:\n\n```python\n@dataclass\nclass Completion:\n    text: str              # Generated text\n    token_ids: list[int]   # Token IDs\n    logprobs: list[float]  # Log probabilities\n    metadata: dict         # Custom metadata\n```\n\n---\n\n## Built-in Loss Functions\n\n### Loss Functions\n\nLoss functions are in the `forge.losses` module:\n\n```python\nfrom forge.losses import SimpleGRPOLoss, ReinforceLoss\n\n# SimpleGRPOLoss for GRPO training\nloss_fn = SimpleGRPOLoss(beta=0.1)\n\n# Forward pass\nloss = loss_fn(\n    logprobs=logprobs,\n    ref_logprobs=ref_logprobs,\n    advantages=advantages,\n    padding_mask=padding_mask\n)\n```\n\n### ReinforceLoss\n\n```python\nfrom forge.losses.reinforce_loss import ReinforceLoss\n\n# With optional importance ratio clipping\nloss_fn = ReinforceLoss(clip_ratio=0.2)\n```\n\n---\n\n## Common Issues and Solutions\n\n### Issue: Not Enough GPUs\n\n**Symptoms**: \"Insufficient GPU resources\" error\n\n**Solutions**:\n```yaml\n# Reduce service requirements\nservices:\n  generator:\n    procs: 1\n    with_gpus: true\n  trainer:\n    procs: 1\n    with_gpus: true\n  # Remove ref_model (uses generator weights)\n```\n\nOr use CPU for reference model:\n```yaml\nref_model:\n  with_gpus: false\n```\n\n### Issue: OOM During Generation\n\n**Symptoms**: CUDA OOM in vLLM\n\n**Solutions**:\n```yaml\n# Reduce batch size\ngrpo:\n  n_samples: 4  # Reduce from 8\n\n# Or reduce sequence length\ntraining:\n  seq_len: 2048\n```\n\n### Issue: Slow Weight Sync\n\n**Symptoms**: Long pauses between training and generation\n\n**Solutions**:\n```bash\n# Enable RDMA (if available)\nexport TORCHSTORE_USE_RDMA=1\n\n# Or reduce sync frequency\ntraining:\n  sync_interval: 10  # Sync every 10 steps\n```\n\n### Issue: Policy Collapse\n\n**Symptoms**: Entropy drops to zero, reward stops improving\n\n**Solutions**:\n```yaml\n# Increase KL penalty\ngrpo:\n  beta: 0.2  # Increase from 0.1\n\n# Or add entropy bonus\ntraining:\n  entropy_coef: 0.01\n```\n\n---\n\n## Resources\n\n- **Documentation**: https:\u002F\u002Fmeta-pytorch.org\u002Ftorchforge\n- **GitHub**: https:\u002F\u002Fgithub.com\u002Fmeta-pytorch\u002Ftorchforge\n- **Discord**: https:\u002F\u002Fdiscord.gg\u002FYsTYBh6PD9\n- **TorchTitan**: https:\u002F\u002Fgithub.com\u002Fpytorch\u002Ftorchtitan\n- **Monarch**: https:\u002F\u002Fgithub.com\u002Fmeta-pytorch\u002Fmonarch\n\n",{"name":5,"tags":24,"author":32,"license":15,"version":33,"description":7,"dependencies":34},[25,26,27,28,29,30,31],"Reinforcement Learning","PyTorch","GRPO","SFT","Monarch","TorchTitan","Meta","Orchestra Research","1.0.0",[35,36,37,38],"torch>=2.9.0","torchtitan>=0.2.0","vllm","monarch","main","af388a77c2fe5e0b4fd879cca4300a0ffa9a6579","skills\u002Fams-imp-post-training-torchforge",1,"https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis","https:\u002F\u002Fgithub.com\u002FTanNhatCMS","https:\u002F\u002Fmrkiet.name.vn","2026-11-07 14:23:08",{"source":8,"skillName":5,"commands":48,"prompt":58,"zipAvailable":59},[49,52,55],{"tool":50,"command":51},"npx","npx skills add TanNhatCMS\u002Fjarvis --skill post-training-torchforge",{"tool":53,"command":54},"bunx","bunx skills add TanNhatCMS\u002Fjarvis --skill post-training-torchforge",{"tool":56,"command":57},"pnpm","pnpm dlx skills add TanNhatCMS\u002Fjarvis --skill post-training-torchforge","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fams-imp-post-training-torchforge\u002FSKILL.md",true,[61,67],{"path":62,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":65,"sourceHtmlUrl":66},"references\u002Fapi-reference.md","REFERENCE","参考文档",8685,"https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fams-imp-post-training-torchforge\u002Freferences\u002Fapi-reference.md",{"path":68,"resourceType":63,"resourceTypeDesc":64,"sizeBytes":69,"sourceHtmlUrl":70},"references\u002Ftroubleshooting.md",6709,"https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fams-imp-post-training-torchforge\u002Freferences\u002Ftroubleshooting.md",[72,79,87,94,101,108],{"skillNo":73,"name":74,"nameZh":6,"description":75,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":76,"categories":77,"keywords":78,"publishedAt":19},"01M7EWYJR0J8DARY7E59Z6Y9MV","web-scrape","Turn web pages into LLM-ready markdown via the Firecrawl API (renders JS, anti-bot handling). Use when browser_open\u002Ffetch fails on JS-heavy pages, or when you need clean markdown\u002Fstructured extraction from a site. Requires FIRECRAWL_API_KEY.","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fweb-scrape\u002FSKILL.md",[],[],{"skillNo":80,"name":81,"nameZh":6,"description":82,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":83,"sourceHtmlUrl":84,"categories":85,"keywords":86,"publishedAt":19},"01M7EWYBN5P6F2AJG3FR24T9BP","design-taste-frontend","tasteskill v2, the anti-slop frontend skill. Art direction for websites, distilled from a study of 247 curated gallery sites (One Page Love, Awwwards Sites of the Day). Deals researched directions (idea, composition, type, palette, photographs, section blueprints) from a library, enforces a strict no-slop house style with a real-browser inspector, and runs a look-and-fix loop on screenshots. Use this whenever you build, design, redesign, restyle, polish or review a landing page, marketing site, portfolio, product page, one-pager, hero or any public-facing web page, in any stack (HTML\u002FCSS, React, Next.js, Vue, Svelte, Astro, Tailwind) — also when the user says a page looks generic, templated, \"AI-made\", sloppy, or asks to make it look designed, premium, award-level or less like every other site. Not for dashboards or app screens.","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Ftaste-design\u002FLICENSE","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Ftaste-design\u002FSKILL.md",[],[],{"skillNo":88,"name":89,"nameZh":6,"description":90,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":91,"categories":92,"keywords":93,"publishedAt":19},"01M7EWY8FQFNQ2GB7NZZWGSSDX","test-driven-development","Use when implementing any feature or bugfix, before writing implementation code","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fsuperpowers-tdd\u002FSKILL.md",[],[],{"skillNo":95,"name":96,"nameZh":6,"description":97,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":98,"categories":99,"keywords":100,"publishedAt":19},"01M7EWY5SHBFRH1HDHVWM9K7TV","systematic-debugging","Use when encountering any bug, test failure, or unexpected behavior, before proposing fixes","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fsuperpowers-debugging\u002FSKILL.md",[],[],{"skillNo":102,"name":103,"nameZh":6,"description":104,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":105,"categories":106,"keywords":107,"publishedAt":19},"01M7EWY3NM0Z0JAKS7HDZ24K66","requesting-code-review","Use when completing tasks, implementing major features, or before merging to verify work meets requirements","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fsuperpowers-code-review\u002FSKILL.md",[],[],{"skillNo":109,"name":110,"nameZh":6,"description":111,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":112,"categories":113,"keywords":114,"publishedAt":19},"01M7EWY0WRMWX4M9ZFY0WKC593","brainstorming","You MUST use this before any creative work - creating features, building components, adding functionality, modifying behavior, or planning anything new, in software or out of it (a talk, a business, a renovation).","https:\u002F\u002Fgithub.com\u002FTanNhatCMS\u002Fjarvis\u002Fblob\u002Fmain\u002Fskills\u002Fsuperpowers-brainstorming\u002FSKILL.md",[],[]]