[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M6QNF7160DP7VAXVZYASBYHD":3,"skill-same:01M6QNF7160DP7VAXVZYASBYHD":85},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":39,"linkOnly":40,"markdownContent":41,"markdownBody":42,"frontMatter":43,"refName":56,"commitSha":57,"skillDirectory":58,"forksCount":14,"repositoryHtmlUrl":59,"repositoryHomepageUrl":60,"ownerHtmlUrl":61,"ownerBlogUrl":62,"ownerDescription":63,"updatedAt":64,"install":65,"resources":78},"01M6QNF7160DP7VAXVZYASBYHD","ray-train",null,"Distributed training orchestration across clusters. Scales PyTorch\u002FTensorFlow\u002FHuggingFace from laptop to 1000s of nodes. Built-in hyperparameter tuning with Ray Tune, fault tolerance, elastic scaling. Use when training massive models across multiple machines or running distributed hyperparameter sweeps.","Tashima-Tarsh\u002FDisha6.6","Tashima-Tarsh","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F269906498?v=4","01M6PBH98X03XERH0HFT4PFB1P","NITISH KUMAR","01M6PBH9DA0M6KM3Z12P848F0K",0,"MIT","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fqc-ray-train\u002FSKILL.md",[],[19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38],"Ray","Ray Train","train","orchestra","tensorflow","tolerance","Multi-Node","Elastic Scaling","Hyperparameter Tuning","fault tolerance","scaling","across","node","tuning","elastic","Transformers","Distributed Training","training","pytorch","orchestration","2026-11-10 10:12:30",false,"---\nname: ray-train\ndescription: Distributed training orchestration across clusters. Scales PyTorch\u002FTensorFlow\u002FHuggingFace from laptop to 1000s of nodes. Built-in hyperparameter tuning with Ray Tune, fault tolerance, elastic scaling. Use when training massive models across multiple machines or running distributed hyperparameter sweeps.\nversion: 1.0.0\nauthor: Orchestra Research\nlicense: MIT\ntags: [Ray Train, Distributed Training, Orchestration, Ray, Hyperparameter Tuning, Fault Tolerance, Elastic Scaling, Multi-Node, PyTorch, TensorFlow]\ndependencies: [\"ray[train]\", torch, transformers]\ncategory: \"ai-research\"\n---\n\n# Ray Train - Distributed Training Orchestration\n\n## Quick start\n\nRay Train scales machine learning training from single GPU to multi-node clusters with minimal code changes.\n\n**Installation**:\n```bash\npip install -U \"ray[train]\"\n```\n\n**Basic PyTorch training** (single node):\n\n```python\nimport ray\nfrom ray import train\nfrom ray.train import ScalingConfig\nfrom ray.train.torch import TorchTrainer\nimport torch\nimport torch.nn as nn\n\n# Define training function\ndef train_func(config):\n    # Your normal PyTorch code\n    model = nn.Linear(10, 1)\n    optimizer = torch.optim.SGD(model.parameters(), lr=0.01)\n\n    # Prepare for distributed (Ray handles device placement)\n    model = train.torch.prepare_model(model)\n\n    for epoch in range(10):\n        # Your training loop\n        output = model(torch.randn(32, 10))\n        loss = output.sum()\n        loss.backward()\n        optimizer.step()\n        optimizer.zero_grad()\n\n        # Report metrics (logged automatically)\n        train.report({\"loss\": loss.item(), \"epoch\": epoch})\n\n# Run distributed training\ntrainer = TorchTrainer(\n    train_func,\n    scaling_config=ScalingConfig(\n        num_workers=4,  # 4 GPUs\u002Fworkers\n        use_gpu=True\n    )\n)\n\nresult = trainer.fit()\nprint(f\"Final loss: {result.metrics['loss']}\")\n```\n\n**That's it!** Ray handles:\n- Distributed coordination\n- GPU allocation\n- Fault tolerance\n- Checkpointing\n- Metric aggregation\n\n## Common workflows\n\n### Workflow 1: Scale existing PyTorch code\n\n**Original single-GPU code**:\n```python\nmodel = MyModel().cuda()\noptimizer = torch.optim.Adam(model.parameters())\n\nfor epoch in range(epochs):\n    for batch in dataloader:\n        loss = model(batch)\n        loss.backward()\n        optimizer.step()\n```\n\n**Ray Train version** (scales to multi-GPU\u002Fmulti-node):\n```python\nfrom ray.train.torch import TorchTrainer\nfrom ray import train\n\ndef train_func(config):\n    model = MyModel()\n    optimizer = torch.optim.Adam(model.parameters())\n\n    # Prepare for distributed (automatic device placement)\n    model = train.torch.prepare_model(model)\n    dataloader = train.torch.prepare_data_loader(dataloader)\n\n    for epoch in range(epochs):\n        for batch in dataloader:\n            loss = model(batch)\n            loss.backward()\n            optimizer.step()\n\n            # Report metrics\n            train.report({\"loss\": loss.item()})\n\n# Scale to 8 GPUs\ntrainer = TorchTrainer(\n    train_func,\n    scaling_config=ScalingConfig(num_workers=8, use_gpu=True)\n)\ntrainer.fit()\n```\n\n**Benefits**: Same code runs on 1 GPU or 1000 GPUs\n\n### Workflow 2: HuggingFace Transformers integration\n\n```python\nfrom ray.train.huggingface import TransformersTrainer\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments\n\ndef train_func(config):\n    # Load model and tokenizer\n    model = AutoModelForCausalLM.from_pretrained(\"gpt2\")\n    tokenizer = AutoTokenizer.from_pretrained(\"gpt2\")\n\n    # Training arguments (HuggingFace API)\n    training_args = TrainingArguments(\n        output_dir=\".\u002Foutput\",\n        num_train_epochs=3,\n        per_device_train_batch_size=8,\n        learning_rate=2e-5,\n    )\n\n    # Ray automatically handles distributed training\n    from transformers import Trainer\n    trainer = Trainer(\n        model=model,\n        args=training_args,\n        train_dataset=train_dataset,\n    )\n\n    trainer.train()\n\n# Scale to multi-node (2 nodes × 8 GPUs = 16 workers)\ntrainer = TransformersTrainer(\n    train_func,\n    scaling_config=ScalingConfig(\n        num_workers=16,\n        use_gpu=True,\n        resources_per_worker={\"GPU\": 1}\n    )\n)\n\nresult = trainer.fit()\n```\n\n### Workflow 3: Hyperparameter tuning with Ray Tune\n\n```python\nfrom ray import tune\nfrom ray.train.torch import TorchTrainer\nfrom ray.tune.schedulers import ASHAScheduler\n\ndef train_func(config):\n    # Use hyperparameters from config\n    lr = config[\"lr\"]\n    batch_size = config[\"batch_size\"]\n\n    model = MyModel()\n    optimizer = torch.optim.Adam(model.parameters(), lr=lr)\n\n    model = train.torch.prepare_model(model)\n\n    for epoch in range(10):\n        # Training loop\n        loss = train_epoch(model, optimizer, batch_size)\n        train.report({\"loss\": loss, \"epoch\": epoch})\n\n# Define search space\nparam_space = {\n    \"lr\": tune.loguniform(1e-5, 1e-2),\n    \"batch_size\": tune.choice([16, 32, 64, 128])\n}\n\n# Run 20 trials with early stopping\ntuner = tune.Tuner(\n    TorchTrainer(\n        train_func,\n        scaling_config=ScalingConfig(num_workers=4, use_gpu=True)\n    ),\n    param_space=param_space,\n    tune_config=tune.TuneConfig(\n        num_samples=20,\n        scheduler=ASHAScheduler(metric=\"loss\", mode=\"min\")\n    )\n)\n\nresults = tuner.fit()\nbest = results.get_best_result(metric=\"loss\", mode=\"min\")\nprint(f\"Best hyperparameters: {best.config}\")\n```\n\n**Result**: Distributed hyperparameter search across cluster\n\n### Workflow 4: Checkpointing and fault tolerance\n\n```python\nfrom ray import train\nfrom ray.train import Checkpoint\n\ndef train_func(config):\n    model = MyModel()\n    optimizer = torch.optim.Adam(model.parameters())\n\n    # Try to resume from checkpoint\n    checkpoint = train.get_checkpoint()\n    if checkpoint:\n        with checkpoint.as_directory() as checkpoint_dir:\n            state = torch.load(f\"{checkpoint_dir}\u002Fmodel.pt\")\n            model.load_state_dict(state[\"model\"])\n            optimizer.load_state_dict(state[\"optimizer\"])\n            start_epoch = state[\"epoch\"]\n    else:\n        start_epoch = 0\n\n    model = train.torch.prepare_model(model)\n\n    for epoch in range(start_epoch, 100):\n        loss = train_epoch(model, optimizer)\n\n        # Save checkpoint every 10 epochs\n        if epoch % 10 == 0:\n            checkpoint = Checkpoint.from_directory(\n                train.get_context().get_trial_dir()\n            )\n            torch.save({\n                \"model\": model.state_dict(),\n                \"optimizer\": optimizer.state_dict(),\n                \"epoch\": epoch\n            }, checkpoint.path \u002F \"model.pt\")\n\n            train.report({\"loss\": loss}, checkpoint=checkpoint)\n\ntrainer = TorchTrainer(\n    train_func,\n    scaling_config=ScalingConfig(num_workers=8, use_gpu=True)\n)\n\n# Automatically resumes from checkpoint if training fails\nresult = trainer.fit()\n```\n\n### Workflow 5: Multi-node training\n\n```python\nfrom ray.train import ScalingConfig\n\n# Connect to Ray cluster\nray.init(address=\"auto\")  # Or ray.init(\"ray:\u002F\u002Fhead-node:10001\")\n\n# Train across 4 nodes × 8 GPUs = 32 workers\ntrainer = TorchTrainer(\n    train_func,\n    scaling_config=ScalingConfig(\n        num_workers=32,\n        use_gpu=True,\n        resources_per_worker={\"GPU\": 1, \"CPU\": 4},\n        placement_strategy=\"SPREAD\"  # Spread across nodes\n    )\n)\n\nresult = trainer.fit()\n```\n\n**Launch Ray cluster**:\n```bash\n# On head node\nray start --head --port=6379\n\n# On worker nodes\nray start --address=\u003Chead-node-ip>:6379\n```\n\n## When to use vs alternatives\n\n**Use Ray Train when**:\n- Training across multiple machines (multi-node)\n- Need hyperparameter tuning at scale\n- Want fault tolerance (auto-restart failed workers)\n- Elastic scaling (add\u002Fremove nodes during training)\n- Unified framework (same code for PyTorch\u002FTF\u002FHF)\n\n**Key advantages**:\n- **Multi-node orchestration**: Easiest multi-node setup\n- **Ray Tune integration**: Best-in-class hyperparameter tuning\n- **Fault tolerance**: Automatic recovery from failures\n- **Elastic**: Add\u002Fremove nodes without restarting\n- **Framework agnostic**: PyTorch, TensorFlow, HuggingFace, XGBoost\n\n**Use alternatives instead**:\n- **Accelerate**: Single-node multi-GPU, simpler\n- **PyTorch Lightning**: High-level abstractions, callbacks\n- **DeepSpeed**: Maximum performance, complex setup\n- **Raw DDP**: Maximum control, minimal overhead\n\n## Common issues\n\n**Issue: Ray cluster not connecting**\n\nCheck ray status:\n```bash\nray status\n\n# Should show:\n# - Nodes: 4\n# - GPUs: 32\n# - Workers: Ready\n```\n\nIf not connected:\n```bash\n# Restart head node\nray stop\nray start --head --port=6379 --dashboard-host=0.0.0.0\n\n# Restart worker nodes\nray stop\nray start --address=\u003Chead-ip>:6379\n```\n\n**Issue: Out of memory**\n\nReduce workers or use gradient accumulation:\n```python\nscaling_config=ScalingConfig(\n    num_workers=4,  # Reduce from 8\n    use_gpu=True\n)\n\n# In train_func, accumulate gradients\nfor i, batch in enumerate(dataloader):\n    loss = model(batch) \u002F accumulation_steps\n    loss.backward()\n\n    if (i + 1) % accumulation_steps == 0:\n        optimizer.step()\n        optimizer.zero_grad()\n```\n\n**Issue: Slow training**\n\nCheck if data loading is bottleneck:\n```python\nimport time\n\ndef train_func(config):\n    for epoch in range(epochs):\n        start = time.time()\n        for batch in dataloader:\n            data_time = time.time() - start\n            # Train...\n            start = time.time()\n            print(f\"Data loading: {data_time:.3f}s\")\n```\n\nIf data loading is slow, increase workers:\n```python\ndataloader = DataLoader(dataset, num_workers=8)\n```\n\n## Advanced topics\n\n**Multi-node setup**: See [references\u002Fmulti-node.md](references\u002Fmulti-node.md) for Ray cluster deployment on AWS, GCP, Kubernetes, and SLURM.\n\n**Hyperparameter tuning**: See [references\u002Fhyperparameter-tuning.md](references\u002Fhyperparameter-tuning.md) for Ray Tune integration, search algorithms (Optuna, HyperOpt), and population-based training.\n\n**Custom training loops**: See [references\u002Fcustom-loops.md](references\u002Fcustom-loops.md) for advanced Ray Train usage, custom backends, and integration with other frameworks.\n\n## Hardware requirements\n\n- **Single node**: 1+ GPUs (or CPUs)\n- **Multi-node**: 2+ machines with network connectivity\n- **Cloud**: AWS, GCP, Azure (Ray autoscaling)\n- **On-prem**: Kubernetes, SLURM clusters\n\n**Supported accelerators**:\n- NVIDIA GPUs (CUDA)\n- AMD GPUs (ROCm)\n- TPUs (Google Cloud)\n- CPUs\n\n## Resources\n\n- Docs: https:\u002F\u002Fdocs.ray.io\u002Fen\u002Flatest\u002Ftrain\u002Ftrain.html\n- GitHub: https:\u002F\u002Fgithub.com\u002Fray-project\u002Fray ⭐ 36,000+\n- Version: 2.40.0+\n- Examples: https:\u002F\u002Fdocs.ray.io\u002Fen\u002Flatest\u002Ftrain\u002Fexamples.html\n- Slack: https:\u002F\u002Fforms.gle\u002F9TSdDYUgxYs8SA9e8\n- Used by: OpenAI, Uber, Spotify, Shopify, Instacart\n\n\n","# Ray Train - Distributed Training Orchestration\n\n## Quick start\n\nRay Train scales machine learning training from single GPU to multi-node clusters with minimal code changes.\n\n**Installation**:\n```bash\npip install -U \"ray[train]\"\n```\n\n**Basic PyTorch training** (single node):\n\n```python\nimport ray\nfrom ray import train\nfrom ray.train import ScalingConfig\nfrom ray.train.torch import TorchTrainer\nimport torch\nimport torch.nn as nn\n\n# Define training function\ndef train_func(config):\n    # Your normal PyTorch code\n    model = nn.Linear(10, 1)\n    optimizer = torch.optim.SGD(model.parameters(), lr=0.01)\n\n    # Prepare for distributed (Ray handles device placement)\n    model = train.torch.prepare_model(model)\n\n    for epoch in range(10):\n        # Your training loop\n        output = model(torch.randn(32, 10))\n        loss = output.sum()\n        loss.backward()\n        optimizer.step()\n        optimizer.zero_grad()\n\n        # Report metrics (logged automatically)\n        train.report({\"loss\": loss.item(), \"epoch\": epoch})\n\n# Run distributed training\ntrainer = TorchTrainer(\n    train_func,\n    scaling_config=ScalingConfig(\n        num_workers=4,  # 4 GPUs\u002Fworkers\n        use_gpu=True\n    )\n)\n\nresult = trainer.fit()\nprint(f\"Final loss: {result.metrics['loss']}\")\n```\n\n**That's it!** Ray handles:\n- Distributed coordination\n- GPU allocation\n- Fault tolerance\n- Checkpointing\n- Metric aggregation\n\n## Common workflows\n\n### Workflow 1: Scale existing PyTorch code\n\n**Original single-GPU code**:\n```python\nmodel = MyModel().cuda()\noptimizer = torch.optim.Adam(model.parameters())\n\nfor epoch in range(epochs):\n    for batch in dataloader:\n        loss = model(batch)\n        loss.backward()\n        optimizer.step()\n```\n\n**Ray Train version** (scales to multi-GPU\u002Fmulti-node):\n```python\nfrom ray.train.torch import TorchTrainer\nfrom ray import train\n\ndef train_func(config):\n    model = MyModel()\n    optimizer = torch.optim.Adam(model.parameters())\n\n    # Prepare for distributed (automatic device placement)\n    model = train.torch.prepare_model(model)\n    dataloader = train.torch.prepare_data_loader(dataloader)\n\n    for epoch in range(epochs):\n        for batch in dataloader:\n            loss = model(batch)\n            loss.backward()\n            optimizer.step()\n\n            # Report metrics\n            train.report({\"loss\": loss.item()})\n\n# Scale to 8 GPUs\ntrainer = TorchTrainer(\n    train_func,\n    scaling_config=ScalingConfig(num_workers=8, use_gpu=True)\n)\ntrainer.fit()\n```\n\n**Benefits**: Same code runs on 1 GPU or 1000 GPUs\n\n### Workflow 2: HuggingFace Transformers integration\n\n```python\nfrom ray.train.huggingface import TransformersTrainer\nfrom transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments\n\ndef train_func(config):\n    # Load model and tokenizer\n    model = AutoModelForCausalLM.from_pretrained(\"gpt2\")\n    tokenizer = AutoTokenizer.from_pretrained(\"gpt2\")\n\n    # Training arguments (HuggingFace API)\n    training_args = TrainingArguments(\n        output_dir=\".\u002Foutput\",\n        num_train_epochs=3,\n        per_device_train_batch_size=8,\n        learning_rate=2e-5,\n    )\n\n    # Ray automatically handles distributed training\n    from transformers import Trainer\n    trainer = Trainer(\n        model=model,\n        args=training_args,\n        train_dataset=train_dataset,\n    )\n\n    trainer.train()\n\n# Scale to multi-node (2 nodes × 8 GPUs = 16 workers)\ntrainer = TransformersTrainer(\n    train_func,\n    scaling_config=ScalingConfig(\n        num_workers=16,\n        use_gpu=True,\n        resources_per_worker={\"GPU\": 1}\n    )\n)\n\nresult = trainer.fit()\n```\n\n### Workflow 3: Hyperparameter tuning with Ray Tune\n\n```python\nfrom ray import tune\nfrom ray.train.torch import TorchTrainer\nfrom ray.tune.schedulers import ASHAScheduler\n\ndef train_func(config):\n    # Use hyperparameters from config\n    lr = config[\"lr\"]\n    batch_size = config[\"batch_size\"]\n\n    model = MyModel()\n    optimizer = torch.optim.Adam(model.parameters(), lr=lr)\n\n    model = train.torch.prepare_model(model)\n\n    for epoch in range(10):\n        # Training loop\n        loss = train_epoch(model, optimizer, batch_size)\n        train.report({\"loss\": loss, \"epoch\": epoch})\n\n# Define search space\nparam_space = {\n    \"lr\": tune.loguniform(1e-5, 1e-2),\n    \"batch_size\": tune.choice([16, 32, 64, 128])\n}\n\n# Run 20 trials with early stopping\ntuner = tune.Tuner(\n    TorchTrainer(\n        train_func,\n        scaling_config=ScalingConfig(num_workers=4, use_gpu=True)\n    ),\n    param_space=param_space,\n    tune_config=tune.TuneConfig(\n        num_samples=20,\n        scheduler=ASHAScheduler(metric=\"loss\", mode=\"min\")\n    )\n)\n\nresults = tuner.fit()\nbest = results.get_best_result(metric=\"loss\", mode=\"min\")\nprint(f\"Best hyperparameters: {best.config}\")\n```\n\n**Result**: Distributed hyperparameter search across cluster\n\n### Workflow 4: Checkpointing and fault tolerance\n\n```python\nfrom ray import train\nfrom ray.train import Checkpoint\n\ndef train_func(config):\n    model = MyModel()\n    optimizer = torch.optim.Adam(model.parameters())\n\n    # Try to resume from checkpoint\n    checkpoint = train.get_checkpoint()\n    if checkpoint:\n        with checkpoint.as_directory() as checkpoint_dir:\n            state = torch.load(f\"{checkpoint_dir}\u002Fmodel.pt\")\n            model.load_state_dict(state[\"model\"])\n            optimizer.load_state_dict(state[\"optimizer\"])\n            start_epoch = state[\"epoch\"]\n    else:\n        start_epoch = 0\n\n    model = train.torch.prepare_model(model)\n\n    for epoch in range(start_epoch, 100):\n        loss = train_epoch(model, optimizer)\n\n        # Save checkpoint every 10 epochs\n        if epoch % 10 == 0:\n            checkpoint = Checkpoint.from_directory(\n                train.get_context().get_trial_dir()\n            )\n            torch.save({\n                \"model\": model.state_dict(),\n                \"optimizer\": optimizer.state_dict(),\n                \"epoch\": epoch\n            }, checkpoint.path \u002F \"model.pt\")\n\n            train.report({\"loss\": loss}, checkpoint=checkpoint)\n\ntrainer = TorchTrainer(\n    train_func,\n    scaling_config=ScalingConfig(num_workers=8, use_gpu=True)\n)\n\n# Automatically resumes from checkpoint if training fails\nresult = trainer.fit()\n```\n\n### Workflow 5: Multi-node training\n\n```python\nfrom ray.train import ScalingConfig\n\n# Connect to Ray cluster\nray.init(address=\"auto\")  # Or ray.init(\"ray:\u002F\u002Fhead-node:10001\")\n\n# Train across 4 nodes × 8 GPUs = 32 workers\ntrainer = TorchTrainer(\n    train_func,\n    scaling_config=ScalingConfig(\n        num_workers=32,\n        use_gpu=True,\n        resources_per_worker={\"GPU\": 1, \"CPU\": 4},\n        placement_strategy=\"SPREAD\"  # Spread across nodes\n    )\n)\n\nresult = trainer.fit()\n```\n\n**Launch Ray cluster**:\n```bash\n# On head node\nray start --head --port=6379\n\n# On worker nodes\nray start --address=\u003Chead-node-ip>:6379\n```\n\n## When to use vs alternatives\n\n**Use Ray Train when**:\n- Training across multiple machines (multi-node)\n- Need hyperparameter tuning at scale\n- Want fault tolerance (auto-restart failed workers)\n- Elastic scaling (add\u002Fremove nodes during training)\n- Unified framework (same code for PyTorch\u002FTF\u002FHF)\n\n**Key advantages**:\n- **Multi-node orchestration**: Easiest multi-node setup\n- **Ray Tune integration**: Best-in-class hyperparameter tuning\n- **Fault tolerance**: Automatic recovery from failures\n- **Elastic**: Add\u002Fremove nodes without restarting\n- **Framework agnostic**: PyTorch, TensorFlow, HuggingFace, XGBoost\n\n**Use alternatives instead**:\n- **Accelerate**: Single-node multi-GPU, simpler\n- **PyTorch Lightning**: High-level abstractions, callbacks\n- **DeepSpeed**: Maximum performance, complex setup\n- **Raw DDP**: Maximum control, minimal overhead\n\n## Common issues\n\n**Issue: Ray cluster not connecting**\n\nCheck ray status:\n```bash\nray status\n\n# Should show:\n# - Nodes: 4\n# - GPUs: 32\n# - Workers: Ready\n```\n\nIf not connected:\n```bash\n# Restart head node\nray stop\nray start --head --port=6379 --dashboard-host=0.0.0.0\n\n# Restart worker nodes\nray stop\nray start --address=\u003Chead-ip>:6379\n```\n\n**Issue: Out of memory**\n\nReduce workers or use gradient accumulation:\n```python\nscaling_config=ScalingConfig(\n    num_workers=4,  # Reduce from 8\n    use_gpu=True\n)\n\n# In train_func, accumulate gradients\nfor i, batch in enumerate(dataloader):\n    loss = model(batch) \u002F accumulation_steps\n    loss.backward()\n\n    if (i + 1) % accumulation_steps == 0:\n        optimizer.step()\n        optimizer.zero_grad()\n```\n\n**Issue: Slow training**\n\nCheck if data loading is bottleneck:\n```python\nimport time\n\ndef train_func(config):\n    for epoch in range(epochs):\n        start = time.time()\n        for batch in dataloader:\n            data_time = time.time() - start\n            # Train...\n            start = time.time()\n            print(f\"Data loading: {data_time:.3f}s\")\n```\n\nIf data loading is slow, increase workers:\n```python\ndataloader = DataLoader(dataset, num_workers=8)\n```\n\n## Advanced topics\n\n**Multi-node setup**: See [references\u002Fmulti-node.md](references\u002Fmulti-node.md) for Ray cluster deployment on AWS, GCP, Kubernetes, and SLURM.\n\n**Hyperparameter tuning**: See [references\u002Fhyperparameter-tuning.md](references\u002Fhyperparameter-tuning.md) for Ray Tune integration, search algorithms (Optuna, HyperOpt), and population-based training.\n\n**Custom training loops**: See [references\u002Fcustom-loops.md](references\u002Fcustom-loops.md) for advanced Ray Train usage, custom backends, and integration with other frameworks.\n\n## Hardware requirements\n\n- **Single node**: 1+ GPUs (or CPUs)\n- **Multi-node**: 2+ machines with network connectivity\n- **Cloud**: AWS, GCP, Azure (Ray autoscaling)\n- **On-prem**: Kubernetes, SLURM clusters\n\n**Supported accelerators**:\n- NVIDIA GPUs (CUDA)\n- AMD GPUs (ROCm)\n- TPUs (Google Cloud)\n- CPUs\n\n## Resources\n\n- Docs: https:\u002F\u002Fdocs.ray.io\u002Fen\u002Flatest\u002Ftrain\u002Ftrain.html\n- GitHub: https:\u002F\u002Fgithub.com\u002Fray-project\u002Fray ⭐ 36,000+\n- Version: 2.40.0+\n- Examples: https:\u002F\u002Fdocs.ray.io\u002Fen\u002Flatest\u002Ftrain\u002Fexamples.html\n- Slack: https:\u002F\u002Fforms.gle\u002F9TSdDYUgxYs8SA9e8\n- Used by: OpenAI, Uber, Spotify, Shopify, Instacart\n\n\n",{"name":5,"tags":44,"author":49,"license":15,"version":50,"category":51,"description":7,"dependencies":52},[20,35,45,19,27,46,26,25,47,48],"Orchestration","Fault Tolerance","PyTorch","TensorFlow","Orchestra Research","1.0.0","ai-research",[53,54,55],"ray[train]","torch","transformers","main","0da4485785ee599388900be50db1635489d53056",".agents\u002Fskills\u002Fqc-ray-train","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6","https:\u002F\u002Fdisha6.6.thenitishkr.in\u002F","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh","www.thenitishkr","thenitishkr | AI • Cyber-security • Research\r\n","2026-11-08 03:33:25",{"source":8,"skillName":5,"commands":66,"prompt":76,"zipAvailable":77},[67,70,73],{"tool":68,"command":69},"npx","npx skills add Tashima-Tarsh\u002FDisha6.6 --skill ray-train",{"tool":71,"command":72},"bunx","bunx skills add Tashima-Tarsh\u002FDisha6.6 --skill ray-train",{"tool":74,"command":75},"pnpm","pnpm dlx skills add Tashima-Tarsh\u002FDisha6.6 --skill ray-train","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fqc-ray-train\u002FSKILL.md",true,[79],{"path":80,"resourceType":81,"resourceTypeDesc":82,"sizeBytes":83,"sourceHtmlUrl":84},"references\u002Fmulti-node.md","REFERENCE","参考文档",13526,"https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fqc-ray-train\u002Freferences\u002Fmulti-node.md",[86,118,143,170,195,218],{"skillNo":87,"name":88,"nameZh":6,"description":89,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":90,"categories":91,"keywords":97,"publishedAt":39},"01M6QRD07D4TGNDK6VJ4EHF8BV","youtube-full","Use when the user needs YouTube transcripts, video search, channel browsing, playlist extraction, or content monitoring. Trigger phrases: 'get the transcript for', 'search YouTube for', 'what are the latest videos on', 'list this playlist', 'monitor this channel', or any request involving a YouTube URL, video ID, or @handle. Do NOT use for downloading video or audio files, YouTube engagement data (likes, comments), or private\u002Fage-restricted videos.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fyoutube-full\u002FSKILL.md",[92],{"categoryNo":93,"categoryName":94,"taxonomyType":95,"taxonomyTypeDesc":96},"01JBSEEDCAT000000000000016","图像与多媒体","SCENARIO","使用场景",[98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117],"youtube full","youtube","full","transcript","search","monitoring","data","Video","Audio","list","files","do","get","age","trigger","URL","for","browsing","request","youtube transcripts",{"skillNo":119,"name":120,"nameZh":6,"description":121,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":122,"categories":123,"keywords":124,"publishedAt":39},"01M6QRCJ3112VYPA97YG7KGMC2","x-twitter-growth","X\u002FTwitter growth engine for building audience, crafting viral content, and analyzing engagement. Use when the user wants to grow on X\u002FTwitter, write tweets or threads, analyze their X profile, research competitors on X, plan a posting strategy, or optimize engagement. Complements social-content (generic multi-platform) with X-specific depth: algorithm mechanics, thread engineering, reply strategy, profile optimization, and competitive intelligence via web search.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fx-twitter-growth\u002FSKILL.md",[],[125,126,127,128,102,129,130,131,132,133,134,135,136,137,138,139,140,141,142,114],"twitter","x twitter growth","growth","research","content","engineering","plan","web search","competitors","analyze","engine","write","strategy","use","platform","grow","posting","competitive",{"skillNo":144,"name":145,"nameZh":6,"description":146,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":147,"categories":148,"keywords":152,"publishedAt":39},"01M6QRBN3EHQNJY42RERP7A149","write-a-skill","Create new agent skills with proper structure, progressive disclosure, and bundled resources. Use when user wants to create, write, build, or author a new skill.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fwrite-a-skill\u002FSKILL.md",[149],{"categoryNo":150,"categoryName":151,"taxonomyType":95,"taxonomyTypeDesc":96},"01JBSEEDCAT000000000000023","Skill与智能体开发",[153,136,154,155,156,157,158,159,138,160,161,56,162,163,164,165,166,167,168,169],"write a skill","github","create","https","productivity","matt pocock","direct","example","build","new skill","structure","new agent","resources","with","tree","progressive disclosure","author",{"skillNo":171,"name":172,"nameZh":6,"description":173,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":174,"categories":175,"keywords":176,"publishedAt":39},"01M6QRB6G8P33PNGAF9Y69EJ9T","wrap-up","Close out a launched Claude Managed Agent — recap every primitive the founder now owns, regenerate the single-file overview page, and suggest the next 1-2 upgrades. Use when the user says \"wrap up\", \"close this out\", \"what do I own now\", \"give me the summary\", \"recap the agent\", or when the orchestrator routes phase=wrap-up. primitives_inventory.py tables everything owned (agent, environment, session, memory, outcome, deployment); overview_page.py regenerates a self-contained .\u002Fmy-agent\u002Fagent-overview.html; upgrade_suggester.py ranks the next moves from recorded deferrals plus standing hardening steps. Companion to run-without-you; the last stop before phase=done.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fwrap-up\u002FSKILL.md",[],[177,178,179,180,181,182,183,184,185,138,186,109,187,188,189,190,191,192,193,194],"wrap up","up","wrap","gemini","deployment","run","file","hardening","closeout","stop","upgrade","founder","close","companion","you","tables","directions","phase",{"skillNo":196,"name":197,"nameZh":6,"description":198,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":199,"categories":200,"keywords":202,"publishedAt":39},"01M6QRA5AYC1RBFZ39GTWGS3N8","workflow-builder","Design and write deterministic multi-agent workflow scripts (.js files in .claude\u002Fworkflows\u002F) for Claude Code's Workflow tool. Use when a user wants to build, create, author, scaffold, or run a custom Claude Code workflow, orchestrate sub-agents (fan-out, pipeline, loop, judge-panel), or automate a repeatable multi-step task across fresh-context agents.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fworkflow-builder\u002FSKILL.md",[201],{"categoryNo":150,"categoryName":151,"taxonomyType":95,"taxonomyTypeDesc":96},[203,204,205,206,207,155,182,208,209,210,211,212,136,213,108,30,214,215,216,217],"workflow","workflow builder","builder","design","workflows","code","orchestrate","pipeline","loop","multi-agent","context","fresh","scripts","judge","claude code workflow",{"skillNo":219,"name":220,"nameZh":6,"description":221,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":222,"categories":223,"keywords":224,"publishedAt":39},"01M6QR7QMCZESKQA0ERW1E657M","weekly-review","Use when someone wants to run a weekly review, close open loops, audit stalled projects and commitments, get their system back to trusted, restart a lapsed review habit, or says \"\u002Fcs:weekly-review\". Walks David Allen's three-phase loop — GET CLEAR, GET CURRENT, GET CREATIVE — with deterministic scripts that inventory open loops, gate the checklist with named gaps, and score commitment health 0-100.","https:\u002F\u002Fgithub.com\u002FTashima-Tarsh\u002FDisha6.6\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fweekly-review\u002FSKILL.md",[],[225,226,227,228,229,230,231,232,233,211,234,235,236,237,238,239,110,240,241,194],"review","weekly review","weekly","deterministic scripts","reflect","feeds","inventory","open","team","projects","project-management","scanner","conversation","back","project","sprint","cadence"]