wandb

Weights & Biases

Monitor, analyze, and compare W&B training runs.

Setup

wandb login
# Or set WANDB_API_KEY in environment

Scripts

Characterize a Run (Full Health Analysis)

~/clawd/venv/bin/python3 ~/clawd/skills/wandb/scripts/characterize_run.py ENTITY/PROJECT/RUN_ID

Analyzes:

• Loss curve trend (start → current, % change, direction)
• Gradient norm health (exploding/vanishing detection)
• Eval metrics (if present)
• Stall detection (heartbeat age)
• Progress & ETA estimate
• Config highlights
• Overall health verdict

Options: --json for machine-readable output.

Watch All Running Jobs

~/clawd/venv/bin/python3 ~/clawd/skills/wandb/scripts/watch_runs.py ENTITY [--projects p1,p2]

Quick health summary of all running jobs plus recent failures/completions. Ideal for morning briefings.

Options:

• --projects p1,p2 — Specific projects to check
• --all-projects — Check all projects
• --hours N — Hours to look back for finished runs (default: 24)
• --json — Machine-readable output

Compare Two Runs

~/clawd/venv/bin/python3 ~/clawd/skills/wandb/scripts/compare_runs.py ENTITY/PROJECT/RUN_A ENTITY/PROJECT/RUN_B

Side-by-side comparison:

• Config differences (highlights important params)
• Loss curves at same steps
• Gradient norm comparison
• Eval metrics
• Performance (tokens/sec, steps/hour)
• Winner verdict

Python API Quick Reference

import wandb
api = wandb.Api()

# Get runs
runs = api.runs("entity/project", {"state": "running"})

# Run properties
run.state      # running | finished | failed | crashed | canceled
run.name       # display name
run.id         # unique identifier
run.summary    # final/current metrics
run.config     # hyperparameters
run.heartbeat_at # stall detection

# Get history
history = list(run.scan_history(keys=["train/loss", "train/grad_norm"]))

Metric Key Variations

Scripts handle these automatically:

• Loss: train/loss, loss, train_loss, training_loss
• Gradients: train/grad_norm, grad_norm, gradient_norm
• Steps: train/global_step, global_step, step, _step
• Eval: eval/loss, eval_loss, eval/accuracy, eval_acc

Health Thresholds

• Gradients > 10: Exploding (critical)
• Gradients > 5: Spiky (warning)
• Gradients < 0.0001: Vanishing (warning)
• Heartbeat > 30min: Stalled (critical)
• Heartbeat > 10min: Slow (warning)

Integration Notes

For morning briefings, use watch_runs.py --json and parse the output.

For detailed analysis of a specific run, use characterize_run.py.

For A/B testing or hyperparameter comparisons, use compare_runs.py.