[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7BR1P03CS2HSWJK4EPR5PCX":3,"skill-same:01M7BR1P03CS2HSWJK4EPR5PCX":141},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":24,"commitSha":25,"skillDirectory":26,"forksCount":27,"repositoryHtmlUrl":28,"repositoryHomepageUrl":6,"ownerHtmlUrl":29,"ownerBlogUrl":30,"ownerDescription":31,"updatedAt":32,"install":33,"resources":46},"01M7BR1P03CS2HSWJK4EPR5PCX","spinning-up-deep-rl",null,"Knowledge base from \"Spinning Up in Deep RL\" by Joshua Achiam (OpenAI, MIT-licensed). Use when applying Achiam's frameworks for RL fundamentals and MDPs, the model-free algorithm taxonomy, policy gradient derivations, the six reference algorithms (VPG, TRPO, PPO, DDPG, TD3, SAC), debugging silently-failing RL code, or running rigorous multi-seed RL experiments.","devlify-marketplace\u002Fdevlify-skills","devlify-marketplace","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F196752365?v=4","01M7BPJC94F4G2AG0T3N220BQ3","Sadiq Abdullahi","01M7BPJC9QENFCHFM5NEFAJR1Y",1,"MIT","https:\u002F\u002Fgithub.com\u002Fdevlify-marketplace\u002Fdevlify-skills\u002Fblob\u002Fmain\u002Fengineering\u002Fspinning-up-deep-rl\u002Fskills\u002Fspinning-up-deep-rl\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: spinning-up-deep-rl\ndescription: \"Knowledge base from \\\"Spinning Up in Deep RL\\\" by Joshua Achiam (OpenAI, MIT-licensed). Use when applying Achiam's frameworks for RL fundamentals and MDPs, the model-free algorithm taxonomy, policy gradient derivations, the six reference algorithms (VPG, TRPO, PPO, DDPG, TD3, SAC), debugging silently-failing RL code, or running rigorous multi-seed RL experiments.\"\n---\n\n# Spinning Up in Deep RL\n\n**Author**: Joshua Achiam (OpenAI) | **Source**: spinningup.readthedocs.io, MIT | **Chapters**: 20 | **Generated**: 2026-08-25\n\n## How to Use This Skill\n\n- **No argument** — load the core frameworks below\n- **A topic** — ask about `advantage function`, `target networks`, `entropy regularization`;\n  I resolve it through the Topic Index and read that chapter file\n- **`chNN`** — I load that chapter's summary\n- **\"what chapters do you have?\"** — the full index\n\n```\n\u002Fcs:spinning-up-deep-rl                      # core frameworks + chapter index\n\u002Fcs:spinning-up-deep-rl entropy regularization   # topic index -> ch19, read that chapter\n\u002Fcs:spinning-up-deep-rl ch09                 # one chapter summary\n```\n\nWhen you ask about something not in Core Frameworks, I read the relevant chapter file before\nanswering rather than guessing from the index.\n\n---\n\n## Core Frameworks & Mental Models\n\n### The RL problem\n`pi* = argmax_pi J(pi)`, where `J(pi) = E_{tau~pi}[R(tau)]`. Every algorithm approximates\nthis; where it substitutes a different objective (a Bellman residual, a surrogate), that\nsubstitution is the source of its failure modes. Four value functions — `V^pi`, `Q^pi`, `V*`,\n`Q*` — all obey Bellman self-consistency, and `a*(s) = argmax_a Q*(s,a)` is why Q-learning\nis a viable family at all. Advantage `A^pi(s,a) = Q^pi(s,a) - V^pi(s)` is the relative-quality\nsignal policy gradients run on. (ch07)\n\n### The two branching questions\nPlace any algorithm by asking: **does it have or learn a model**, and **what does it learn**\n(policy, Q-function, value function, model). That generates the whole landscape. (ch08)\n\n### Policy optimization vs Q-learning — the central trade-off\n- **Policy optimization** is *principled: you directly optimize the thing you want.* Stable\n  and reliable. On-policy, so it cannot reuse data, so it is sample-hungry.\n- **Q-learning** only *indirectly* optimizes performance, by training `Q_theta` to satisfy a\n  self-consistency equation. Many failure modes, so less stable. But substantially more\n  sample efficient when it works, because it reuses everything.\n- **Satisfying the Bellman equations well carries no guarantee of good policy performance.**\n- The two are not exclusive — DDPG and SAC live between them deliberately. (ch08)\n\n### The policy gradient template\n`grad J = E[ sum_t grad log pi_theta(a_t|s_t) * Phi_t ]`. Five valid choices of `Phi_t`:\nfull return, reward-to-go, reward-to-go minus a baseline, `Q^pi`, and `A^pi`. All share an\nexpectation and differ in variance. Two rules get you from the first to the last:\n- **Don't let the past distract you** — drop rewards obtained before the action. Those terms\n  had zero mean and nonzero variance: pure noise.\n- **Baselines** — by the EGLP lemma, any state-only `b(s)` can be added or subtracted freely.\n  The standard choice is `V^pi(s_t)`, learned by MSE regression onto reward-to-go. (ch09)\n\n### The policy-gradient loss is not a loss function\nIts data distribution depends on the parameters, and it does not measure performance even in\nexpectation. Only at the current parameters, with data from those parameters, does it have\nthe negative gradient of performance. **You can send it to negative infinity while performance\ncraters, and it usually will. Only average return means anything.** (ch09)\n\n### Broken RL code almost always fails silently\nIt runs fine; the agent just never learns. Usually something is computed with the wrong\nequation, on the wrong distribution, or piped to the wrong place. **If it doesn't work, assume\nthere's a bug** before touching hyperparameters. Debug by measuring everything and reading the\ncode critically. The archetype is one missing `squeeze`: a `[N]` vs `[N,1]` shape mismatch is\nbroadcast-compatible, raises nothing, and silently turns the Bellman backup into an `[N,N]`\nmatrix. (ch10, ch12)\n\n### Learn by doing\nWrite your own implementations, **shortest correct version** of each, **simplest algorithms\nfirst**. VPG, DQN, A2C, PPO, DDPG, roughly in that order; ~250-300 lines each. Single-threaded\nbefore parallel. **Iterate fast in simple environments — under 5 minutes turnaround at the\ndebug stage.** Do not attempt Atari or Humanoid before the toy task works. Read papers for\ntheir ablations and supplementary material, but **do not overfit to paper details** (the\noriginal DDPG's architecture, init scheme and batch norm are not strictly necessary) **or to\nexisting implementations** (their abstractions serve reuse, not your single use case). (ch10)\n\n### Rigor: four standards\n1. **Fair comparisons** — tune the baseline as hard as your method; never handicap it.\n2. **Remove stochasticity as a confounder** — at least 3 seeds, 10 or more to be thorough.\n   Two seed groups can produce curves that look like different distributions.\n3. **High-integrity experiments** — launch fresh final runs and precommit to reporting them.\n   Tuning produces hypotheses; final runs produce conclusions.\n4. **Check each claim separately** — ablate every design decision. (ch10)\n\n### The safe-step family (on-policy)\nVPG takes an unconstrained gradient step, so a single bad step can collapse performance.\n**TRPO** constrains the step in **KL-divergence between policies, not distance in parameter\nspace**, then backtracking-line-searches until the exact constraint holds. **PPO** drops the\nconstraint and instead **clips the objective so the policy gains nothing by moving far**,\nwhich is first-order, far simpler, and empirically at least as good. (ch14, ch15, ch16)\n\n### The overestimation family (off-policy)\nDDPG amortizes the intractable continuous `max_a Q(s,a)` into a learned policy:\n`max_a Q(s,a) ~= Q(s, mu(s))`. It needs a **replay buffer** (licensed because the Bellman\nequation is indifferent to how data was collected) and **target networks** (because the target\notherwise depends on the parameters being trained). Its failure mode is **Q-value\noverestimation, which the policy actively exploits**. **TD3** answers with clipped double-Q,\ndelayed policy updates and target policy smoothing. **SAC** adds **entropy regularization**,\nmaking the explore-exploit trade-off an explicit coefficient `alpha`. (ch17, ch18, ch19)\n\n---\n\n## Chapter Index\n\n| # | Title | Key Frameworks |\n|---|-------|----------------|\n| [ch01](chapters\u002Fch01-introduction.md) | Introduction | The missing middle step, Code Design Philosophy |\n| [ch02](chapters\u002Fch02-installation.md) | Installation | Install-then-verify, MuJoCo optionality |\n| [ch03](chapters\u002Fch03-algorithm-lineup.md) | Algorithms: What's Included and Why | The two lineages, on\u002Foff-policy trade-off, code template |\n| [ch04](chapters\u002Fch04-running-experiments.md) | Running Experiments | One flag per kwarg, ExperimentGrid, save-dir suffixes |\n| [ch05](chapters\u002Fch05-experiment-outputs.md) | Experiment Outputs | Tools not files, watch-then-measure |\n| [ch06](chapters\u002Fch06-plotting-results.md) | Plotting Results | `Performance` alias, prefix autocompletion, seed averaging |\n| [ch07](chapters\u002Fch07-key-concepts-in-rl.md) | Part 1: Key Concepts in RL | MDPs, four value functions, Bellman equations, advantage |\n| [ch08](chapters\u002Fch08-kinds-of-rl-algorithms.md) | Part 2: Kinds of RL Algorithms | Taxonomy, model bias, policy-opt vs Q-learning |\n| [ch09](chapters\u002Fch09-intro-to-policy-optimization.md) | Part 3: Intro to Policy Optimization | Log-derivative trick, EGLP lemma, reward-to-go, baselines |\n| [ch10](chapters\u002Fch10-spinning-up-as-a-researcher.md) | Spinning Up as a Deep RL Researcher | Learn by doing, three idea frames, four rigor standards |\n| [ch11](chapters\u002Fch11-key-papers-in-deep-rl.md) | Key Papers in Deep RL | 13-section topic map |\n| [ch12](chapters\u002Fch12-exercises.md) | Exercises | Problem Set 1 and 2, the silent DDPG bug |\n| [ch13](chapters\u002Fch13-benchmarks.md) | Benchmarks | The parity disclosure, family-specific metrics |\n| [ch14](chapters\u002Fch14-vpg.md) | Vanilla Policy Gradient | The six-step loop |\n| [ch15](chapters\u002Fch15-trpo.md) | Trust Region Policy Optimization | KL trust region, line search, conjugate gradient |\n| [ch16](chapters\u002Fch16-ppo.md) | Proximal Policy Optimization | PPO-Clip, KL early stopping |\n| [ch17](chapters\u002Fch17-ddpg.md) | Deep Deterministic Policy Gradient | MSBE, replay buffers, target networks, polyak |\n| [ch18](chapters\u002Fch18-td3.md) | Twin Delayed DDPG | Clipped double-Q, delayed updates, target smoothing |\n| [ch19](chapters\u002Fch19-sac.md) | Soft Actor-Critic | Entropy regularization, reparameterization, squashed Gaussian |\n| [ch20](chapters\u002Fch20-logger-and-utilities.md) | Logger, MPI Tools and Run Utils | EpochLogger pattern, MPI PyTorch order |\n\n## Topic Index\n\n- **Advantage function** ch07, ch09, ch14\n- **Baselines** ch09\n- **Bellman equations** ch07, ch17\n- **Benchmarks \u002F parity** ch13, ch01\n- **Clipped double-Q** ch18, ch19\n- **Continuous action spaces** ch07, ch17\n- **Debugging \u002F silent failure** ch10, ch12\n- **DDPG** ch17, ch03, ch08\n- **Entropy regularization** ch19\n- **Exploration vs exploitation** ch14, ch17, ch19\n- **GAE** ch09, ch14\n- **Installation** ch02\n- **KL divergence \u002F trust region** ch15, ch16\n- **Logging** ch20, ch05\n- **MDPs** ch07\n- **Model-based RL** ch08\n- **MPI \u002F parallelization** ch20, ch02, ch04\n- **MSBE** ch17\n- **Off-policy** ch03, ch08, ch17\n- **On-policy** ch03, ch08, ch14\n- **Papers \u002F literature** ch11, ch10\n- **Plotting** ch06, ch13\n- **Policies (categorical, Gaussian, squashed)** ch07, ch19\n- **Policy gradient derivation** ch09\n- **PPO** ch16, ch03\n- **Q-learning** ch08, ch07\n- **Replay buffer** ch17\n- **Reparameterization trick** ch19, ch10\n- **Research process \u002F rigor** ch10, ch13\n- **Reward-to-go** ch09\n- **Running experiments** ch04, ch05\n- **SAC** ch19, ch03, ch08\n- **Seeds \u002F variance** ch10, ch13, ch04\n- **Target networks \u002F polyak** ch17, ch18\n- **TD3** ch18, ch12\n- **TRPO** ch15, ch03\n- **Value functions** ch07, ch09\n- **VPG** ch14, ch09\n\n## Supporting Files\n\n- [glossary.md](glossary.md) · [patterns.md](patterns.md) · [cheatsheet.md](cheatsheet.md)\n\n## Scope & Limits\n\nCovers the Spinning Up documentation only, as of the January 2020 PyTorch update. It does\n**not** cover: DQN and the discrete-action value-learning family (referenced, never\nimplemented here), recurrent or convolutional architectures, partially-observed settings,\nmodel-based implementations, exploration\u002Fmeta-RL\u002Fhierarchy beyond ch11's reading list, or any\ndeep RL work after early 2020. The six implementations are educational; ch13 says which are\nresearch-grade. For topics beyond this source, I say so rather than improvising.\n\n---\n\n*Compiled from OpenAI's Spinning Up in Deep RL documentation (MIT, Copyright (c) 2018 OpenAI),\nprimarily developed by Joshua Achiam. Structured study notes, not a reproduction of the source.*\n","# Spinning Up in Deep RL\n\n**Author**: Joshua Achiam (OpenAI) | **Source**: spinningup.readthedocs.io, MIT | **Chapters**: 20 | **Generated**: 2026-08-25\n\n## How to Use This Skill\n\n- **No argument** — load the core frameworks below\n- **A topic** — ask about `advantage function`, `target networks`, `entropy regularization`;\n  I resolve it through the Topic Index and read that chapter file\n- **`chNN`** — I load that chapter's summary\n- **\"what chapters do you have?\"** — the full index\n\n```\n\u002Fcs:spinning-up-deep-rl                      # core frameworks + chapter index\n\u002Fcs:spinning-up-deep-rl entropy regularization   # topic index -> ch19, read that chapter\n\u002Fcs:spinning-up-deep-rl ch09                 # one chapter summary\n```\n\nWhen you ask about something not in Core Frameworks, I read the relevant chapter file before\nanswering rather than guessing from the index.\n\n---\n\n## Core Frameworks & Mental Models\n\n### The RL problem\n`pi* = argmax_pi J(pi)`, where `J(pi) = E_{tau~pi}[R(tau)]`. Every algorithm approximates\nthis; where it substitutes a different objective (a Bellman residual, a surrogate), that\nsubstitution is the source of its failure modes. Four value functions — `V^pi`, `Q^pi`, `V*`,\n`Q*` — all obey Bellman self-consistency, and `a*(s) = argmax_a Q*(s,a)` is why Q-learning\nis a viable family at all. Advantage `A^pi(s,a) = Q^pi(s,a) - V^pi(s)` is the relative-quality\nsignal policy gradients run on. (ch07)\n\n### The two branching questions\nPlace any algorithm by asking: **does it have or learn a model**, and **what does it learn**\n(policy, Q-function, value function, model). That generates the whole landscape. (ch08)\n\n### Policy optimization vs Q-learning — the central trade-off\n- **Policy optimization** is *principled: you directly optimize the thing you want.* Stable\n  and reliable. On-policy, so it cannot reuse data, so it is sample-hungry.\n- **Q-learning** only *indirectly* optimizes performance, by training `Q_theta` to satisfy a\n  self-consistency equation. Many failure modes, so less stable. But substantially more\n  sample efficient when it works, because it reuses everything.\n- **Satisfying the Bellman equations well carries no guarantee of good policy performance.**\n- The two are not exclusive — DDPG and SAC live between them deliberately. (ch08)\n\n### The policy gradient template\n`grad J = E[ sum_t grad log pi_theta(a_t|s_t) * Phi_t ]`. Five valid choices of `Phi_t`:\nfull return, reward-to-go, reward-to-go minus a baseline, `Q^pi`, and `A^pi`. All share an\nexpectation and differ in variance. Two rules get you from the first to the last:\n- **Don't let the past distract you** — drop rewards obtained before the action. Those terms\n  had zero mean and nonzero variance: pure noise.\n- **Baselines** — by the EGLP lemma, any state-only `b(s)` can be added or subtracted freely.\n  The standard choice is `V^pi(s_t)`, learned by MSE regression onto reward-to-go. (ch09)\n\n### The policy-gradient loss is not a loss function\nIts data distribution depends on the parameters, and it does not measure performance even in\nexpectation. Only at the current parameters, with data from those parameters, does it have\nthe negative gradient of performance. **You can send it to negative infinity while performance\ncraters, and it usually will. Only average return means anything.** (ch09)\n\n### Broken RL code almost always fails silently\nIt runs fine; the agent just never learns. Usually something is computed with the wrong\nequation, on the wrong distribution, or piped to the wrong place. **If it doesn't work, assume\nthere's a bug** before touching hyperparameters. Debug by measuring everything and reading the\ncode critically. The archetype is one missing `squeeze`: a `[N]` vs `[N,1]` shape mismatch is\nbroadcast-compatible, raises nothing, and silently turns the Bellman backup into an `[N,N]`\nmatrix. (ch10, ch12)\n\n### Learn by doing\nWrite your own implementations, **shortest correct version** of each, **simplest algorithms\nfirst**. VPG, DQN, A2C, PPO, DDPG, roughly in that order; ~250-300 lines each. Single-threaded\nbefore parallel. **Iterate fast in simple environments — under 5 minutes turnaround at the\ndebug stage.** Do not attempt Atari or Humanoid before the toy task works. Read papers for\ntheir ablations and supplementary material, but **do not overfit to paper details** (the\noriginal DDPG's architecture, init scheme and batch norm are not strictly necessary) **or to\nexisting implementations** (their abstractions serve reuse, not your single use case). (ch10)\n\n### Rigor: four standards\n1. **Fair comparisons** — tune the baseline as hard as your method; never handicap it.\n2. **Remove stochasticity as a confounder** — at least 3 seeds, 10 or more to be thorough.\n   Two seed groups can produce curves that look like different distributions.\n3. **High-integrity experiments** — launch fresh final runs and precommit to reporting them.\n   Tuning produces hypotheses; final runs produce conclusions.\n4. **Check each claim separately** — ablate every design decision. (ch10)\n\n### The safe-step family (on-policy)\nVPG takes an unconstrained gradient step, so a single bad step can collapse performance.\n**TRPO** constrains the step in **KL-divergence between policies, not distance in parameter\nspace**, then backtracking-line-searches until the exact constraint holds. **PPO** drops the\nconstraint and instead **clips the objective so the policy gains nothing by moving far**,\nwhich is first-order, far simpler, and empirically at least as good. (ch14, ch15, ch16)\n\n### The overestimation family (off-policy)\nDDPG amortizes the intractable continuous `max_a Q(s,a)` into a learned policy:\n`max_a Q(s,a) ~= Q(s, mu(s))`. It needs a **replay buffer** (licensed because the Bellman\nequation is indifferent to how data was collected) and **target networks** (because the target\notherwise depends on the parameters being trained). Its failure mode is **Q-value\noverestimation, which the policy actively exploits**. **TD3** answers with clipped double-Q,\ndelayed policy updates and target policy smoothing. **SAC** adds **entropy regularization**,\nmaking the explore-exploit trade-off an explicit coefficient `alpha`. (ch17, ch18, ch19)\n\n---\n\n## Chapter Index\n\n| # | Title | Key Frameworks |\n|---|-------|----------------|\n| [ch01](chapters\u002Fch01-introduction.md) | Introduction | The missing middle step, Code Design Philosophy |\n| [ch02](chapters\u002Fch02-installation.md) | Installation | Install-then-verify, MuJoCo optionality |\n| [ch03](chapters\u002Fch03-algorithm-lineup.md) | Algorithms: What's Included and Why | The two lineages, on\u002Foff-policy trade-off, code template |\n| [ch04](chapters\u002Fch04-running-experiments.md) | Running Experiments | One flag per kwarg, ExperimentGrid, save-dir suffixes |\n| [ch05](chapters\u002Fch05-experiment-outputs.md) | Experiment Outputs | Tools not files, watch-then-measure |\n| [ch06](chapters\u002Fch06-plotting-results.md) | Plotting Results | `Performance` alias, prefix autocompletion, seed averaging |\n| [ch07](chapters\u002Fch07-key-concepts-in-rl.md) | Part 1: Key Concepts in RL | MDPs, four value functions, Bellman equations, advantage |\n| [ch08](chapters\u002Fch08-kinds-of-rl-algorithms.md) | Part 2: Kinds of RL Algorithms | Taxonomy, model bias, policy-opt vs Q-learning |\n| [ch09](chapters\u002Fch09-intro-to-policy-optimization.md) | Part 3: Intro to Policy Optimization | Log-derivative trick, EGLP lemma, reward-to-go, baselines |\n| [ch10](chapters\u002Fch10-spinning-up-as-a-researcher.md) | Spinning Up as a Deep RL Researcher | Learn by doing, three idea frames, four rigor standards |\n| [ch11](chapters\u002Fch11-key-papers-in-deep-rl.md) | Key Papers in Deep RL | 13-section topic map |\n| [ch12](chapters\u002Fch12-exercises.md) | Exercises | Problem Set 1 and 2, the silent DDPG bug |\n| [ch13](chapters\u002Fch13-benchmarks.md) | Benchmarks | The parity disclosure, family-specific metrics |\n| [ch14](chapters\u002Fch14-vpg.md) | Vanilla Policy Gradient | The six-step loop |\n| [ch15](chapters\u002Fch15-trpo.md) | Trust Region Policy Optimization | KL trust region, line search, conjugate gradient |\n| [ch16](chapters\u002Fch16-ppo.md) | Proximal Policy Optimization | PPO-Clip, KL early stopping |\n| [ch17](chapters\u002Fch17-ddpg.md) | Deep Deterministic Policy Gradient | MSBE, replay buffers, target networks, polyak |\n| [ch18](chapters\u002Fch18-td3.md) | Twin Delayed DDPG | Clipped double-Q, delayed updates, target smoothing |\n| [ch19](chapters\u002Fch19-sac.md) | Soft Actor-Critic | Entropy regularization, reparameterization, squashed Gaussian |\n| [ch20](chapters\u002Fch20-logger-and-utilities.md) | Logger, MPI Tools and Run Utils | EpochLogger pattern, MPI PyTorch order |\n\n## Topic Index\n\n- **Advantage function** ch07, ch09, ch14\n- **Baselines** ch09\n- **Bellman equations** ch07, ch17\n- **Benchmarks \u002F parity** ch13, ch01\n- **Clipped double-Q** ch18, ch19\n- **Continuous action spaces** ch07, ch17\n- **Debugging \u002F silent failure** ch10, ch12\n- **DDPG** ch17, ch03, ch08\n- **Entropy regularization** ch19\n- **Exploration vs exploitation** ch14, ch17, ch19\n- **GAE** ch09, ch14\n- **Installation** ch02\n- **KL divergence \u002F trust region** ch15, ch16\n- **Logging** ch20, ch05\n- **MDPs** ch07\n- **Model-based RL** ch08\n- **MPI \u002F parallelization** ch20, ch02, ch04\n- **MSBE** ch17\n- **Off-policy** ch03, ch08, ch17\n- **On-policy** ch03, ch08, ch14\n- **Papers \u002F literature** ch11, ch10\n- **Plotting** ch06, ch13\n- **Policies (categorical, Gaussian, squashed)** ch07, ch19\n- **Policy gradient derivation** ch09\n- **PPO** ch16, ch03\n- **Q-learning** ch08, ch07\n- **Replay buffer** ch17\n- **Reparameterization trick** ch19, ch10\n- **Research process \u002F rigor** ch10, ch13\n- **Reward-to-go** ch09\n- **Running experiments** ch04, ch05\n- **SAC** ch19, ch03, ch08\n- **Seeds \u002F variance** ch10, ch13, ch04\n- **Target networks \u002F polyak** ch17, ch18\n- **TD3** ch18, ch12\n- **TRPO** ch15, ch03\n- **Value functions** ch07, ch09\n- **VPG** ch14, ch09\n\n## Supporting Files\n\n- [glossary.md](glossary.md) · [patterns.md](patterns.md) · [cheatsheet.md](cheatsheet.md)\n\n## Scope & Limits\n\nCovers the Spinning Up documentation only, as of the January 2020 PyTorch update. It does\n**not** cover: DQN and the discrete-action value-learning family (referenced, never\nimplemented here), recurrent or convolutional architectures, partially-observed settings,\nmodel-based implementations, exploration\u002Fmeta-RL\u002Fhierarchy beyond ch11's reading list, or any\ndeep RL work after early 2020. The six implementations are educational; ch13 says which are\nresearch-grade. For topics beyond this source, I say so rather than improvising.\n\n---\n\n*Compiled from OpenAI's Spinning Up in Deep RL documentation (MIT, Copyright (c) 2018 OpenAI),\nprimarily developed by Joshua Achiam. Structured study notes, not a reproduction of the source.*\n",{"name":5,"description":7},"main","7e219d4f922f94392cb65883ba910846c3c2e71b","engineering\u002Fspinning-up-deep-rl\u002Fskills\u002Fspinning-up-deep-rl",0,"https:\u002F\u002Fgithub.com\u002Fdevlify-marketplace\u002Fdevlify-skills","https:\u002F\u002Fgithub.com\u002Fdevlify-marketplace","devlify.com.ng","Software Developer & full-stack dev building secure web and mobile applications, AI\u002FML solutions, and scalable digital products. Founder of Devlify ","2026-08-30 00:40:50",{"source":8,"skillName":5,"commands":34,"prompt":44,"zipAvailable":45},[35,38,41],{"tool":36,"command":37},"npx","npx skills add devlify-marketplace\u002Fdevlify-skills --skill spinning-up-deep-rl",{"tool":39,"command":40},"bunx","bunx skills add devlify-marketplace\u002Fdevlify-skills --skill spinning-up-deep-rl",{"tool":42,"command":43},"pnpm","pnpm dlx skills add devlify-marketplace\u002Fdevlify-skills --skill spinning-up-deep-rl","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fdevlify-marketplace\u002Fdevlify-skills\u002Fblob\u002Fmain\u002Fengineering\u002Fspinning-up-deep-rl\u002Fskills\u002Fspinning-up-deep-rl\u002FSKILL.md",true,[47,53,57,61,65,69,73,77,81,85,89,93,97,101,105,109,113,117,121,125,129,133,137],{"path":48,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":51,"sourceHtmlUrl":52},"chapters\u002Fch01-introduction.md","REFERENCE","参考文档",6145,"https:\u002F\u002Fgithub.com\u002Fdevlify-marketplace\u002Fdevlify-skills\u002Fblob\u002Fmain\u002Fengineering\u002Fspinning-up-deep-rl\u002Fskills\u002Fspinning-up-deep-rl\u002Fchapters\u002Fch01-introduction.md",{"path":54,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":55,"sourceHtmlUrl":56},"chapters\u002Fch02-installation.md",3875,"https:\u002F\u002Fgithub.com\u002Fdevlify-marketplace\u002Fdevlify-skills\u002Fblob\u002Fmain\u002Fengineering\u002Fspinning-up-deep-rl\u002Fskills\u002Fspinning-up-deep-rl\u002Fchapters\u002Fch02-installation.md",{"path":58,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":59,"sourceHtmlUrl":60},"chapters\u002Fch03-algorithm-lineup.md",5795,"https:\u002F\u002Fgithub.com\u002Fdevlify-marketplace\u002Fdevlify-skills\u002Fblob\u002Fmain\u002Fengineering\u002Fspinning-up-deep-rl\u002Fskills\u002Fspinning-up-deep-rl\u002Fchapters\u002Fch03-algorithm-lineup.md",{"path":62,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":63,"sourceHtmlUrl":64},"chapters\u002Fch04-running-experiments.md",6184,"https:\u002F\u002Fgithub.com\u002Fdevlify-marketplace\u002Fdevlify-skills\u002Fblob\u002Fmain\u002Fengineering\u002Fspinning-up-deep-rl\u002Fskills\u002Fspinning-up-deep-rl\u002Fchapters\u002Fch04-running-experiments.md",{"path":66,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":67,"sourceHtmlUrl":68},"chapters\u002Fch05-experiment-outputs.md",4588,"https:\u002F\u002Fgithub.com\u002Fdevlify-marketplace\u002Fdevlify-skills\u002Fblob\u002Fmain\u002Fengineering\u002Fspinning-up-deep-rl\u002Fskills\u002Fspinning-up-deep-rl\u002Fchapters\u002Fch05-experiment-outputs.md",{"path":70,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":71,"sourceHtmlUrl":72},"chapters\u002Fch06-plotting-results.md",3928,"https:\u002F\u002Fgithub.com\u002Fdevlify-marketplace\u002Fdevlify-skills\u002Fblob\u002Fmain\u002Fengineering\u002Fspinning-up-deep-rl\u002Fskills\u002Fspinning-up-deep-rl\u002Fchapters\u002Fch06-plotting-results.md",{"path":74,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":75,"sourceHtmlUrl":76},"chapters\u002Fch07-key-concepts-in-rl.md",8029,"https:\u002F\u002Fgithub.com\u002Fdevlify-marketplace\u002Fdevlify-skills\u002Fblob\u002Fmain\u002Fengineering\u002Fspinning-up-deep-rl\u002Fskills\u002Fspinning-up-deep-rl\u002Fchapters\u002Fch07-key-concepts-in-rl.md",{"path":78,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":79,"sourceHtmlUrl":80},"chapters\u002Fch08-kinds-of-rl-algorithms.md",8219,"https:\u002F\u002Fgithub.com\u002Fdevlify-marketplace\u002Fdevlify-skills\u002Fblob\u002Fmain\u002Fengineering\u002Fspinning-up-deep-rl\u002Fskills\u002Fspinning-up-deep-rl\u002Fchapters\u002Fch08-kinds-of-rl-algorithms.md",{"path":82,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":83,"sourceHtmlUrl":84},"chapters\u002Fch09-intro-to-policy-optimization.md",8712,"https:\u002F\u002Fgithub.com\u002Fdevlify-marketplace\u002Fdevlify-skills\u002Fblob\u002Fmain\u002Fengineering\u002Fspinning-up-deep-rl\u002Fskills\u002Fspinning-up-deep-rl\u002Fchapters\u002Fch09-intro-to-policy-optimization.md",{"path":86,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":87,"sourceHtmlUrl":88},"chapters\u002Fch10-spinning-up-as-a-researcher.md",10392,"https:\u002F\u002Fgithub.com\u002Fdevlify-marketplace\u002Fdevlify-skills\u002Fblob\u002Fmain\u002Fengineering\u002Fspinning-up-deep-rl\u002Fskills\u002Fspinning-up-deep-rl\u002Fchapters\u002Fch10-spinning-up-as-a-researcher.md",{"path":90,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":91,"sourceHtmlUrl":92},"chapters\u002Fch11-key-papers-in-deep-rl.md",4608,"https:\u002F\u002Fgithub.com\u002Fdevlify-marketplace\u002Fdevlify-skills\u002Fblob\u002Fmain\u002Fengineering\u002Fspinning-up-deep-rl\u002Fskills\u002Fspinning-up-deep-rl\u002Fchapters\u002Fch11-key-papers-in-deep-rl.md",{"path":94,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":95,"sourceHtmlUrl":96},"chapters\u002Fch12-exercises.md",5528,"https:\u002F\u002Fgithub.com\u002Fdevlify-marketplace\u002Fdevlify-skills\u002Fblob\u002Fmain\u002Fengineering\u002Fspinning-up-deep-rl\u002Fskills\u002Fspinning-up-deep-rl\u002Fchapters\u002Fch12-exercises.md",{"path":98,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":99,"sourceHtmlUrl":100},"chapters\u002Fch13-benchmarks.md",4986,"https:\u002F\u002Fgithub.com\u002Fdevlify-marketplace\u002Fdevlify-skills\u002Fblob\u002Fmain\u002Fengineering\u002Fspinning-up-deep-rl\u002Fskills\u002Fspinning-up-deep-rl\u002Fchapters\u002Fch13-benchmarks.md",{"path":102,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":103,"sourceHtmlUrl":104},"chapters\u002Fch14-vpg.md",4968,"https:\u002F\u002Fgithub.com\u002Fdevlify-marketplace\u002Fdevlify-skills\u002Fblob\u002Fmain\u002Fengineering\u002Fspinning-up-deep-rl\u002Fskills\u002Fspinning-up-deep-rl\u002Fchapters\u002Fch14-vpg.md",{"path":106,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":107,"sourceHtmlUrl":108},"chapters\u002Fch15-trpo.md",6053,"https:\u002F\u002Fgithub.com\u002Fdevlify-marketplace\u002Fdevlify-skills\u002Fblob\u002Fmain\u002Fengineering\u002Fspinning-up-deep-rl\u002Fskills\u002Fspinning-up-deep-rl\u002Fchapters\u002Fch15-trpo.md",{"path":110,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":111,"sourceHtmlUrl":112},"chapters\u002Fch16-ppo.md",7002,"https:\u002F\u002Fgithub.com\u002Fdevlify-marketplace\u002Fdevlify-skills\u002Fblob\u002Fmain\u002Fengineering\u002Fspinning-up-deep-rl\u002Fskills\u002Fspinning-up-deep-rl\u002Fchapters\u002Fch16-ppo.md",{"path":114,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":115,"sourceHtmlUrl":116},"chapters\u002Fch17-ddpg.md",7541,"https:\u002F\u002Fgithub.com\u002Fdevlify-marketplace\u002Fdevlify-skills\u002Fblob\u002Fmain\u002Fengineering\u002Fspinning-up-deep-rl\u002Fskills\u002Fspinning-up-deep-rl\u002Fchapters\u002Fch17-ddpg.md",{"path":118,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":119,"sourceHtmlUrl":120},"chapters\u002Fch18-td3.md",6037,"https:\u002F\u002Fgithub.com\u002Fdevlify-marketplace\u002Fdevlify-skills\u002Fblob\u002Fmain\u002Fengineering\u002Fspinning-up-deep-rl\u002Fskills\u002Fspinning-up-deep-rl\u002Fchapters\u002Fch18-td3.md",{"path":122,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":123,"sourceHtmlUrl":124},"chapters\u002Fch19-sac.md",7852,"https:\u002F\u002Fgithub.com\u002Fdevlify-marketplace\u002Fdevlify-skills\u002Fblob\u002Fmain\u002Fengineering\u002Fspinning-up-deep-rl\u002Fskills\u002Fspinning-up-deep-rl\u002Fchapters\u002Fch19-sac.md",{"path":126,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":127,"sourceHtmlUrl":128},"chapters\u002Fch20-logger-and-utilities.md",4807,"https:\u002F\u002Fgithub.com\u002Fdevlify-marketplace\u002Fdevlify-skills\u002Fblob\u002Fmain\u002Fengineering\u002Fspinning-up-deep-rl\u002Fskills\u002Fspinning-up-deep-rl\u002Fchapters\u002Fch20-logger-and-utilities.md",{"path":130,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":131,"sourceHtmlUrl":132},"cheatsheet.md",5010,"https:\u002F\u002Fgithub.com\u002Fdevlify-marketplace\u002Fdevlify-skills\u002Fblob\u002Fmain\u002Fengineering\u002Fspinning-up-deep-rl\u002Fskills\u002Fspinning-up-deep-rl\u002Fcheatsheet.md",{"path":134,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":135,"sourceHtmlUrl":136},"glossary.md",5446,"https:\u002F\u002Fgithub.com\u002Fdevlify-marketplace\u002Fdevlify-skills\u002Fblob\u002Fmain\u002Fengineering\u002Fspinning-up-deep-rl\u002Fskills\u002Fspinning-up-deep-rl\u002Fglossary.md",{"path":138,"resourceType":49,"resourceTypeDesc":50,"sizeBytes":139,"sourceHtmlUrl":140},"patterns.md",7927,"https:\u002F\u002Fgithub.com\u002Fdevlify-marketplace\u002Fdevlify-skills\u002Fblob\u002Fmain\u002Fengineering\u002Fspinning-up-deep-rl\u002Fskills\u002Fspinning-up-deep-rl\u002Fpatterns.md",[142,149,156,163,170,177],{"skillNo":143,"name":144,"nameZh":6,"description":145,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":146,"categories":147,"keywords":148,"publishedAt":19},"01M7BRV5Q3BJT22HG25WJFTKZN","syllabus","Generates a curated supplementary reading list from any course syllabus using Consensus academic search. Grill-me intake (syllabus input format + course audience + year range) plus a grouping forcing-options checkpoint before any search runs — so the reading list matches the course's level and recency need. Parses the syllabus to extract topics and learning outcomes, searches Consensus for recent peer-reviewed papers per topic, and produces a professionally formatted .docx with clickable Consensus links, plain-language summaries calibrated to audience level, and Bloom-higher-order discussion questions tied to course learning goals. Use when the user uploads a syllabus, course outline, or curriculum document and wants supplementary readings (e.g., 'create a reading list from this syllabus', 'find recent papers for my course') — even casual mentions with a syllabus attached should trigger this skill.","https:\u002F\u002Fgithub.com\u002Fdevlify-marketplace\u002Fdevlify-skills\u002Fblob\u002Fmain\u002Fresearch\u002Fsyllabus\u002Fskills\u002Fsyllabus\u002FSKILL.md",[],[],{"skillNo":150,"name":151,"nameZh":6,"description":152,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":153,"categories":154,"keywords":155,"publishedAt":19},"01M7BRTW6RRTHXT6SBE41AY5GD","research","Default entry point for any research request — a hybrid router that classifies the question deterministically and either delegates to a specialist research skill (pulse for trends\u002Fsentiment, grants for NIH funding, litreview for academic literature, syllabus for course reading, patent for prior-art + IP landscape, dossier for entity research, deepread for evidence-first reading of supplied documents) or runs its own plan-decompose-multi-source-search-synthesize-cite fallback workflow when no specialist matches. Always surfaces the routing decision so users can override. Use when the user makes any research request that doesn't obviously match a more-specific specialist skill (e.g., \"research [topic]\", \"look into [topic]\", \"what do we know about [topic]\", \"investigate [topic]\", \"find me information on [topic]\", \"do some research on [topic]\", \"I need to understand [topic]\"). Output is a markdown briefing (default) or .docx document (on request) with full citations and an audit log.","https:\u002F\u002Fgithub.com\u002Fdevlify-marketplace\u002Fdevlify-skills\u002Fblob\u002Fmain\u002Fresearch\u002Fresearch\u002Fskills\u002Fresearch\u002FSKILL.md",[],[],{"skillNo":157,"name":158,"nameZh":6,"description":159,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":160,"categories":161,"keywords":162,"publishedAt":19},"01M7BRTMK3PGPNH76R2HY57AKC","pulse","Multi-source recency research skill that takes the pulse of any topic across Reddit, Hacker News, the open web, and optionally X\u002FTwitter within a configurable recent window (default 30 days). Forcing intake clarifies topic specificity, angle (trend\u002Fsentiment\u002Fproblems\u002Fopportunities\u002Fcomparison), time window, and platform scope before searching. Returns a synthesized briefing with citations, engagement metrics, and cross-platform pattern analysis. Use when the user requests multi-source recency intelligence on a topic (e.g., 'pulse on [topic]', 'what's happening with [topic]', 'what are people saying about [topic]', 'current conversation about [topic]', 'take the pulse of [topic]', 'trending: [topic]', 'find me info on [topic]'), and for competitor research, trend discovery, tool comparisons, and audience sentiment analysis.","https:\u002F\u002Fgithub.com\u002Fdevlify-marketplace\u002Fdevlify-skills\u002Fblob\u002Fmain\u002Fresearch\u002Fpulse\u002Fskills\u002Fpulse\u002FSKILL.md",[],[],{"skillNo":164,"name":165,"nameZh":6,"description":166,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":167,"categories":168,"keywords":169,"publishedAt":19},"01M7BRTAR0YJWNZ2AP3QTH39BW","patent","Patent prior-art and landscape intelligence skill — not generic patent help. Commits to one of five sub-use-cases via forcing intake (novelty search \u002F freedom-to-operate \u002F competitive landscape \u002F acquisition diligence \u002F litigation prior-art) before any search runs. Searches Google Patents, Espacenet, USPTO, and optionally Lens.org for citation-graph signals. Output is an editable Word document (.docx) with verdict, ranked closest art (claim-text extracted), CPC-class-aware landscape, family-resolved hits, geographic coverage, FTO flags where applicable, strategy recommendations, and full audit log. Use when the user asks for patent searching or analysis (e.g., 'prior art search for [invention]', 'freedom to operate analysis for [product]'). Produces search signal, not legal advice — always recommends consulting a patent attorney before filing or licensing decisions. Trademark, copyright, and trade-secret questions are out of scope.","https:\u002F\u002Fgithub.com\u002Fdevlify-marketplace\u002Fdevlify-skills\u002Fblob\u002Fmain\u002Fresearch\u002Fpatent\u002Fskills\u002Fpatent\u002FSKILL.md",[],[],{"skillNo":171,"name":172,"nameZh":6,"description":173,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":174,"categories":175,"keywords":176,"publishedAt":19},"01M7BRT3J0V5A9V3VJ96G97AGW","notebooklm","Browser automation skill for controlling Google's NotebookLM. Use when the user wants anything done in NotebookLM (e.g., 'open NotebookLM', 'check my [name] notebook', 'ask my notebook about X', 'add [source] to NotebookLM', 'generate a Video Overview from my notebook', 'use NotebookLM Studio'). Handles reading and querying notebooks, adding sources (URLs, text, files, YouTube links, synthesized content), generating Studio outputs (Audio\u002FVideo Overviews, Mind Maps, Reports incl. Briefing Doc\u002FStudy Guide\u002FFAQ, Flashcards, Quiz, slide decks, infographics — discover the exact set from the live Studio panel; the UI evolves fast), and creating new notebooks. Requires browser automation environment — fails gracefully when unavailable.","https:\u002F\u002Fgithub.com\u002Fdevlify-marketplace\u002Fdevlify-skills\u002Fblob\u002Fmain\u002Fresearch\u002Fnotebooklm\u002Fskills\u002Fnotebooklm\u002FSKILL.md",[],[],{"skillNo":178,"name":179,"nameZh":6,"description":180,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":181,"categories":182,"keywords":183,"publishedAt":19},"01M7BRSS7P0CJP7X9ERDFYN6GC","litreview","Academic literature orientation skill that searches papers via free keyless APIs (PubMed E-utilities + OpenAlex) by default — with the Consensus MCP as an optional enhancement lane when connected — builds a strategic search plan using PICO (default) or SPIDER \u002F Decomposition \u002F hybrid as fallbacks, and synthesizes findings into a formatted Word (.docx) research guide. Grill-me intake (research question specificity + framework hint + tentative depth) before the recon search; a second forcing checkpoint after Phase 2 confirms framework + sub-areas + depth before searches consume budget. Configurable depth (5\u002F10\u002F20 queries) controls coverage vs. speed. Output is a 'launching pad' — an orientation guide that lets a researcher dive in confidently, not a finished review. Use when the user starts literature-oriented research (e.g., 'litreview on [topic]', 'literature review on [topic]', 'I'm starting a literature review on X', 'I'm writing a paper on X', 'help me research X', 'I'm doing research on X', 'can you help me research X'). Do NOT use for single one-off paper searches wanting a quick list — that's a plain PubMed\u002FOpenAlex (or Consensus) query.","https:\u002F\u002Fgithub.com\u002Fdevlify-marketplace\u002Fdevlify-skills\u002Fblob\u002Fmain\u002Fresearch\u002Flitreview\u002Fskills\u002Flitreview\u002FSKILL.md",[],[]]