core-optimization
Compose Optax optimizers, gradient transformations, wrappers, and update loops.
- License 许可
- Apache-2.0
- 最近更新
- 2026-09-03
- Stars
- ★ 351
- Forks
- 27
01 前言 Front Matter
- name
- core-optimization
- license
- Apache 2.0
- metadata
- { "disco-role": "operating" }
- description
- Compose Optax optimizers, gradient transformations, wrappers, and update loops.
- disable-model-invocation
- true
03 SKILL.md 正文
Core Optimization
Use this sub-skill when the user wants to choose, compose, or debug an Optax optimizer pipeline. This is the main route for ordinary gradient-transformation workflows.
Include here
- Base optimizers such as
adam,adamw,sgd,lamb,lion,rmsprop,lbfgs,adafactor, and related aliases. - Gradient transformations such as clipping, weight decay, momentum/trace-style updates, adaptive scaling, and schedule-driven scaling.
- Composition helpers:
chain,named_chain,partition, and parameter masking or freezing workflows. - Wrapper-style behaviors such as
lookahead,MultiSteps,apply_if_finite,skip_large_updates, andskip_not_finite. - The update loop itself:
init,update, andapply_updates.
Exclude or route elsewhere
- Loss-function selection and schedule families that are being chosen for their own sake: use
losses-and-schedules. - Projections, assignment, tree utilities, linear algebra helpers, second-order helpers, and contrib/experimental algorithms: use
advanced-topics. - Anything that is really a metric, objective, or data-splitting question rather than an optimizer question.
Good questions for this route
- “Which Optax optimizer should I start with?”
- “How do I chain clipping with AdamW?”
- “How do I freeze some parameters but not others?”
- “How do I apply updates safely?”
- “How do I inject a schedule into an optimizer hyperparameter?”
Read first
../../references/core-workflows.mdfor the canonical update loop, composition patterns, and schedule injection.../../references/troubleshooting.mdfor shape mismatch, state reuse, and backend confusion.../../scripts/optax_skill_doctor.pywhen a quick import-and-smoke check is useful.
Core workflow
- Choose a base optimizer or low-level transform.
- Add wrappers or composition helpers if the user needs clipping, freezing, masking, or delayed updates.
- Decide whether schedules should be part of the optimizer hyperparameters or a separate training-loop concern.
- Initialize the optimizer state from the parameter tree.
- Update with gradients, then apply the returned updates back to the parameter tree.
Signals that this route is correct
- The user is thinking about parameters, gradients, optimizer state, or
apply_updates. - The user wants to compare two optimizers or learn how to combine them.
- The user is debugging a training loop and the issue is about the transformation pipeline rather than the loss value.
Common mistakes
- Reusing state from one optimizer pipeline with a different pipeline.
- Passing a parameter tree whose structure does not match the gradient tree.
- Forgetting that some transforms need the current parameters during
update(...). - Treating a learning-rate schedule as if it were a loss or data-processing problem.
Useful examples
../../references/examples-index.mdpoints to notebooks such asmlp_mnist.ipynb,flax_example.ipynb,lbfgs.ipynb, andlookahead_mnist.ipynb, which show optimizer pipelines in real training loops.
04 同仓库更多 Skills
passnet-triton-opt
Make a single PassNet Triton kernel fast AND numerically faithful: performance model (when a replacement can win at all), block/grid/warp tuning, autotune policy, launch overhead, and per-op numeric recipes to pass the dtype baseline tolerances. Use when a pass matches and is correct but speedup below expected, or when correctness fails by small numeric margins.
passnet-solve
END-TO-END playbook for solving one PassNet sample: analyze the computation graphs, decide the optimization strategy (what to fuse, what to replace, what to leave alone), drive the iteration loop, and maximize the sample score. This is the ENTRY skill — invoke it first for any PassNet optimization task; it tells you when to use passnet-pattern-fusion, passnet-triton-opt and passnet-feedback.
passnet-skill
PassNet GPU kernel optimization via compiler passes. Design and implement Triton-based optimization passes, create pass files under ./pass_dir/, self-evaluate with pass_evaluator, and iterate to maximize GPU speedup.
passnet-pattern-fusion
Author PassNet pass files: write patterns that actually MATCH the FX graph, pick fusion regions (which ops to absorb into one Triton kernel), and structure multi-pass submissions with the mandatory shared-dispatch architecture. Use when creating or fixing pattern()/replacement_args()/replacement_func() files, when a pass "failed to match", or when deciding how to fuse multiple kernels/ops.
passnet-orchestrate
Round/batch planning and decision support for PassNet graph optimization. Use for multi-sample triage, worker allocation, eval-budget policy, and post-eval keep/revert decisions. For solving one concrete sample end to end, use passnet-solve as the entry skill; consult this skill only when you need broader planning or a specific decision gate.
passnet-feedback
Fast, deterministic feedback for PassNet work: pre-flight pattern/match verification WITHOUT burning a GPU evaluation, per-node bottleneck analysis of a sample's graphs, and eval-log parsing into per-variant status + estimated score + failure classification. Use BEFORE every GPU evaluation (check_pattern), at the START of a sample (analyze_graph), and AFTER every evaluation (parse_eval_log).