distributed-tuning
Routes KerasTuner's chief/worker Oracle coordination over gRPC using the KERASTUNER_ORACLE_IP, KERASTUNER_ORACLE_PORT, and KERASTUNER_TUNER_ID environment contract.
- License 许可
- Apache-2.0
- 最近更新
- 2026-09-03
- Stars
- ★ 351
- Forks
- 27
01 前言 Front Matter
- id
- distributed-tuning
- name
- distributed-tuning
- license
- Apache 2.0
- metadata
- { "disco-role": "operating" }
- description
- Routes KerasTuner's chief/worker Oracle coordination over gRPC using the KERASTUNER_ORACLE_IP, KERASTUNER_ORACLE_PORT, and KERASTUNER_TUNER_ID environment contract.
- disable-model-invocation
- true
02 资源文件 4
- 参考文档references/api-reference.md6.5 KB
- 参考文档references/troubleshooting.md6.6 KB
- 参考文档references/workflows.md7.2 KB
- 脚本scripts/check_distributed_env.py2.7 KB
03 SKILL.md 正文
Distributed Tuning
Use this sub-skill when a KerasTuner search must coordinate multiple tuner
processes through a chief Oracle, or when a KERASTUNER_ORACLE_*/
KERASTUNER_TUNER_ID deployment is failing.
Scope
- Configure the three distributed Oracle environment variables as one unit.
- Assign chief and worker roles, launch the gRPC service, and diagnose network and shared-storage prerequisites.
- Understand OracleClient request, status, reporting, and shutdown behavior.
- Keep distributed Oracle coordination separate from the model/search-space design handled by the main KerasTuner workflow.
Do not use this route for a local single-process search, CloudOracle-managed
distribution, or general HyperParameters/tuner selection. In particular,
the Tuner public contract supports only single-worker distribution
strategies; Oracle-level parallel tuning over its chief/worker RPC protocol is
not a promise that a multi-worker distribution_strategy is supported.
Route in this order
- Read api-reference.md for the verified environment contract, role detection, RPC surface, and reporting semantics.
- From the skill root, run
python sub-skills/distributed-tuning/scripts/check_distributed_env.pybefore importing or constructing a distributed tuner. It only inspects environment variables; it never starts a server or opens a socket. - Follow workflows.md for the launch sequence, shared project directory, status handling, and orderly shutdown.
- Use troubleshooting.md for missing variables, accidental roles, connection stalls, filesystem divergence, and cleanup failures.
Non-negotiable launch contract
Set all of these together on every participating process:
KERASTUNER_ORACLE_IP: the address used both to bind the chief service and to reach it from workers.KERASTUNER_ORACLE_PORT: the same available TCP port for the chief service.KERASTUNER_TUNER_ID: a deliberate identifier, unique at the independent tuner/group coordination scope.
An ID containing the case-sensitive substring chief is treated as the chief
side. The chief starts OracleServicer; other IDs construct OracleClient and
proxy Oracle calls. A partial environment is invalid: KerasTuner raises
RuntimeError when the IP is set without the port or tuner ID. The bundled
checker is intentionally stricter and treats empty/whitespace values as
missing, including port/ID values supplied without an IP; do not bypass this
preflight because runtime role detection is key-presence based.
Use a reachable non-loopback address for different hosts, an open and reserved
port, the same directory/project_name on a shared writable filesystem, and
an environment where the grpcio runtime dependency is installed. The RPC
transport is insecure gRPC, so keep the address on a trusted network or behind
an appropriate network control; this protocol does not provide TLS.
Role and lifecycle guardrails
- Start the same tuner program with the chief environment on exactly one
process/group. Its
search()path saves the Oracle and starts theOracleServicerserver, which otherwise would not block the process. - Start workers with the same Oracle IP/port and non-chief IDs. Clients wait
for the chief service with
wait_for_ready=Truefor up to 60 minutes per RPC; this tolerates startup races but does not repair a wrong address, blocked port, or an absent chief. - Give independent tuner processes/groups distinct IDs. If a training backend groups processes into one multi-worker strategy, preserve that backend's required ID convention rather than inventing a second Oracle group.
- Keep the project directory shared and writable until all workers have
finished. Do not let multiple processes independently delete it with
overwrite=True. - Let workers observe
STOPPEDand exit their search loop; do not kill the chief before ongoing trials and registered tuner IDs have drained. The chief's server wait loop exits after its stop signal and drain condition.
The test helper keras_tuner.test_utils.mock_distribute is evidence for the
environment protocol only. It patches os.environ, creates threads, and
picks a port for tests; it is not a production launcher or runtime dependency.
Verification focus
A useful check must show: an entirely absent environment remains valid ordinary local mode, no partial environment is accepted as ordinary local mode, and a complete set is accepted without a socket or server; a chief ID is not confused with a worker ID; and all participants refer to one reachable endpoint and one shared project directory. Do not use the mock test helper as proof that a real cross-host network or filesystem is configured.
04 同仓库更多 Skills
passnet-triton-opt
Make a single PassNet Triton kernel fast AND numerically faithful: performance model (when a replacement can win at all), block/grid/warp tuning, autotune policy, launch overhead, and per-op numeric recipes to pass the dtype baseline tolerances. Use when a pass matches and is correct but speedup below expected, or when correctness fails by small numeric margins.
passnet-solve
END-TO-END playbook for solving one PassNet sample: analyze the computation graphs, decide the optimization strategy (what to fuse, what to replace, what to leave alone), drive the iteration loop, and maximize the sample score. This is the ENTRY skill — invoke it first for any PassNet optimization task; it tells you when to use passnet-pattern-fusion, passnet-triton-opt and passnet-feedback.
passnet-skill
PassNet GPU kernel optimization via compiler passes. Design and implement Triton-based optimization passes, create pass files under ./pass_dir/, self-evaluate with pass_evaluator, and iterate to maximize GPU speedup.
passnet-pattern-fusion
Author PassNet pass files: write patterns that actually MATCH the FX graph, pick fusion regions (which ops to absorb into one Triton kernel), and structure multi-pass submissions with the mandatory shared-dispatch architecture. Use when creating or fixing pattern()/replacement_args()/replacement_func() files, when a pass "failed to match", or when deciding how to fuse multiple kernels/ops.
passnet-orchestrate
Round/batch planning and decision support for PassNet graph optimization. Use for multi-sample triage, worker allocation, eval-budget policy, and post-eval keep/revert decisions. For solving one concrete sample end to end, use passnet-solve as the entry skill; consult this skill only when you need broader planning or a specific decision gate.
passnet-feedback
Fast, deterministic feedback for PassNet work: pre-flight pattern/match verification WITHOUT burning a GPU evaluation, per-node bottleneck analysis of a sample's graphs, and eval-log parsing into per-variant status + estimated score + failure classification. Use BEFORE every GPU evaluation (check_pattern), at the START of a sample (analyze_graph), and AFTER every evaluation (parse_eval_log).