pipeline-authoring
Build, configure, run, schedule, debug, and validate ZenML step and pipeline workflows, artifacts, materializers, settings, hooks, wait/resume, and local execution patterns.
- License 许可
- Apache-2.0
- 最近更新
- 2026-09-03
- Stars
- ★ 351
- Forks
- 27
01 前言 Front Matter
- name
- pipeline-authoring
- license
- Apache 2.0
- metadata
- { "disco-role": "operating" }
- description
- Build, configure, run, schedule, debug, and validate ZenML step and pipeline workflows, artifacts, materializers, settings, hooks, wait/resume, and local execution patterns.
- disable-model-invocation
- true
02 资源文件 5
- 参考文档references/api-reference.md15.0 KB
- 参考文档references/troubleshooting.md14.9 KB
- 参考文档references/workflows.md15.3 KB
- 脚本scripts/inspect_pipeline_api.py4.4 KB
- 脚本scripts/pipeline_smoke.py5.3 KB
03 SKILL.md 正文
Pipeline Authoring
Use this sub-skill when the task is to write or debug ZenML user code built with @step, @pipeline, artifacts, materializers, pipeline settings, schedules, caching, retries, hooks, dynamic pipelines, or local execution.
Natural Triggers
- User asks for a ZenML pipeline, workflow, DAG, step, artifact, materializer, cache, retry, hook, schedule, dynamic pipeline, wait/resume flow, or local run.
- User has Python functions and wants them wrapped in
@stepor composed with@pipeline. - User needs typed multi-output artifacts, custom output materializers,
DockerSettings,ResourceSettings, or YAML/runtime configuration. - User is debugging cache misses, skipped hooks, invalid materializer output keys, dynamic
.load()/.chunk()/.map()behavior, or duplicate child pipeline runs.
Route First
- Read API reference for decorator signatures, settings keys, output naming, materializer contracts, schedules, hooks, retries, and dynamic APIs.
- Read Workflow recipes for concrete static, dynamic, scheduled, materialized, hook-enabled, and isolated-local patterns.
- Read Troubleshooting before changing cache/retry/hook behavior, dynamic child pipeline structure, Docker/resource settings, or local ZenML configuration.
- Run inspect_pipeline_api.py when the installed ZenML version may differ from this skill, or before relying on a signature.
- Run pipeline_smoke.py with
--check-importsfor a safe import probe, or with--runfor an opt-in temporary local pipeline smoke.
Boundary Rules
- Own user-facing pipeline code: decorators, step signatures, dataflow, artifact typing, materializer selection, settings dictionaries, caching, retries, hooks, dynamic execution, wait/resume, schedules, and local smoke checks.
- Route stack component registration, orchestrator implementation, step operator implementation, integration extras, and component flavor code to stacks-and-integrations.
- Route CLI/client resource listing, schedule lifecycle commands, run audit scripts, and remote-server queries to cli-and-client.
- Route FastAPI server, store, trigger model, migration, and RBAC internals to server-and-stores.
- Route deployment services, model/agent deployment examples, production serving, and credential-heavy operational examples to deployments-and-agents.
- Route repository test selection, formatting, linting, docs maintenance, and CI-equivalent checks to maintenance.
Working Pattern
- Start with ordinary typed Python functions, make them importable at module level, then add
@stepand@pipelineonly where orchestration is needed. - Prefer
with_options(...)for per-run or per-invocation changes; reserveconfigure(...)for deliberate in-place global mutation of a step or pipeline object. - Name multiple outputs with
typing.Annotatedbefore assigning per-output materializers or downstream references. - Keep parameters JSON-serializable; pass complex datasets, models, directories, and custom objects as artifacts with explicit type annotations or materializers.
- For dynamic pipelines, use
.load()only for Python control flow decisions and.chunk(index=...),.map(...),.product(...),unmapped(...), or futures to wire DAG dependencies. - Treat cache, retry, and hook behavior together: cache hits skip step execution and step hooks, retries fire attempt hooks repeatedly, and hook failures are recorded without aborting the run.
Safe Validation
python scripts/inspect_pipeline_api.py --check-importsverifies that the expected public APIs import from the active environment.python scripts/inspect_pipeline_api.py --jsonprints decorator/settings/materializer signatures for the active ZenML installation.python scripts/pipeline_smoke.py --check-importschecks importability without creating a ZenML repository or run.python scripts/pipeline_smoke.py --runcreates a temporary isolated ZenML config and local repository, runs a tiny local pipeline, and cleans up by default.
04 同仓库更多 Skills
passnet-triton-opt
Make a single PassNet Triton kernel fast AND numerically faithful: performance model (when a replacement can win at all), block/grid/warp tuning, autotune policy, launch overhead, and per-op numeric recipes to pass the dtype baseline tolerances. Use when a pass matches and is correct but speedup below expected, or when correctness fails by small numeric margins.
passnet-solve
END-TO-END playbook for solving one PassNet sample: analyze the computation graphs, decide the optimization strategy (what to fuse, what to replace, what to leave alone), drive the iteration loop, and maximize the sample score. This is the ENTRY skill — invoke it first for any PassNet optimization task; it tells you when to use passnet-pattern-fusion, passnet-triton-opt and passnet-feedback.
passnet-skill
PassNet GPU kernel optimization via compiler passes. Design and implement Triton-based optimization passes, create pass files under ./pass_dir/, self-evaluate with pass_evaluator, and iterate to maximize GPU speedup.
passnet-pattern-fusion
Author PassNet pass files: write patterns that actually MATCH the FX graph, pick fusion regions (which ops to absorb into one Triton kernel), and structure multi-pass submissions with the mandatory shared-dispatch architecture. Use when creating or fixing pattern()/replacement_args()/replacement_func() files, when a pass "failed to match", or when deciding how to fuse multiple kernels/ops.
passnet-orchestrate
Round/batch planning and decision support for PassNet graph optimization. Use for multi-sample triage, worker allocation, eval-budget policy, and post-eval keep/revert decisions. For solving one concrete sample end to end, use passnet-solve as the entry skill; consult this skill only when you need broader planning or a specific decision gate.
passnet-feedback
Fast, deterministic feedback for PassNet work: pre-flight pattern/match verification WITHOUT burning a GPU evaluation, per-node bottleneck analysis of a sample's graphs, and eval-log parsing into per-variant status + estimated score + failure classification. Use BEFORE every GPU evaluation (check_pattern), at the START of a sample (analyze_graph), and AFTER every evaluation (parse_eval_log).