dagster-architect
Architect-level Dagster design — software-defined assets over tasks, partitions and backfills, resources and IO managers, schedules and sensors, and the asset checks that make data quality part of orchestration. Use when orchestration should be modelled as data products.
- License 许可
- MIT
- 最近更新
- 2026-11-13
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- dagster-architect
- license
- MIT
- metadata
- { "crusor-path": "data-engineering-architect/scheduling-architect/dagster-architect", "crusor-domain": "data-engineering-architect", "crusor-origin": "original", "crusor-sources": "docs.dagster.io/concepts/assets/software-defined-assets; docs.dagster.io/guides/partitions; docs.dagster.io/concepts/assets/asset-checks", "crusor-triggers": "dagster, software defined asset, asset check, io manager, partitions dagster, sensors, dagster cloud, asset graph", "crusor-verified-on": "2026-10-30" }
- description
- Architect-level Dagster design — software-defined assets over tasks, partitions and backfills, resources and IO managers, schedules and sensors, and the asset checks that make data quality part of orchestration. Use when orchestration should be modelled as data products.
- compatibility
- Dagster, Dagster+
03 SKILL.md 正文
Dagster Architect
Dagster models data assets, not tasks. The orchestration question becomes "is this table fresh and correct", which is the question you actually have.
When to use
- The team thinks in tables/datasets rather than jobs.
- Quality checks should gate materialisation.
- Multiple environments and teams share a lineage.
When NOT to use
- A handful of cron jobs with no dependencies — Dagster is overhead.
- An existing Airflow estate with no pain — migration is expensive.
Decision 1 — Assets
- Define an asset per materialised table/partition; its dependencies are the upstream assets, not task ordering.
- Asset keys are stable identifiers; renaming is a migration for downstream consumers.
- Group by domain; keep the graph readable rather than clever.
- Ops for side effects (email, trigger) — do not disguise them as assets.
Decision 2 — Partitions
- Partition by day/hour/region/dynamic key per asset; backfills become a first-class operation.
- Partition mappings connect differently partitioned assets — the usual place for correctness bugs.
- Backfill in bounded chunks with concurrency limits; a 2-year backfill is not one run.
- Partitioned assets need idempotent materialisation (overwrite per partition).
Decision 3 — Resources and IO
- Resources carry connections/config (warehouse, S3, dbt) and are injected, not imported globally.
- IO managers decide where outputs live: warehouse tables, files, or nothing (in-place writes).
- Environment-specific config in YAML per deployment; no code branches for envs.
- Secrets from the platform's secret manager, never in
dagster.yaml.
Decision 4 — Scheduling
- Schedules for predictable cadence; sensors for event-driven (new file, upstream freshness, external API).
- Freshness policies declare the expected update cadence; violations surface as alerts, not silent staleness.
- Auto-materialisation policies for incremental chains — but verify they do not thrash.
Decision 5 — Checks
- Asset checks: row-count bands, null rates, uniqueness, freshness, business rules.
- Severity: error blocks downstream, warn records; do not make every check fatal.
- Checks run with the asset and appear in the UI next to lineage — that is the whole value.
- Alert routing per asset group to the owning team.
Failure modes at scale
- Task-shaped DAGs ported wholesale to Dagster; assets are just renames.
- Partition mapping wrong; a daily asset feeds an hourly one with stale data.
- Unbounded backfill saturates the warehouse during business hours.
- Checks defined but not wired to block downstream; bad data flows through.
- Resources configured per deployment by hand; staging points at production.
Verification receipts
- Asset graph for one domain, with grouping and owners.
- Backfill executed in bounded chunks with concurrency limits.
- Partition mapping verified between two differently partitioned assets.
- One asset check failing and blocking downstream, observed.
- Freshness policy configured with an alert on violation.
04 同仓库更多 Skills
interaction-patterns-architect
Architect-level interaction design in code — choosing established patterns over invented ones, feedback and latency behaviour, destructive-action safeguards, navigation and overlay conventions, and the state model behind every interactive surface. Use when a screen has to behave, not just look.
design-system-ops-architect
Architect-level operation of a design system in code — token pipelines and versioning, component API and variant policy, the catalogue as the contract, adoption metrics, and the governance that stops a system from forking into three. Use when a component library must survive several teams.
design-handoff-architect
Architect-level design handoff — what must be specified before implementation starts, tokens over raw values, responsive and content-limit contracts, state matrices, and the review loop that catches drift between design and code. Use when a design is about to become code.
content-microcopy-architect
Architect-level UI content — where strings live, voice and terminology consistency, error and empty-state copy as part of the interface, translation-ready authoring, and the content model a product needs before it multiplies screens. Use when the words in the interface matter as much as the pixels.
ui-ux-implementation-architect
Architect-level direction for turning a design into working UI — the handoff contract, component ownership versus screen-level layout, interaction and state conventions, design QA gates, and the review habits that keep an implementation faithful without frozen pixels. Use before any specific UI implementation skill.
test-strategy-architect
Architect-level test strategy — allocating assertions across unit, integration, contract and end-to-end layers, designing for testability through dependency boundaries, fixture and data strategy, and the flake and coverage policies that keep a suite trustworthy. Use when defining how a codebase is tested.