AGENT SKILL

dagster-architect

Architect-level Dagster design — software-defined assets over tasks, partitions and backfills, resources and IO managers, schedules and sensors, and the asset checks that make data quality part of orchestration. Use when orchestration should be modelled as data products.

License 许可
MIT
最近更新
2026-11-13
Stars
★ 0
Forks
0

01 前言 Front Matter

name
dagster-architect
license
MIT
metadata
{ "crusor-path": "data-engineering-architect/scheduling-architect/dagster-architect", "crusor-domain": "data-engineering-architect", "crusor-origin": "original", "crusor-sources": "docs.dagster.io/concepts/assets/software-defined-assets; docs.dagster.io/guides/partitions; docs.dagster.io/concepts/assets/asset-checks", "crusor-triggers": "dagster, software defined asset, asset check, io manager, partitions dagster, sensors, dagster cloud, asset graph", "crusor-verified-on": "2026-10-30" }
description
Architect-level Dagster design — software-defined assets over tasks, partitions and backfills, resources and IO managers, schedules and sensors, and the asset checks that make data quality part of orchestration. Use when orchestration should be modelled as data products.
compatibility
Dagster, Dagster+

03 SKILL.md 正文

Dagster Architect

Dagster models data assets, not tasks. The orchestration question becomes "is this table fresh and correct", which is the question you actually have.

When to use

  • The team thinks in tables/datasets rather than jobs.
  • Quality checks should gate materialisation.
  • Multiple environments and teams share a lineage.

When NOT to use

  • A handful of cron jobs with no dependencies — Dagster is overhead.
  • An existing Airflow estate with no pain — migration is expensive.

Decision 1 — Assets

  • Define an asset per materialised table/partition; its dependencies are the upstream assets, not task ordering.
  • Asset keys are stable identifiers; renaming is a migration for downstream consumers.
  • Group by domain; keep the graph readable rather than clever.
  • Ops for side effects (email, trigger) — do not disguise them as assets.

Decision 2 — Partitions

  • Partition by day/hour/region/dynamic key per asset; backfills become a first-class operation.
  • Partition mappings connect differently partitioned assets — the usual place for correctness bugs.
  • Backfill in bounded chunks with concurrency limits; a 2-year backfill is not one run.
  • Partitioned assets need idempotent materialisation (overwrite per partition).

Decision 3 — Resources and IO

  • Resources carry connections/config (warehouse, S3, dbt) and are injected, not imported globally.
  • IO managers decide where outputs live: warehouse tables, files, or nothing (in-place writes).
  • Environment-specific config in YAML per deployment; no code branches for envs.
  • Secrets from the platform's secret manager, never in dagster.yaml.

Decision 4 — Scheduling

  • Schedules for predictable cadence; sensors for event-driven (new file, upstream freshness, external API).
  • Freshness policies declare the expected update cadence; violations surface as alerts, not silent staleness.
  • Auto-materialisation policies for incremental chains — but verify they do not thrash.

Decision 5 — Checks

  • Asset checks: row-count bands, null rates, uniqueness, freshness, business rules.
  • Severity: error blocks downstream, warn records; do not make every check fatal.
  • Checks run with the asset and appear in the UI next to lineage — that is the whole value.
  • Alert routing per asset group to the owning team.

Failure modes at scale

  • Task-shaped DAGs ported wholesale to Dagster; assets are just renames.
  • Partition mapping wrong; a daily asset feeds an hourly one with stale data.
  • Unbounded backfill saturates the warehouse during business hours.
  • Checks defined but not wired to block downstream; bad data flows through.
  • Resources configured per deployment by hand; staging points at production.

Verification receipts

  1. Asset graph for one domain, with grouping and owners.
  2. Backfill executed in bounded chunks with concurrency limits.
  3. Partition mapping verified between two differently partitioned assets.
  4. One asset check failing and blocking downstream, observed.
  5. Freshness policy configured with an alert on violation.

04 同仓库更多 Skills

SriAkhilSJ
SriAkhilSJ/PulseAIRepo

interaction-patterns-architect

Architect-level interaction design in code — choosing established patterns over invented ones, feedback and latency behaviour, destructive-action safeguards, navigation and overlay conventions, and the state model behind every interactive surface. Use when a screen has to behave, not just look.

★ 0
SriAkhilSJ
SriAkhilSJ/PulseAIRepo

design-system-ops-architect

Architect-level operation of a design system in code — token pipelines and versioning, component API and variant policy, the catalogue as the contract, adoption metrics, and the governance that stops a system from forking into three. Use when a component library must survive several teams.

★ 0
SriAkhilSJ
SriAkhilSJ/PulseAIRepo

design-handoff-architect

Architect-level design handoff — what must be specified before implementation starts, tokens over raw values, responsive and content-limit contracts, state matrices, and the review loop that catches drift between design and code. Use when a design is about to become code.

★ 0
SriAkhilSJ
SriAkhilSJ/PulseAIRepo

content-microcopy-architect

Architect-level UI content — where strings live, voice and terminology consistency, error and empty-state copy as part of the interface, translation-ready authoring, and the content model a product needs before it multiplies screens. Use when the words in the interface matter as much as the pixels.

★ 0
SriAkhilSJ
SriAkhilSJ/PulseAIRepo

ui-ux-implementation-architect

Architect-level direction for turning a design into working UI — the handoff contract, component ownership versus screen-level layout, interaction and state conventions, design QA gates, and the review habits that keep an implementation faithful without frozen pixels. Use before any specific UI implementation skill.

★ 0
SriAkhilSJ
SriAkhilSJ/PulseAIRepo

test-strategy-architect

Architect-level test strategy — allocating assertions across unit, integration, contract and end-to-end layers, designing for testability through dependency boundaries, fixture and data strategy, and the flake and coverage policies that keep a suite trustworthy. Use when defining how a codebase is tested.

★ 0