AGENT SKILL

cdc-architect

Architect-level change data capture — log-based versus query-based capture, Debezium-style connectors, schema history, tombstones and deletes, exactly-once into a sink, snapshot-then-stream bootstrap, and the operational failure modes that silently diverge replicas. Use when a replica must track an operational database.

License 许可
MIT
最近更新
2026-11-13
Stars
★ 0
Forks
0

01 前言 Front Matter

name
cdc-architect
license
MIT
metadata
{ "crusor-path": "data-engineering-architect/ingestion-architect/cdc-architect", "crusor-domain": "data-engineering-architect", "crusor-origin": "original", "crusor-sources": "debezium.io/documentation; postgresql.org/docs/current/logical-replication.html; docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html", "crusor-triggers": "cdc, change data capture, debezium, logical replication, wal, binlog, tombstones, snapshot, replica divergence", "crusor-verified-on": "2026-10-30" }
description
Architect-level change data capture — log-based versus query-based capture, Debezium-style connectors, schema history, tombstones and deletes, exactly-once into a sink, snapshot-then-stream bootstrap, and the operational failure modes that silently diverge replicas. Use when a replica must track an operational database.
compatibility
Debezium, Kafka Connect, Postgres logical replication, DynamoDB Streams

03 SKILL.md 正文

CDC Architect

CDC is a second copy of your database with a lag and its own failure modes. The architecture question is how divergence is detected before a human notices.

When to use

  • Analytics or search must track an OLTP database without querying it directly.
  • Row-level changes matter (updates and deletes, not just inserts).
  • Polling by updated_at cannot keep up or misses deletes.

When NOT to use

  • Small tables where a nightly full extract is trivial.
  • The source has no changelog and no reliable ordering column.

Decision 1 — Capture method

Method Trade-off
log-based (WAL/binlog) complete, ordered, no query load; needs privileges and slots
query-based (timestamp/watermark) simple; misses deletes and hard deletes between polls
trigger-based complete; adds write load to the source
dual-write from app no lag, but app now owns consistency — avoid
  • Log-based is the default for correctness; the operational cost is a replication slot you must monitor.
  • Query-based is acceptable only for append-mostly, soft-delete data.

Decision 2 — Bootstrap

consistent snapshot → stream from that snapshot's LSN/position → replay deltas
  • Snapshot and stream must be bound to the same position, or rows inserted during the snapshot are lost or doubled.
  • Incremental snapshots for large tables avoid a long lock window.
  • Re-snapshotting a table means a coordinated swap, not a rebuild in place.

Decision 3 — Events

  • Event contains: before, after, op type, source position, timestamp, schema version.
  • Deletes need tombstones or an explicit delete event — otherwise "missing" is ambiguous downstream.
  • Keying by primary key lets downstream compact to current state.
  • Ordering is per key, not global; a consumer that assumes global order is broken.

Decision 4 — Sinks

  • Idempotent upserts keyed by primary key and event position.
  • Exactly-once is only within a transactional sink; otherwise design for duplicates.
  • Schema changes propagate: a new source column must not break the sink. Additive safe, renames/drops need a versioned migration.
  • DDL changes are a common outage: capture them and alert.

Decision 5 — Operations

  • Monitor: replication slot lag, connector lag (seconds not rows), error counts, and schema-history failures.
  • A stalled connector holds WAL and can fill the source disk — the most dangerous failure in this design.
  • Slot retention must be bounded and alerted; a dropped connector must release its slot.
  • Reconcile periodically: row counts and checksums per table between source and sink.

Failure modes at scale

  • Connector down for a day; WAL fills the primary disk and takes down production.
  • Deletes never captured; the replica grows forever with dead rows.
  • Snapshot taken with concurrent writes and no position binding; silent duplicates.
  • Schema rename propagated; downstream jobs fail with confusing null columns.
  • Lag measured by offset, hiding a 40-minute real delay.

Verification receipts

  1. Capture method chosen with the correctness rationale.
  2. Lag dashboard in seconds with an alert threshold.
  3. Delete propagation test: row removed at source, removed downstream.
  4. Reconciliation query comparing counts/checksums per table.
  5. Slot retention and disk-growth alarm verified.

04 同仓库更多 Skills

SriAkhilSJ
SriAkhilSJ/PulseAIRepo

interaction-patterns-architect

Architect-level interaction design in code — choosing established patterns over invented ones, feedback and latency behaviour, destructive-action safeguards, navigation and overlay conventions, and the state model behind every interactive surface. Use when a screen has to behave, not just look.

★ 0
SriAkhilSJ
SriAkhilSJ/PulseAIRepo

design-system-ops-architect

Architect-level operation of a design system in code — token pipelines and versioning, component API and variant policy, the catalogue as the contract, adoption metrics, and the governance that stops a system from forking into three. Use when a component library must survive several teams.

★ 0
SriAkhilSJ
SriAkhilSJ/PulseAIRepo

design-handoff-architect

Architect-level design handoff — what must be specified before implementation starts, tokens over raw values, responsive and content-limit contracts, state matrices, and the review loop that catches drift between design and code. Use when a design is about to become code.

★ 0
SriAkhilSJ
SriAkhilSJ/PulseAIRepo

content-microcopy-architect

Architect-level UI content — where strings live, voice and terminology consistency, error and empty-state copy as part of the interface, translation-ready authoring, and the content model a product needs before it multiplies screens. Use when the words in the interface matter as much as the pixels.

★ 0
SriAkhilSJ
SriAkhilSJ/PulseAIRepo

ui-ux-implementation-architect

Architect-level direction for turning a design into working UI — the handoff contract, component ownership versus screen-level layout, interaction and state conventions, design QA gates, and the review habits that keep an implementation faithful without frozen pixels. Use before any specific UI implementation skill.

★ 0
SriAkhilSJ
SriAkhilSJ/PulseAIRepo

test-strategy-architect

Architect-level test strategy — allocating assertions across unit, integration, contract and end-to-end layers, designing for testability through dependency boundaries, fixture and data strategy, and the flake and coverage policies that keep a suite trustworthy. Use when defining how a codebase is tested.

★ 0