cdc-architect
Architect-level change data capture — log-based versus query-based capture, Debezium-style connectors, schema history, tombstones and deletes, exactly-once into a sink, snapshot-then-stream bootstrap, and the operational failure modes that silently diverge replicas. Use when a replica must track an operational database.
- License 许可
- MIT
- 最近更新
- 2026-11-13
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- cdc-architect
- license
- MIT
- metadata
- { "crusor-path": "data-engineering-architect/ingestion-architect/cdc-architect", "crusor-domain": "data-engineering-architect", "crusor-origin": "original", "crusor-sources": "debezium.io/documentation; postgresql.org/docs/current/logical-replication.html; docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html", "crusor-triggers": "cdc, change data capture, debezium, logical replication, wal, binlog, tombstones, snapshot, replica divergence", "crusor-verified-on": "2026-10-30" }
- description
- Architect-level change data capture — log-based versus query-based capture, Debezium-style connectors, schema history, tombstones and deletes, exactly-once into a sink, snapshot-then-stream bootstrap, and the operational failure modes that silently diverge replicas. Use when a replica must track an operational database.
- compatibility
- Debezium, Kafka Connect, Postgres logical replication, DynamoDB Streams
03 SKILL.md 正文
CDC Architect
CDC is a second copy of your database with a lag and its own failure modes. The architecture question is how divergence is detected before a human notices.
When to use
- Analytics or search must track an OLTP database without querying it directly.
- Row-level changes matter (updates and deletes, not just inserts).
- Polling by
updated_atcannot keep up or misses deletes.
When NOT to use
- Small tables where a nightly full extract is trivial.
- The source has no changelog and no reliable ordering column.
Decision 1 — Capture method
| Method | Trade-off |
|---|---|
| log-based (WAL/binlog) | complete, ordered, no query load; needs privileges and slots |
| query-based (timestamp/watermark) | simple; misses deletes and hard deletes between polls |
| trigger-based | complete; adds write load to the source |
| dual-write from app | no lag, but app now owns consistency — avoid |
- Log-based is the default for correctness; the operational cost is a replication slot you must monitor.
- Query-based is acceptable only for append-mostly, soft-delete data.
Decision 2 — Bootstrap
consistent snapshot → stream from that snapshot's LSN/position → replay deltas
- Snapshot and stream must be bound to the same position, or rows inserted during the snapshot are lost or doubled.
- Incremental snapshots for large tables avoid a long lock window.
- Re-snapshotting a table means a coordinated swap, not a rebuild in place.
Decision 3 — Events
- Event contains: before, after, op type, source position, timestamp, schema version.
- Deletes need tombstones or an explicit delete event — otherwise "missing" is ambiguous downstream.
- Keying by primary key lets downstream compact to current state.
- Ordering is per key, not global; a consumer that assumes global order is broken.
Decision 4 — Sinks
- Idempotent upserts keyed by primary key and event position.
- Exactly-once is only within a transactional sink; otherwise design for duplicates.
- Schema changes propagate: a new source column must not break the sink. Additive safe, renames/drops need a versioned migration.
- DDL changes are a common outage: capture them and alert.
Decision 5 — Operations
- Monitor: replication slot lag, connector lag (seconds not rows), error counts, and schema-history failures.
- A stalled connector holds WAL and can fill the source disk — the most dangerous failure in this design.
- Slot retention must be bounded and alerted; a dropped connector must release its slot.
- Reconcile periodically: row counts and checksums per table between source and sink.
Failure modes at scale
- Connector down for a day; WAL fills the primary disk and takes down production.
- Deletes never captured; the replica grows forever with dead rows.
- Snapshot taken with concurrent writes and no position binding; silent duplicates.
- Schema rename propagated; downstream jobs fail with confusing null columns.
- Lag measured by offset, hiding a 40-minute real delay.
Verification receipts
- Capture method chosen with the correctness rationale.
- Lag dashboard in seconds with an alert threshold.
- Delete propagation test: row removed at source, removed downstream.
- Reconciliation query comparing counts/checksums per table.
- Slot retention and disk-growth alarm verified.
04 同仓库更多 Skills
interaction-patterns-architect
Architect-level interaction design in code — choosing established patterns over invented ones, feedback and latency behaviour, destructive-action safeguards, navigation and overlay conventions, and the state model behind every interactive surface. Use when a screen has to behave, not just look.
design-system-ops-architect
Architect-level operation of a design system in code — token pipelines and versioning, component API and variant policy, the catalogue as the contract, adoption metrics, and the governance that stops a system from forking into three. Use when a component library must survive several teams.
design-handoff-architect
Architect-level design handoff — what must be specified before implementation starts, tokens over raw values, responsive and content-limit contracts, state matrices, and the review loop that catches drift between design and code. Use when a design is about to become code.
content-microcopy-architect
Architect-level UI content — where strings live, voice and terminology consistency, error and empty-state copy as part of the interface, translation-ready authoring, and the content model a product needs before it multiplies screens. Use when the words in the interface matter as much as the pixels.
ui-ux-implementation-architect
Architect-level direction for turning a design into working UI — the handoff contract, component ownership versus screen-level layout, interaction and state conventions, design QA gates, and the review habits that keep an implementation faithful without frozen pixels. Use before any specific UI implementation skill.
test-strategy-architect
Architect-level test strategy — allocating assertions across unit, integration, contract and end-to-end layers, designing for testability through dependency boundaries, fixture and data strategy, and the flake and coverage policies that keep a suite trustworthy. Use when defining how a codebase is tested.