kafka-architect
Architect-level Kafka design — topic and partition strategy, key choice and ordering guarantees, replication and min-ISR, retention and compaction, consumer groups and rebalancing, and the lag and disk failures that take a cluster down. Use when Kafka (or MSK/Confluent) carries event data.
- License 许可
- MIT
- 最近更新
- 2026-11-13
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- kafka-architect
- license
- MIT
- metadata
- { "crusor-path": "data-engineering-architect/event-streams-architect/kafka-architect", "crusor-domain": "data-engineering-architect", "crusor-origin": "original", "crusor-sources": "kafka.apache.org/documentation; docs.confluent.io/platform/current; docs.aws.amazon.com/msk", "crusor-triggers": "kafka, topic, partition, consumer group, offset, isr, retention, compaction, broker, msk, rebalance", "crusor-verified-on": "2026-10-30" }
- description
- Architect-level Kafka design — topic and partition strategy, key choice and ordering guarantees, replication and min-ISR, retention and compaction, consumer groups and rebalancing, and the lag and disk failures that take a cluster down. Use when Kafka (or MSK/Confluent) carries event data.
- compatibility
- Apache Kafka, MSK, Confluent Cloud, Redpanda
03 SKILL.md 正文
Kafka Architect
Kafka is an ordered log per partition. Every design decision follows from the key: it decides ordering, parallelism, and hot spots simultaneously.
When to use
- Event streams with multiple consumers and replay.
- Durable buffering between producers and consumers.
- Event sourcing or log-based integration.
When NOT to use
- Simple task queues (SQS-style semantics are simpler).
- Low volume where an HTTP call or a table would do.
Decision 1 — Topics and partitions
- One topic per event type with a stable schema; not one topic for everything.
- Partitions = target throughput ÷ per-partition throughput; more partitions mean more parallelism but slower rebalances and more files.
- Ordering is guaranteed within a partition only — if you need per-entity order, key by entity.
- Partition count is easy to increase and hard to decrease; start conservative and leave headroom.
Decision 2 — Keys
| Key | Consequence |
|---|---|
| entity id | per-entity order, uneven partitions if skewed |
| none (round-robin) | even spread, no ordering |
| tenant id | isolation and order per tenant, hot tenants |
| composite (tenant+id) | spread with intra-entity order |
- A skewed key creates a hot partition: one broker saturates while others idle. Check key distribution against real data before launch.
Decision 3 — Durability
- Replication factor 3,
min.insync.replicas=2,acks=allfor anything that must not be lost. unclean.leader.election=false— availability is not worth silent data loss.- Producer idempotence on; transactions only when consuming and producing atomically.
- Retention by time and size; compacted topics for entity state (latest per key).
Decision 4 — Consumers
- One consumer group per application; partitioning of work across members is automatic.
- Rebalances stop consumption — use cooperative/sticky assignment and static membership for long-running consumers.
- Commit offsets after processing, not before; at-least-once is the honest default.
- Lag in time, not messages: offset lag hides a slow consumer.
Decision 5 — Operations
- Monitor: under-replicated partitions, offline partitions, ISR shrinks, disk usage, controller changes, consumer lag.
- Disk is the classic outage: retention × throughput must fit with headroom; alert at 70%.
- Broker upgrades and rolling restarts need a plan for the partition movement they trigger.
- Schema registry with backward-compatible evolution (
event-schema-architect).
Failure modes at scale
- Key by a low-cardinality field; one partition takes 60% of traffic.
- Retention set to 7 days on a topic with a 10-day backlog after an incident.
- Consumer commits before processing; an outage loses a batch.
- Partition count increased during peak; rebalance storm for an hour.
- Disk full on one broker because replicas are unevenly distributed.
Verification receipts
- Topic inventory: partitions, replication, retention, cleanup policy.
- Key-distribution analysis on real data.
- Failure test: broker killed, production continues, no data loss.
- Lag dashboard in seconds with alerting.
- Disk capacity model with growth projection and alert thresholds.
04 同仓库更多 Skills
interaction-patterns-architect
Architect-level interaction design in code — choosing established patterns over invented ones, feedback and latency behaviour, destructive-action safeguards, navigation and overlay conventions, and the state model behind every interactive surface. Use when a screen has to behave, not just look.
design-system-ops-architect
Architect-level operation of a design system in code — token pipelines and versioning, component API and variant policy, the catalogue as the contract, adoption metrics, and the governance that stops a system from forking into three. Use when a component library must survive several teams.
design-handoff-architect
Architect-level design handoff — what must be specified before implementation starts, tokens over raw values, responsive and content-limit contracts, state matrices, and the review loop that catches drift between design and code. Use when a design is about to become code.
content-microcopy-architect
Architect-level UI content — where strings live, voice and terminology consistency, error and empty-state copy as part of the interface, translation-ready authoring, and the content model a product needs before it multiplies screens. Use when the words in the interface matter as much as the pixels.
ui-ux-implementation-architect
Architect-level direction for turning a design into working UI — the handoff contract, component ownership versus screen-level layout, interaction and state conventions, design QA gates, and the review habits that keep an implementation faithful without frozen pixels. Use before any specific UI implementation skill.
test-strategy-architect
Architect-level test strategy — allocating assertions across unit, integration, contract and end-to-end layers, designing for testability through dependency boundaries, fixture and data strategy, and the flake and coverage policies that keep a suite trustworthy. Use when defining how a codebase is tested.