AGENT SKILL

kafka-architect

Architect-level Kafka design — topic and partition strategy, key choice and ordering guarantees, replication and min-ISR, retention and compaction, consumer groups and rebalancing, and the lag and disk failures that take a cluster down. Use when Kafka (or MSK/Confluent) carries event data.

License 许可
MIT
最近更新
2026-11-13
Stars
★ 0
Forks
0

01 前言 Front Matter

name
kafka-architect
license
MIT
metadata
{ "crusor-path": "data-engineering-architect/event-streams-architect/kafka-architect", "crusor-domain": "data-engineering-architect", "crusor-origin": "original", "crusor-sources": "kafka.apache.org/documentation; docs.confluent.io/platform/current; docs.aws.amazon.com/msk", "crusor-triggers": "kafka, topic, partition, consumer group, offset, isr, retention, compaction, broker, msk, rebalance", "crusor-verified-on": "2026-10-30" }
description
Architect-level Kafka design — topic and partition strategy, key choice and ordering guarantees, replication and min-ISR, retention and compaction, consumer groups and rebalancing, and the lag and disk failures that take a cluster down. Use when Kafka (or MSK/Confluent) carries event data.
compatibility
Apache Kafka, MSK, Confluent Cloud, Redpanda

03 SKILL.md 正文

Kafka Architect

Kafka is an ordered log per partition. Every design decision follows from the key: it decides ordering, parallelism, and hot spots simultaneously.

When to use

  • Event streams with multiple consumers and replay.
  • Durable buffering between producers and consumers.
  • Event sourcing or log-based integration.

When NOT to use

  • Simple task queues (SQS-style semantics are simpler).
  • Low volume where an HTTP call or a table would do.

Decision 1 — Topics and partitions

  • One topic per event type with a stable schema; not one topic for everything.
  • Partitions = target throughput ÷ per-partition throughput; more partitions mean more parallelism but slower rebalances and more files.
  • Ordering is guaranteed within a partition only — if you need per-entity order, key by entity.
  • Partition count is easy to increase and hard to decrease; start conservative and leave headroom.

Decision 2 — Keys

Key Consequence
entity id per-entity order, uneven partitions if skewed
none (round-robin) even spread, no ordering
tenant id isolation and order per tenant, hot tenants
composite (tenant+id) spread with intra-entity order
  • A skewed key creates a hot partition: one broker saturates while others idle. Check key distribution against real data before launch.

Decision 3 — Durability

  • Replication factor 3, min.insync.replicas=2, acks=all for anything that must not be lost.
  • unclean.leader.election=false — availability is not worth silent data loss.
  • Producer idempotence on; transactions only when consuming and producing atomically.
  • Retention by time and size; compacted topics for entity state (latest per key).

Decision 4 — Consumers

  • One consumer group per application; partitioning of work across members is automatic.
  • Rebalances stop consumption — use cooperative/sticky assignment and static membership for long-running consumers.
  • Commit offsets after processing, not before; at-least-once is the honest default.
  • Lag in time, not messages: offset lag hides a slow consumer.

Decision 5 — Operations

  • Monitor: under-replicated partitions, offline partitions, ISR shrinks, disk usage, controller changes, consumer lag.
  • Disk is the classic outage: retention × throughput must fit with headroom; alert at 70%.
  • Broker upgrades and rolling restarts need a plan for the partition movement they trigger.
  • Schema registry with backward-compatible evolution (event-schema-architect).

Failure modes at scale

  • Key by a low-cardinality field; one partition takes 60% of traffic.
  • Retention set to 7 days on a topic with a 10-day backlog after an incident.
  • Consumer commits before processing; an outage loses a batch.
  • Partition count increased during peak; rebalance storm for an hour.
  • Disk full on one broker because replicas are unevenly distributed.

Verification receipts

  1. Topic inventory: partitions, replication, retention, cleanup policy.
  2. Key-distribution analysis on real data.
  3. Failure test: broker killed, production continues, no data loss.
  4. Lag dashboard in seconds with alerting.
  5. Disk capacity model with growth projection and alert thresholds.

04 同仓库更多 Skills

SriAkhilSJ
SriAkhilSJ/PulseAIRepo

interaction-patterns-architect

Architect-level interaction design in code — choosing established patterns over invented ones, feedback and latency behaviour, destructive-action safeguards, navigation and overlay conventions, and the state model behind every interactive surface. Use when a screen has to behave, not just look.

★ 0
SriAkhilSJ
SriAkhilSJ/PulseAIRepo

design-system-ops-architect

Architect-level operation of a design system in code — token pipelines and versioning, component API and variant policy, the catalogue as the contract, adoption metrics, and the governance that stops a system from forking into three. Use when a component library must survive several teams.

★ 0
SriAkhilSJ
SriAkhilSJ/PulseAIRepo

design-handoff-architect

Architect-level design handoff — what must be specified before implementation starts, tokens over raw values, responsive and content-limit contracts, state matrices, and the review loop that catches drift between design and code. Use when a design is about to become code.

★ 0
SriAkhilSJ
SriAkhilSJ/PulseAIRepo

content-microcopy-architect

Architect-level UI content — where strings live, voice and terminology consistency, error and empty-state copy as part of the interface, translation-ready authoring, and the content model a product needs before it multiplies screens. Use when the words in the interface matter as much as the pixels.

★ 0
SriAkhilSJ
SriAkhilSJ/PulseAIRepo

ui-ux-implementation-architect

Architect-level direction for turning a design into working UI — the handoff contract, component ownership versus screen-level layout, interaction and state conventions, design QA gates, and the review habits that keep an implementation faithful without frozen pixels. Use before any specific UI implementation skill.

★ 0
SriAkhilSJ
SriAkhilSJ/PulseAIRepo

test-strategy-architect

Architect-level test strategy — allocating assertions across unit, integration, contract and end-to-end layers, designing for testability through dependency boundaries, fixture and data strategy, and the flake and coverage policies that keep a suite trustworthy. Use when defining how a codebase is tested.

★ 0