[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7B42268SWDTMNBT93PFAQQE":3,"skill-same:01M7B42268SWDTMNBT93PFAQQE":78},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":27,"commitSha":28,"skillDirectory":29,"forksCount":14,"repositoryHtmlUrl":30,"repositoryHomepageUrl":6,"ownerHtmlUrl":31,"ownerBlogUrl":6,"ownerDescription":32,"updatedAt":33,"install":34,"resources":47},"01M7B42268SWDTMNBT93PFAQQE","ddia-systems",null,"Design data systems by understanding storage engines, replication, partitioning, transactions, and consistency models. Use when the user mentions \"database choice\", \"which database should I use\", \"SQL or NoSQL\", \"replication lag\", \"partitioning strategy\", \"consistency vs availability\", \"stream processing\", \"ACID transactions\", \"eventual consistency\", \"my queries are slow at scale\", or \"data is inconsistent across replicas\". Also trigger when choosing a datastore, designing data pipelines, or debugging distributed-system consistency issues. Covers data models, batch\u002Fstream processing, and distributed consensus. For system design, see system-design. For resilience, see release-it.","Div0011\u002FFrontend-Pipeline","Div0011","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F198384399?v=4","01M7AX16GDZ3959RM6WMDTTFV4","Div","01M7AX16HF6EPV40MA8G0G0JRC",0,"MIT","https:\u002F\u002Fgithub.com\u002FDiv0011\u002FFrontend-Pipeline\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fwondelai-skills\u002F.windsurf\u002Fskills\u002Fddia-systems\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: ddia-systems\ndescription: 'Design data systems by understanding storage engines, replication, partitioning, transactions, and consistency models. Use when the user mentions \"database choice\", \"which database should I use\", \"SQL or NoSQL\", \"replication lag\", \"partitioning strategy\", \"consistency vs availability\", \"stream processing\", \"ACID transactions\", \"eventual consistency\", \"my queries are slow at scale\", or \"data is inconsistent across replicas\". Also trigger when choosing a datastore, designing data pipelines, or debugging distributed-system consistency issues. Covers data models, batch\u002Fstream processing, and distributed consensus. For system design, see system-design. For resilience, see release-it.'\nlicense: MIT\nmetadata:\n  author: wondelai\n  version: \"1.4.0\"\n---\n\n# Designing Data-Intensive Applications Framework\n\nA principled approach to building reliable, scalable, and maintainable data systems. Apply these principles when choosing databases, designing schemas, architecting distributed systems, or reasoning about consistency and fault tolerance.\n\n## Core Principle\n\n**Data outlives code.** Applications are rewritten and frameworks come and go, but data persists for decades -- prioritize the long-term correctness, durability, and evolvability of the data layer. Most applications are data-intensive, not compute-intensive: the hard problems are data volume, complexity, and rate of change, and explicit consistency\u002Favailability\u002Flatency trade-offs separate robust systems from fragile ones.\n\n## Scoring\n\n**Goal: 10\u002F10.** Score a data architecture by the seven Quick Diagnostic rows below: award ~1.4 points per row answered \"yes\" with evidence (deliberate, documented trade-off), 0 where the answer is \"no\" or unknown.\n\n- **9-10:** every domain choice -- data model, storage engine, replication, partitioning, isolation, derived-data, fault handling -- is deliberate, documented, and matched to actual read\u002Fwrite\u002Fconsistency requirements; failover tested.\n- **5-6:** core choices made but two or three diagnostic rows fail -- e.g. default isolation level unknown, hot-key risk unhandled, or failover untested.\n- **\u003C=3:** choices driven by familiarity, not requirements; ignored failure modes (replication lag, write skew, hot partitions) and accidental complexity dominate.\n\nReport the current score, which diagnostic rows failed, and the improvements needed to reach 10\u002F10.\n\n## The DDIA Framework\n\nSeven domains for reasoning about data-intensive systems:\n\n### 1. Data Models and Query Languages\n\n**Core concept:** The data model shapes how you think about the problem. Relational, document, and graph models each impose different constraints and enable different query patterns.\n\n**Why it works:** Choosing the wrong data model forces application code to compensate for representational mismatch, adding accidental complexity that compounds over time.\n\n**Key insights:**\n- Relational models excel at many-to-many relationships and ad-hoc queries; document models at one-to-many relationships and locality; graph models at recursive traversals over interconnected data\n- Schema-on-write (relational) catches errors early; schema-on-read (document) offers flexibility\n- Polyglot persistence -- different stores for different access patterns -- is often the right answer\n- Object-relational impedance mismatch is a real cost; document models reduce it for self-contained aggregates\n\n**Code applications:**\n\n| Context | Pattern | Example |\n|---------|---------|---------|\n| **User profiles with nested data** | Document model for self-contained aggregates | Profile, addresses, and preferences in one MongoDB document |\n| **Social network connections** | Graph model for relationship traversal | Neo4j Cypher: `MATCH (a)-[:FOLLOWS*2]->(b)` for friend-of-friend |\n| **Financial ledger with joins** | Relational model for referential integrity | PostgreSQL foreign keys between accounts, transactions, entries |\n\nSee [references\u002Fdata-models.md](references\u002Fdata-models.md) when picking relational vs document vs graph or evaluating schema-on-read -- adds the full trade-off matrix and query-language comparisons.\n\n### 2. Storage Engines\n\n**Core concept:** Storage engines trade off read performance against write performance. Log-structured engines (LSM trees) optimize writes; page-oriented engines (B-trees) balance reads and writes.\n\n**Key insights:**\n- LSM trees: append-only writes, periodic compaction, excellent write throughput, higher read amplification\n- B-trees: in-place updates, predictable read latency, write amplification from page splits\n- Write amplification (one logical write causing multiple physical writes) matters for SSDs with limited write cycles\n- Column-oriented storage dramatically improves analytical queries through compression and vectorized processing\n- In-memory databases are fast because they avoid encoding overhead, not because they avoid disk\n\n**Code applications:**\n\n| Context | Pattern | Example |\n|---------|---------|---------|\n| **High write throughput** | LSM-tree engine | Cassandra or RocksDB for time-series ingestion at 100K+ writes\u002Fsec |\n| **Mixed read\u002Fwrite OLTP** | B-tree engine | PostgreSQL B-tree indexes for transactional point lookups |\n| **Analytical queries** | Column-oriented storage | ClickHouse or Parquet for scanning billions of rows, few columns |\n\nSee [references\u002Fstorage-engines.md](references\u002Fstorage-engines.md) when a workload is read\u002Fwrite-bound or you must choose indexes -- adds write\u002Fread-path diagrams, compaction strategies, column storage, and a benchmark-driven decision procedure.\n\n### 3. Replication\n\n**Core concept:** Replication keeps copies of data on multiple machines for fault tolerance, scalability, and latency reduction. The core challenge is handling changes consistently.\n\n**Why it works:** Every replication strategy trades off consistency, availability, and latency. Making the trade-off explicit prevents subtle anomalies that surface only under load or failure.\n\n**Key insights:**\n- Single-leader: simple, strong consistency possible, but the leader is a bottleneck and single point of failure\n- Multi-leader: better write availability across data centers, but complex conflict resolution\n- Leaderless: highest availability via quorum reads\u002Fwrites, but needs careful conflict handling\n- Replication lag causes read-your-writes, monotonic-read, and causality violations\n- Synchronous replication guarantees durability but adds latency; asynchronous risks data loss on failover\n- CRDTs and last-writer-wins resolve conflicts with very different correctness guarantees\n\n**Code applications:**\n\n| Context | Pattern | Example |\n|---------|---------|---------|\n| **Read-heavy web app** | Single-leader with read replicas | PostgreSQL primary + read replicas behind pgBouncer |\n| **Multi-region writes** | Multi-leader replication | CockroachDB or Spanner with bounded staleness |\n| **Shopping cart availability** | Leaderless with merge | DynamoDB with last-writer-wins or application-level cart merge |\n\nSee [references\u002Freplication.md](references\u002Freplication.md) when choosing single\u002Fmulti\u002Fleaderless or debugging stale reads -- adds lag anomalies, quorum math, conflict resolution, and CRDTs.\n\n### 4. Partitioning\n\n**Core concept:** Partitioning (sharding) distributes data across nodes so each handles a subset, enabling horizontal scaling beyond a single machine.\n\n**Key insights:**\n- Key-range partitioning supports efficient range scans but risks hotspots on sequential keys\n- Hash partitioning distributes load evenly but destroys sort order, making range queries expensive\n- Local secondary indexes require scatter-gather queries; global secondary indexes require cross-partition updates\n- Hotspots occur even with hashing when a single key is extremely popular (celebrity problem)\n- Rebalancing strategies: fixed partition count, dynamic splitting, or proportional to nodes\n\n**Code applications:**\n\n| Context | Pattern | Example |\n|---------|---------|---------|\n| **Time-series data** | Key-range partitioning by time + source | Partition by `(sensor_id, date)` to avoid current-day write hotspot |\n| **User data at scale** | Hash partitioning on user ID | Cassandra consistent hashing on `user_id` for even distribution |\n| **Celebrity\u002Fhot-key problem** | Key splitting with random suffix | Append random digit to hot key, fan out reads across 10 sub-partitions |\n\nSee [references\u002Fpartitioning.md](references\u002Fpartitioning.md) when sharding or fighting a hot key -- adds rebalancing strategies, request routing, and local-vs-global secondary index trade-offs.\n\n### 5. Transactions and Consistency\n\n**Core concept:** Transactions provide safety guarantees (ACID) that simplify application code by letting you pretend failures and concurrency don't exist -- within the transaction's scope.\n\n**Why it works:** Without transactions, every piece of application code must handle partial failures, races, and concurrent modification. Transactions move that complexity into the database, handled correctly once.\n\n**Key insights:**\n- Isolation levels are a spectrum: read uncommitted, read committed, snapshot isolation, serializable\n- Most databases default to read committed or snapshot isolation -- NOT serializable -- so you must understand the anomalies this permits\n- Write skew: two transactions read the same data, decide, and write different records -- no row lock prevents it\n- Serializable snapshot isolation (SSI) gives full serializability optimistically: no blocking, but aborts on conflict; two-phase locking blocks and deadlocks under contention\n- Distributed transactions (two-phase commit) are expensive and fragile; design around single-partition operations instead\n\n**Code applications:**\n\n| Context | Pattern | Example |\n|---------|---------|---------|\n| **Account balance transfer** | Serializable transaction | `BEGIN; UPDATE accounts ... -100 WHERE id=1; UPDATE accounts ... +100 WHERE id=2; COMMIT;` |\n| **Inventory reservation** | SELECT FOR UPDATE to prevent write skew | `SELECT stock FROM items WHERE id = X FOR UPDATE` before decrementing |\n| **Cross-service operations** | Saga instead of distributed transaction | Charge card, reserve inventory; on failure, run compensating refund |\n\nSee [references\u002Ftransactions.md](references\u002Ftransactions.md) when setting isolation levels or chasing a concurrency bug -- adds per-isolation anomaly tables, write-skew examples, 2PL vs SSI, and distributed-transaction pitfalls.\n\n### 6. Batch and Stream Processing\n\n**Core concept:** Batch processing transforms bounded datasets in bulk; stream processing transforms unbounded event streams continuously. Both compute derived data.\n\n**Why it works:** Separating the system of record from derived data (caches, indexes, materialized views) lets each be optimized independently and rebuilt from source when requirements change.\n\n**Key insights:**\n- MapReduce is conceptually simple but operationally awkward; dataflow engines (Spark, Flink) generalize it with arbitrary DAGs\n- Change data capture (CDC) turns database writes into a stream downstream systems can consume\n- Stream-table duality: a stream is the changelog of a table; a table is the materialized state of a stream\n- Exactly-once semantics require idempotent operations or transactional output\n- Time windowing (tumbling, hopping, session) is essential for aggregating unbounded streams\n\n**Code applications:**\n\n| Context | Pattern | Example |\n|---------|---------|---------|\n| **Daily analytics pipeline** | Batch processing with Spark | Read day's events from S3, aggregate, write to warehouse |\n| **Real-time fraud detection** | Stream processing with Flink | Kafka payment events, rules over 5-second tumbling windows |\n| **Syncing search index** | Change data capture | Debezium captures PostgreSQL WAL, Kafka feeds Elasticsearch |\n| **Audit trail \u002F event replay** | Event sourcing | Store `OrderPlaced`, `OrderShipped` events; rebuild state by replaying |\n\nSee [references\u002Fbatch-stream.md](references\u002Fbatch-stream.md) when designing a pipeline or deriving data from a system of record -- adds dataflow engines, CDC wiring, windowing, and exactly-once techniques.\n\n### 7. Reliability and Fault Tolerance\n\n**Core concept:** Faults are inevitable; failures are not. A reliable system continues operating correctly even when individual components fail. Design for faults, not against them.\n\n**Key insights:**\n- A fault is one component deviating from spec; a failure is the whole system stopping -- fault tolerance prevents the former becoming the latter\n- Hardware faults are random and independent; software faults are correlated and systematic (more dangerous)\n- Human error is the leading cause of outages -- minimize opportunity for mistakes, maximize ability to recover\n- Timeouts are the fundamental fault detector, but tuning is hard: too short causes false positives, too long delays recovery\n- Safety properties (nothing bad happens) must always hold; liveness (something good eventually happens) may be temporarily violated\n- Byzantine fault tolerance is rarely needed outside blockchain; assume crash-stop or crash-recovery\n\n**Code applications:**\n\n| Context | Pattern | Example |\n|---------|---------|---------|\n| **Service communication** | Timeouts + retries with backoff | `retry(max=3, backoff=exponential(base=1s, max=30s))` with jitter |\n| **Leader election** | Consensus algorithm (Raft\u002FPaxos) | etcd or ZooKeeper for distributed locks and leader election |\n| **Graceful degradation** | Circuit breaker | Resilience4j: open circuit after 50% failures in 10-second window |\n\nSee [references\u002Ffault-tolerance.md](references\u002Ffault-tolerance.md) when tuning timeouts\u002Fretries or adding consensus -- adds fault classification, timeout-tuning math, Raft\u002FPaxos mechanics, and safety\u002Fliveness guarantees.\n\n## Common Mistakes\n\n| Mistake | Why It Fails | Fix |\n|---------|-------------|------|\n| **Choosing a database by popularity** | Engines have fundamentally different trade-offs | Match storage engine to actual read\u002Fwrite patterns |\n| **Ignoring replication lag** | Stale reads, phantom reads, lost updates | Implement read-your-writes and monotonic-read guarantees |\n| **Distributed transactions everywhere** | 2PC is slow, fragile; coordinator is a SPOF | Design single-partition operations; use sagas across services |\n| **Hash partitioning everything** | Destroys range query ability | Key-range partitioning for time-series; composite keys for locality |\n| **Assuming serializable isolation** | Defaults are weaker; write skew appears in production | Check the actual default; use explicit locking where needed |\n| **Conflating batch and stream** | Wrong tool adds latency or wasted complexity | Match processing model to data boundedness and latency needs |\n| **Treating all faults as recoverable** | Corruption and Byzantine faults need different handling | Classify faults; design a recovery strategy per class |\n\n## Quick Diagnostic\n\n| Question | If No | Action |\n|----------|-------|--------|\n| Can you explain why you chose this database over alternatives? | Choice was familiarity, not requirements | Evaluate data model fit, read\u002Fwrite ratio, consistency needs, scaling path |\n| Do you know your database's default isolation level? | Latent concurrency bugs | Check docs; test for write skew and phantom reads |\n| Is your replication strategy explicitly chosen? | Implicit consistency\u002Fdurability assumptions | Document sync vs async, failover behavior, lag tolerance |\n| Can your system handle a hot partition key? | One popular entity can down the cluster | Add key-splitting or load shedding for hot keys |\n| Do you separate system of record from derived data? | Every change requires migrating everything | Introduce CDC or event sourcing to decouple |\n| Are timeouts and retries tuned, not defaulted? | Cascading failures or needless delays | Measure p99; set timeouts above p99, below cascade threshold |\n| Have you tested failover in production conditions? | Recovery plan is theoretical | Run chaos experiments: kill leaders, partition networks, fill disks |\n\n## Further Reading\n\nFor the complete treatment with detailed diagrams and research references:\n\n- [*\"Designing Data-Intensive Applications\"*](https:\u002F\u002Fwww.amazon.com\u002FDesigning-Data-Intensive-Applications-Reliable-Maintainable\u002Fdp\u002F1449373321?tag=wondelai00-20) by Martin Kleppmann\n\n## About the Author\n\n**Martin Kleppmann** is a distributed-systems researcher at the University of Cambridge and a former engineer at LinkedIn and Rapportive, known for his work on CRDTs and local-first software. His book *Designing Data-Intensive Applications* (2017) is the definitive reference for engineers building data systems, praised for making distributed-systems concepts accessible and practical.\n","# Designing Data-Intensive Applications Framework\n\nA principled approach to building reliable, scalable, and maintainable data systems. Apply these principles when choosing databases, designing schemas, architecting distributed systems, or reasoning about consistency and fault tolerance.\n\n## Core Principle\n\n**Data outlives code.** Applications are rewritten and frameworks come and go, but data persists for decades -- prioritize the long-term correctness, durability, and evolvability of the data layer. Most applications are data-intensive, not compute-intensive: the hard problems are data volume, complexity, and rate of change, and explicit consistency\u002Favailability\u002Flatency trade-offs separate robust systems from fragile ones.\n\n## Scoring\n\n**Goal: 10\u002F10.** Score a data architecture by the seven Quick Diagnostic rows below: award ~1.4 points per row answered \"yes\" with evidence (deliberate, documented trade-off), 0 where the answer is \"no\" or unknown.\n\n- **9-10:** every domain choice -- data model, storage engine, replication, partitioning, isolation, derived-data, fault handling -- is deliberate, documented, and matched to actual read\u002Fwrite\u002Fconsistency requirements; failover tested.\n- **5-6:** core choices made but two or three diagnostic rows fail -- e.g. default isolation level unknown, hot-key risk unhandled, or failover untested.\n- **\u003C=3:** choices driven by familiarity, not requirements; ignored failure modes (replication lag, write skew, hot partitions) and accidental complexity dominate.\n\nReport the current score, which diagnostic rows failed, and the improvements needed to reach 10\u002F10.\n\n## The DDIA Framework\n\nSeven domains for reasoning about data-intensive systems:\n\n### 1. Data Models and Query Languages\n\n**Core concept:** The data model shapes how you think about the problem. Relational, document, and graph models each impose different constraints and enable different query patterns.\n\n**Why it works:** Choosing the wrong data model forces application code to compensate for representational mismatch, adding accidental complexity that compounds over time.\n\n**Key insights:**\n- Relational models excel at many-to-many relationships and ad-hoc queries; document models at one-to-many relationships and locality; graph models at recursive traversals over interconnected data\n- Schema-on-write (relational) catches errors early; schema-on-read (document) offers flexibility\n- Polyglot persistence -- different stores for different access patterns -- is often the right answer\n- Object-relational impedance mismatch is a real cost; document models reduce it for self-contained aggregates\n\n**Code applications:**\n\n| Context | Pattern | Example |\n|---------|---------|---------|\n| **User profiles with nested data** | Document model for self-contained aggregates | Profile, addresses, and preferences in one MongoDB document |\n| **Social network connections** | Graph model for relationship traversal | Neo4j Cypher: `MATCH (a)-[:FOLLOWS*2]->(b)` for friend-of-friend |\n| **Financial ledger with joins** | Relational model for referential integrity | PostgreSQL foreign keys between accounts, transactions, entries |\n\nSee [references\u002Fdata-models.md](references\u002Fdata-models.md) when picking relational vs document vs graph or evaluating schema-on-read -- adds the full trade-off matrix and query-language comparisons.\n\n### 2. Storage Engines\n\n**Core concept:** Storage engines trade off read performance against write performance. Log-structured engines (LSM trees) optimize writes; page-oriented engines (B-trees) balance reads and writes.\n\n**Key insights:**\n- LSM trees: append-only writes, periodic compaction, excellent write throughput, higher read amplification\n- B-trees: in-place updates, predictable read latency, write amplification from page splits\n- Write amplification (one logical write causing multiple physical writes) matters for SSDs with limited write cycles\n- Column-oriented storage dramatically improves analytical queries through compression and vectorized processing\n- In-memory databases are fast because they avoid encoding overhead, not because they avoid disk\n\n**Code applications:**\n\n| Context | Pattern | Example |\n|---------|---------|---------|\n| **High write throughput** | LSM-tree engine | Cassandra or RocksDB for time-series ingestion at 100K+ writes\u002Fsec |\n| **Mixed read\u002Fwrite OLTP** | B-tree engine | PostgreSQL B-tree indexes for transactional point lookups |\n| **Analytical queries** | Column-oriented storage | ClickHouse or Parquet for scanning billions of rows, few columns |\n\nSee [references\u002Fstorage-engines.md](references\u002Fstorage-engines.md) when a workload is read\u002Fwrite-bound or you must choose indexes -- adds write\u002Fread-path diagrams, compaction strategies, column storage, and a benchmark-driven decision procedure.\n\n### 3. Replication\n\n**Core concept:** Replication keeps copies of data on multiple machines for fault tolerance, scalability, and latency reduction. The core challenge is handling changes consistently.\n\n**Why it works:** Every replication strategy trades off consistency, availability, and latency. Making the trade-off explicit prevents subtle anomalies that surface only under load or failure.\n\n**Key insights:**\n- Single-leader: simple, strong consistency possible, but the leader is a bottleneck and single point of failure\n- Multi-leader: better write availability across data centers, but complex conflict resolution\n- Leaderless: highest availability via quorum reads\u002Fwrites, but needs careful conflict handling\n- Replication lag causes read-your-writes, monotonic-read, and causality violations\n- Synchronous replication guarantees durability but adds latency; asynchronous risks data loss on failover\n- CRDTs and last-writer-wins resolve conflicts with very different correctness guarantees\n\n**Code applications:**\n\n| Context | Pattern | Example |\n|---------|---------|---------|\n| **Read-heavy web app** | Single-leader with read replicas | PostgreSQL primary + read replicas behind pgBouncer |\n| **Multi-region writes** | Multi-leader replication | CockroachDB or Spanner with bounded staleness |\n| **Shopping cart availability** | Leaderless with merge | DynamoDB with last-writer-wins or application-level cart merge |\n\nSee [references\u002Freplication.md](references\u002Freplication.md) when choosing single\u002Fmulti\u002Fleaderless or debugging stale reads -- adds lag anomalies, quorum math, conflict resolution, and CRDTs.\n\n### 4. Partitioning\n\n**Core concept:** Partitioning (sharding) distributes data across nodes so each handles a subset, enabling horizontal scaling beyond a single machine.\n\n**Key insights:**\n- Key-range partitioning supports efficient range scans but risks hotspots on sequential keys\n- Hash partitioning distributes load evenly but destroys sort order, making range queries expensive\n- Local secondary indexes require scatter-gather queries; global secondary indexes require cross-partition updates\n- Hotspots occur even with hashing when a single key is extremely popular (celebrity problem)\n- Rebalancing strategies: fixed partition count, dynamic splitting, or proportional to nodes\n\n**Code applications:**\n\n| Context | Pattern | Example |\n|---------|---------|---------|\n| **Time-series data** | Key-range partitioning by time + source | Partition by `(sensor_id, date)` to avoid current-day write hotspot |\n| **User data at scale** | Hash partitioning on user ID | Cassandra consistent hashing on `user_id` for even distribution |\n| **Celebrity\u002Fhot-key problem** | Key splitting with random suffix | Append random digit to hot key, fan out reads across 10 sub-partitions |\n\nSee [references\u002Fpartitioning.md](references\u002Fpartitioning.md) when sharding or fighting a hot key -- adds rebalancing strategies, request routing, and local-vs-global secondary index trade-offs.\n\n### 5. Transactions and Consistency\n\n**Core concept:** Transactions provide safety guarantees (ACID) that simplify application code by letting you pretend failures and concurrency don't exist -- within the transaction's scope.\n\n**Why it works:** Without transactions, every piece of application code must handle partial failures, races, and concurrent modification. Transactions move that complexity into the database, handled correctly once.\n\n**Key insights:**\n- Isolation levels are a spectrum: read uncommitted, read committed, snapshot isolation, serializable\n- Most databases default to read committed or snapshot isolation -- NOT serializable -- so you must understand the anomalies this permits\n- Write skew: two transactions read the same data, decide, and write different records -- no row lock prevents it\n- Serializable snapshot isolation (SSI) gives full serializability optimistically: no blocking, but aborts on conflict; two-phase locking blocks and deadlocks under contention\n- Distributed transactions (two-phase commit) are expensive and fragile; design around single-partition operations instead\n\n**Code applications:**\n\n| Context | Pattern | Example |\n|---------|---------|---------|\n| **Account balance transfer** | Serializable transaction | `BEGIN; UPDATE accounts ... -100 WHERE id=1; UPDATE accounts ... +100 WHERE id=2; COMMIT;` |\n| **Inventory reservation** | SELECT FOR UPDATE to prevent write skew | `SELECT stock FROM items WHERE id = X FOR UPDATE` before decrementing |\n| **Cross-service operations** | Saga instead of distributed transaction | Charge card, reserve inventory; on failure, run compensating refund |\n\nSee [references\u002Ftransactions.md](references\u002Ftransactions.md) when setting isolation levels or chasing a concurrency bug -- adds per-isolation anomaly tables, write-skew examples, 2PL vs SSI, and distributed-transaction pitfalls.\n\n### 6. Batch and Stream Processing\n\n**Core concept:** Batch processing transforms bounded datasets in bulk; stream processing transforms unbounded event streams continuously. Both compute derived data.\n\n**Why it works:** Separating the system of record from derived data (caches, indexes, materialized views) lets each be optimized independently and rebuilt from source when requirements change.\n\n**Key insights:**\n- MapReduce is conceptually simple but operationally awkward; dataflow engines (Spark, Flink) generalize it with arbitrary DAGs\n- Change data capture (CDC) turns database writes into a stream downstream systems can consume\n- Stream-table duality: a stream is the changelog of a table; a table is the materialized state of a stream\n- Exactly-once semantics require idempotent operations or transactional output\n- Time windowing (tumbling, hopping, session) is essential for aggregating unbounded streams\n\n**Code applications:**\n\n| Context | Pattern | Example |\n|---------|---------|---------|\n| **Daily analytics pipeline** | Batch processing with Spark | Read day's events from S3, aggregate, write to warehouse |\n| **Real-time fraud detection** | Stream processing with Flink | Kafka payment events, rules over 5-second tumbling windows |\n| **Syncing search index** | Change data capture | Debezium captures PostgreSQL WAL, Kafka feeds Elasticsearch |\n| **Audit trail \u002F event replay** | Event sourcing | Store `OrderPlaced`, `OrderShipped` events; rebuild state by replaying |\n\nSee [references\u002Fbatch-stream.md](references\u002Fbatch-stream.md) when designing a pipeline or deriving data from a system of record -- adds dataflow engines, CDC wiring, windowing, and exactly-once techniques.\n\n### 7. Reliability and Fault Tolerance\n\n**Core concept:** Faults are inevitable; failures are not. A reliable system continues operating correctly even when individual components fail. Design for faults, not against them.\n\n**Key insights:**\n- A fault is one component deviating from spec; a failure is the whole system stopping -- fault tolerance prevents the former becoming the latter\n- Hardware faults are random and independent; software faults are correlated and systematic (more dangerous)\n- Human error is the leading cause of outages -- minimize opportunity for mistakes, maximize ability to recover\n- Timeouts are the fundamental fault detector, but tuning is hard: too short causes false positives, too long delays recovery\n- Safety properties (nothing bad happens) must always hold; liveness (something good eventually happens) may be temporarily violated\n- Byzantine fault tolerance is rarely needed outside blockchain; assume crash-stop or crash-recovery\n\n**Code applications:**\n\n| Context | Pattern | Example |\n|---------|---------|---------|\n| **Service communication** | Timeouts + retries with backoff | `retry(max=3, backoff=exponential(base=1s, max=30s))` with jitter |\n| **Leader election** | Consensus algorithm (Raft\u002FPaxos) | etcd or ZooKeeper for distributed locks and leader election |\n| **Graceful degradation** | Circuit breaker | Resilience4j: open circuit after 50% failures in 10-second window |\n\nSee [references\u002Ffault-tolerance.md](references\u002Ffault-tolerance.md) when tuning timeouts\u002Fretries or adding consensus -- adds fault classification, timeout-tuning math, Raft\u002FPaxos mechanics, and safety\u002Fliveness guarantees.\n\n## Common Mistakes\n\n| Mistake | Why It Fails | Fix |\n|---------|-------------|------|\n| **Choosing a database by popularity** | Engines have fundamentally different trade-offs | Match storage engine to actual read\u002Fwrite patterns |\n| **Ignoring replication lag** | Stale reads, phantom reads, lost updates | Implement read-your-writes and monotonic-read guarantees |\n| **Distributed transactions everywhere** | 2PC is slow, fragile; coordinator is a SPOF | Design single-partition operations; use sagas across services |\n| **Hash partitioning everything** | Destroys range query ability | Key-range partitioning for time-series; composite keys for locality |\n| **Assuming serializable isolation** | Defaults are weaker; write skew appears in production | Check the actual default; use explicit locking where needed |\n| **Conflating batch and stream** | Wrong tool adds latency or wasted complexity | Match processing model to data boundedness and latency needs |\n| **Treating all faults as recoverable** | Corruption and Byzantine faults need different handling | Classify faults; design a recovery strategy per class |\n\n## Quick Diagnostic\n\n| Question | If No | Action |\n|----------|-------|--------|\n| Can you explain why you chose this database over alternatives? | Choice was familiarity, not requirements | Evaluate data model fit, read\u002Fwrite ratio, consistency needs, scaling path |\n| Do you know your database's default isolation level? | Latent concurrency bugs | Check docs; test for write skew and phantom reads |\n| Is your replication strategy explicitly chosen? | Implicit consistency\u002Fdurability assumptions | Document sync vs async, failover behavior, lag tolerance |\n| Can your system handle a hot partition key? | One popular entity can down the cluster | Add key-splitting or load shedding for hot keys |\n| Do you separate system of record from derived data? | Every change requires migrating everything | Introduce CDC or event sourcing to decouple |\n| Are timeouts and retries tuned, not defaulted? | Cascading failures or needless delays | Measure p99; set timeouts above p99, below cascade threshold |\n| Have you tested failover in production conditions? | Recovery plan is theoretical | Run chaos experiments: kill leaders, partition networks, fill disks |\n\n## Further Reading\n\nFor the complete treatment with detailed diagrams and research references:\n\n- [*\"Designing Data-Intensive Applications\"*](https:\u002F\u002Fwww.amazon.com\u002FDesigning-Data-Intensive-Applications-Reliable-Maintainable\u002Fdp\u002F1449373321?tag=wondelai00-20) by Martin Kleppmann\n\n## About the Author\n\n**Martin Kleppmann** is a distributed-systems researcher at the University of Cambridge and a former engineer at LinkedIn and Rapportive, known for his work on CRDTs and local-first software. His book *Designing Data-Intensive Applications* (2017) is the definitive reference for engineers building data systems, praised for making distributed-systems concepts accessible and practical.\n",{"name":5,"license":15,"metadata":24,"description":7},{"author":25,"version":26},"wondelai","1.4.0","main","ba06aff3a0c008524409a6bdcf301dfe4e8f165b",".agents\u002Fskills\u002Fwondelai-skills\u002F.windsurf\u002Fskills\u002Fddia-systems","https:\u002F\u002Fgithub.com\u002FDiv0011\u002FFrontend-Pipeline","https:\u002F\u002Fgithub.com\u002FDiv0011","B.Tech Computer Engineering undergrad & Full-Stack Developer. Specialized in Generative AI, Next.js, and Python architecture. Building autonomous systems.","2026-09-07 19:51:50",{"source":8,"skillName":5,"commands":35,"prompt":45,"zipAvailable":46},[36,39,42],{"tool":37,"command":38},"npx","npx skills add Div0011\u002FFrontend-Pipeline --skill ddia-systems",{"tool":40,"command":41},"bunx","bunx skills add Div0011\u002FFrontend-Pipeline --skill ddia-systems",{"tool":43,"command":44},"pnpm","pnpm dlx skills add Div0011\u002FFrontend-Pipeline --skill ddia-systems","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002FDiv0011\u002FFrontend-Pipeline\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fwondelai-skills\u002F.windsurf\u002Fskills\u002Fddia-systems\u002FSKILL.md",true,[48,54,58,62,66,70,74],{"path":49,"resourceType":50,"resourceTypeDesc":51,"sizeBytes":52,"sourceHtmlUrl":53},"references\u002Fbatch-stream.md","REFERENCE","参考文档",14302,"https:\u002F\u002Fgithub.com\u002FDiv0011\u002FFrontend-Pipeline\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fwondelai-skills\u002F.windsurf\u002Fskills\u002Fddia-systems\u002Freferences\u002Fbatch-stream.md",{"path":55,"resourceType":50,"resourceTypeDesc":51,"sizeBytes":56,"sourceHtmlUrl":57},"references\u002Fdata-models.md",11808,"https:\u002F\u002Fgithub.com\u002FDiv0011\u002FFrontend-Pipeline\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fwondelai-skills\u002F.windsurf\u002Fskills\u002Fddia-systems\u002Freferences\u002Fdata-models.md",{"path":59,"resourceType":50,"resourceTypeDesc":51,"sizeBytes":60,"sourceHtmlUrl":61},"references\u002Ffault-tolerance.md",15239,"https:\u002F\u002Fgithub.com\u002FDiv0011\u002FFrontend-Pipeline\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fwondelai-skills\u002F.windsurf\u002Fskills\u002Fddia-systems\u002Freferences\u002Ffault-tolerance.md",{"path":63,"resourceType":50,"resourceTypeDesc":51,"sizeBytes":64,"sourceHtmlUrl":65},"references\u002Fpartitioning.md",11051,"https:\u002F\u002Fgithub.com\u002FDiv0011\u002FFrontend-Pipeline\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fwondelai-skills\u002F.windsurf\u002Fskills\u002Fddia-systems\u002Freferences\u002Fpartitioning.md",{"path":67,"resourceType":50,"resourceTypeDesc":51,"sizeBytes":68,"sourceHtmlUrl":69},"references\u002Freplication.md",10958,"https:\u002F\u002Fgithub.com\u002FDiv0011\u002FFrontend-Pipeline\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fwondelai-skills\u002F.windsurf\u002Fskills\u002Fddia-systems\u002Freferences\u002Freplication.md",{"path":71,"resourceType":50,"resourceTypeDesc":51,"sizeBytes":72,"sourceHtmlUrl":73},"references\u002Fstorage-engines.md",11313,"https:\u002F\u002Fgithub.com\u002FDiv0011\u002FFrontend-Pipeline\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fwondelai-skills\u002F.windsurf\u002Fskills\u002Fddia-systems\u002Freferences\u002Fstorage-engines.md",{"path":75,"resourceType":50,"resourceTypeDesc":51,"sizeBytes":76,"sourceHtmlUrl":77},"references\u002Ftransactions.md",13126,"https:\u002F\u002Fgithub.com\u002FDiv0011\u002FFrontend-Pipeline\u002Fblob\u002Fmain\u002F.agents\u002Fskills\u002Fwondelai-skills\u002F.windsurf\u002Fskills\u002Fddia-systems\u002Freferences\u002Ftransactions.md",[79,88,95,102,109,116],{"skillNo":80,"name":81,"nameZh":6,"description":82,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":83,"licenseFileUrl":84,"sourceHtmlUrl":85,"categories":86,"keywords":87,"publishedAt":19},"01M7B9FY1YN4AX6XSKFY1GEB6F","scrapling-official","Scrape web pages using Scrapling with anti-bot bypass (like Cloudflare Turnstile), stealth headless browsing, spiders framework, adaptive scraping, and JavaScript rendering. Use when asked to scrape, crawl, or extract data from websites; web_fetch fails; the site has anti-bot protections; write Python code to scrape\u002Fcrawl; or write spiders.","BSD-3-Clause","https:\u002F\u002Fgithub.com\u002FDiv0011\u002FFrontend-Pipeline\u002Fblob\u002Fmain\u002Fartifacts\u002Fvendor_references\u002FScrapling\u002Fagent-skill\u002FScrapling-Skill\u002FLICENSE.txt","https:\u002F\u002Fgithub.com\u002FDiv0011\u002FFrontend-Pipeline\u002Fblob\u002Fmain\u002Fartifacts\u002Fvendor_references\u002FScrapling\u002Fagent-skill\u002FScrapling-Skill\u002FSKILL.md",[],[],{"skillNo":89,"name":90,"nameZh":6,"description":91,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":92,"categories":93,"keywords":94,"publishedAt":19},"01M7B9FNH5E90KCTDK1MABVBAX","working-with-legacy-code","Safely change and test untested codebases using Feathers' \"Working Effectively with Legacy Code\". Use when the user mentions \"legacy code\", \"no tests\", \"untested codebase\", \"how do I test this\", \"seams\", \"characterization tests\", \"golden master\", \"sprout method\", \"afraid to change this code\", \"monster method\", \"dependency breaking\", or \"inherited a messy codebase\". Also trigger when changing code without tests safely, getting a class under test when constructors, statics, or singletons block it, adding features to tangled modules, or planning incremental test coverage for an old codebase. Covers the legacy-code change algorithm, seams, characterization tests, sprout\u002Fwrap, and dependency-breaking techniques. For refactoring code that already has tests, see refactoring-patterns. For day-to-day code quality, see clean-code.","https:\u002F\u002Fgithub.com\u002FDiv0011\u002FFrontend-Pipeline\u002Fblob\u002Fmain\u002Fartifacts\u002Fskills\u002Fwondelai-skills\u002Fworking-with-legacy-code\u002FSKILL.md",[],[],{"skillNo":96,"name":97,"nameZh":6,"description":98,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":99,"categories":100,"keywords":101,"publishedAt":19},"01M7B9FDRG6DG9PKTEMKZXHCXY","web-typography","Select, pair, and implement typefaces for web projects. Use when the user mentions \"font pairing\", \"which typeface\", \"line height\", \"responsive typography\", \"web font loading\", \"type hierarchy\", \"variable fonts\", \"FOUT\u002FFOIT\", \"typographic scale\", or \"the text is hard to read\". Also trigger when choosing between system fonts and web fonts, optimizing font-loading performance, or designing readable long-form content. Covers readability evaluation, CSS implementation, and performance optimization. For overall UI design systems, see refactoring-ui. For dramatic typographic experiences, see top-design.","https:\u002F\u002Fgithub.com\u002FDiv0011\u002FFrontend-Pipeline\u002Fblob\u002Fmain\u002Fartifacts\u002Fskills\u002Fwondelai-skills\u002Fweb-typography\u002FSKILL.md",[],[],{"skillNo":103,"name":104,"nameZh":6,"description":105,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":106,"categories":107,"keywords":108,"publishedAt":19},"01M7B9F6P2Y8R413YJ3WX339HH","ux-heuristics","Evaluate and improve interface usability using heuristic analysis. Use when the user mentions \"usability audit\", \"users are confused\", \"form usability\", \"navigation problems\", \"Nielsen heuristics\", \"cognitive walkthrough\", or \"is this easy to use\". Also trigger when reviewing a design for usability issues, improving form-completion rates, or evaluating information architecture and navigation. Covers Krug's laws, Nielsen's 10 heuristics, severity ratings, dark-pattern recognition, and accessibility. For visual design fixes, see refactoring-ui. For conversion-focused audits, see cro-methodology.","https:\u002F\u002Fgithub.com\u002FDiv0011\u002FFrontend-Pipeline\u002Fblob\u002Fmain\u002Fartifacts\u002Fskills\u002Fwondelai-skills\u002Fux-heuristics\u002FSKILL.md",[],[],{"skillNo":110,"name":111,"nameZh":6,"description":112,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":113,"categories":114,"keywords":115,"publishedAt":19},"01M7B9EZVNSBCXJFQRFVX29NR5","traction-eos","Implement the Entrepreneurial Operating System (EOS) to align vision and execution across a company. Use when the user mentions \"EOS\", \"Entrepreneurial Operating System\", \"V\u002FTO\", \"quarterly rocks\", \"Level 10 meetings\", \"accountability chart\", \"IDS process\", \"my company feels chaotic\", \"we keep having the same problems\", or \"get the whole team aligned\". Also trigger when a growing company needs meeting structure, goal-setting frameworks, or a systematic way to solve recurring organizational issues. Covers the six EOS components: Vision, People, Data, Issues, Process, Traction. For team motivation design, see drive-motivation. For lean experimentation, see lean-startup.","https:\u002F\u002Fgithub.com\u002FDiv0011\u002FFrontend-Pipeline\u002Fblob\u002Fmain\u002Fartifacts\u002Fskills\u002Fwondelai-skills\u002Ftraction-eos\u002FSKILL.md",[],[],{"skillNo":117,"name":118,"nameZh":6,"description":119,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":120,"categories":121,"keywords":122,"publishedAt":19},"01M7B9ESTNAY9WY8NFW590DWGA","top-design","Create award-winning, immersive web experiences at the level of Awwwards-featured agencies. Use when the user mentions \"Awwwards quality\", \"make my site stunning\", \"scroll animations\", \"parallax storytelling\", \"cinematic web design\", \"portfolio site\", or \"brand experience\". Also trigger when elevating a standard landing page into a memorable digital experience. Covers dramatic typography, purposeful motion, scroll-based composition, and performance-optimized animation. For foundational UI, see refactoring-ui. For type selection, see web-typography.","https:\u002F\u002Fgithub.com\u002FDiv0011\u002FFrontend-Pipeline\u002Fblob\u002Fmain\u002Fartifacts\u002Fskills\u002Fwondelai-skills\u002Ftop-design\u002FSKILL.md",[],[]]