multi-region-design
Design multi-region deployment — active-active vs standby, RPO/RTO, replication and conflicts, routing, failover. Not single-region HA.
- License 许可
- MIT
- 最近更新
- 2026-10-29
- Stars
- ★ 0
- Forks
- 0
01 前言 Front Matter
- name
- multi-region-design
- description
- Design multi-region deployment — active-active vs standby, RPO/RTO, replication and conflicts, routing, failover. Not single-region HA.
03 SKILL.md 正文
multi-region-design
Going multi-region trades money, latency, and consistency for survivability and locality. Decide what failure you're buying protection from, then pick the cheapest topology that meets the RPO/RTO — and rehearse the failover.
Mode: analytical — judgment inside a fixed frame. Keep the output shape; every verdict cites its evidence; unknowns are marked [TBD] or [VERIFY], never filled in.
When to use
- A requirement for regional-outage survival, a stricter availability SLO than one region can give, or lower latency for global users.
- Data residency / sovereignty requires data to stay in specific regions.
- Reviewing an existing "multi-region" setup that has never been failed over.
Not this when: multi-AZ high availability within a region meets the target — polymath-cloud:design-aws-pattern / design-azure-pattern / design-gcp-pattern; or only DR backups are needed — define RPO/RTO, then restore tests (polymath-sre:chaos-experiment).
Inputs
- The driver: SLO (
polymath-sre:slo-design), RTO/RPO per data class, latency target by user region, residency rules, regulatory constraints, and the cost envelope. - Current architecture: stateless tiers, datastores and their replication features, queues/streams, caches, DNS/CDN, third-party dependencies and their regionality.
- The consistency needs of each write: what must be globally unique, what can conflict and merge, what may be stale.
- Historical regional incidents and the dependency map (control planes, identity, DNS, CI/CD, secrets).
Procedure
- Justify it with numbers. Regional outages are rarer than software failures; compare the availability gain to cost and operational complexity. If single-region multi-AZ plus tested backups meets RTO/RPO, stop. Record which failures the design must survive (AZ, region, provider-wide, control plane, bad deploy — multi-region doesn't fix a bad deploy that replicates everywhere).
- Set RPO/RTO per data class (not per system): money movement RPO≈0 (synchronous or no loss), catalog data minutes, analytics hours. RPO ≈ replication lag at failure; RTO = detection + decision + failover + warm-up.
- Choose the topology per tier:
- Backup & restore (RTO hours, RPO hours; cheapest).
- Pilot light / warm standby (RTO minutes–tens of minutes; data replicated, compute scaled down or minimal; scale on failover).
- Active-passive hot standby (RTO minutes; full capacity idle; failover automated or one-button).
- Active-active (RTO≈0 for the surviving regions; both serve traffic). Requires conflict-free data design and traffic partitioning; highest cost and complexity — justify against warm standby.
- Design data replication per store — the hard part:
- Single-writer + async replicas: simple; RPO = lag > 0; promotion needed; cross-region writes pay latency to the leader (or route writes to home region).
- Synchronous cross-region commit (Spanner, CockroachDB, Aurora global with managed switchover): near-zero RPO, but every commit pays inter-region RTT (tens–hundreds of ms); watch quorum availability if a region is lost.
- Multi-writer / active-active (DynamoDB global tables, Cosmos DB multi-region writes, Cassandra multi-DC): conflicts happen — define resolution (last-writer-wins by timestamp loses data; per-field merge, CRDTs, or home-region-per-entity avoid it) and which operations can never be multi-writer (balances, inventory, uniqueness).
- Partition by home region (each user/tenant has one home region for writes; others read replicas or route): avoids conflicts, satisfies residency, makes failover a per-partition decision.
- Events/streams: mirror topics (MirrorMaker 2, Cluster Linking, Event Hubs geo-replication) with consumer-offset translation; plan for duplicates and reordering across the failover boundary (idempotent consumers).
- Caches are rebuilt or regionally local; never the replicated source of truth. Object storage uses cross-region replication; note its RPO is not zero.
- Traffic routing. Global DNS (health-checked, low TTL — remember resolvers and clients that ignore TTL), anycast/global load balancer, or CDN origin failover. Define the health check (a deep check on dependencies, but not so deep that a minor dependency failure flaps the region), automatic vs human-approved failover (automatic failover of a database can cause split-brain/data loss — require a fencing mechanism), and capacity: can the survivor absorb 100% of traffic (N+1 at region level, often ≥ 50–100% headroom, or planned load shedding)?
- Eliminate hidden single-region dependencies: identity provider, secrets, config, feature flags, CI/CD to deploy the fix, DNS provider, observability, payment/SMS providers, container registries, and any "global" service that's actually homed in one region. Draw the dependency map and mark each regional/global/unknown.
- Failover and failback runbook. Detection → decision criteria and who decides → steps (promote DB, shift traffic, scale up, drain queues) → verification → communication (
polymath-chat:post-statuspage-update) → failback (re-seed old primary, reverse replication, catch up, switch traffic back in a window). Failback is a second migration — design it, don't assume it. - Rehearse. Game days with real traffic shift (
polymath-sre:chaos-experiment); measure actual RTO and data loss, not designed values; test split-brain fencing and the "region comes back with stale data" case. A failover never exercised will not work. - Residency and compliance. Map data classes to allowed regions; keep PII home-region-only where required; check that logs, backups, analytics copies, and support tooling don't export it; encryption keys per region.
- Cost and operations. Duplicate compute, cross-region data transfer, replicated storage, double observability; the added deploy/config/migration complexity (schema changes across regions, versions skew during rollout); on-call load. State the incremental cost and compare to the cost of the outage it prevents.
Output
Multi-region design: payments API
Driver: SLO 99.99% (regional outage budget: ~4 min/month); RTO 15 min / RPO 0 for ledger, 5 min for the rest; EU data stays in EU.
Failures covered: region outage, AZ loss. Not covered: bad deploy (canary + rollback — safe-rollout), provider-wide outage.
Topology: active-active for reads/API, single-writer per home region for the ledger.
Stateless: us-east + eu-west, each sized for 100% of its own traffic + 60% of the other's [VERIFY: load test].
Ledger: Postgres per region (home-region partitioning by customer.region); sync replica in a second AZ; async cross-region replica for DR (RPO ≈ lag, p99 2 s) → RPO 0 not met cross-region [TBD: decision — accept ~2 s RPO, or use a synchronous-commit engine and pay the commit latency].
Catalog/prefs: DynamoDB global tables (LWW acceptable; per-field merge for preferences).
Events: Kafka per region, MirrorMaker 2 mirroring; consumers idempotent; offsets translated on failover.
Cache: regional, rebuilt on failover.
Routing: global LB with deep health check (DB + queue) hysteresis 3/5; DNS TTL 30 s as fallback; automatic failover for stateless, human-approved promote for ledger with fencing (old primary demoted by revoking its storage credentials).
Hidden dependencies: IdP (global, multi-region), secrets (replicated), feature flags (cached last-known-good), CI/CD (runs from eu-west), SMS provider (single region → fallback provider).
Failover runbook: detect (3 min) → decide (5 min) → promote + shift (5 min) → verify (2 min) = 15 min target; failback in a planned window after re-seed.
Rehearsal: quarterly game day; last result [TBD]. Residency: EU PII only in eu-west stores; logs scrubbed; per-region KMS keys.
Cost: +[TBD: infra delta] (compute ×1.6, inter-region transfer, double observability) vs outage cost [TBD].
Open: [VERIFY: provider rate limits per region], [TBD: ledger RPO decision with finance].
Quality bar
- The need is justified against cheaper options, and the covered failure types are listed (and the uncovered ones).
- RPO/RTO are per data class, and each store's replication model states its real RPO and conflict behavior.
- Hidden single-region dependencies are mapped; survivor capacity is stated.
- Failover has fencing/split-brain handling, a failback plan, and a rehearsal record or schedule.
- Residency and cost are explicit.
Anti-patterns to avoid
- Active-active by default, without a conflict-resolution design — silent data loss with last-writer-wins on money or inventory.
- A "DR region" that was never failed over, or whose capacity is a fraction of production.
- Automatic database failover without fencing; relying on DNS TTL alone for failover timing.
- Shipping multi-region while identity, secrets, CI/CD, or DNS live in one region.
- Treating multi-region as a substitute for safe deploys.
- Ignoring cross-region latency: synchronous calls between regions in the request path.
04 同仓库更多 Skills
runbook
Write or refresh an operational runbook for an alert/symptom; structured as TL;DR + Pre-checks + Steps + Verification + Rollback + Escalation.
rfc
Draft a lightweight design RFC to docs/rfcs/<slug>.md; Summary, Motivation, Detailed design, Drawbacks, Alternatives, Unresolved questions. A proposal to debate, not a recorded decision (adr).
editorial-pass
Review and tighten prose in a Markdown artifact for a target tone (terse, formal, friendly) without dropping information; preserve quotations and load-bearing constraints.
architecture-review-facilitation
Run an architecture review as a process — entry criteria, reviewer checklist, decision log, and follow-up actions — pairing the design artifact with a red-team pass.
architecture-doc
Write a C4-style architecture document; System Context + Containers (+ Components when interesting) with mermaid diagrams. Describes structure, not a decision (adr) or proposal (rfc).
adr
Write an Architecture Decision Record (Michael Nygard format) to docs/adrs/NNNN-<slug>.md; frontmatter validated by the ADR artifact schema. Records a made decision, not a proposal to debate (rfc).