[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill:01M7AAZFP48XBSTVS2TH3F0Y18":3,"skill-same:01M7AAZFP48XBSTVS2TH3F0Y18":153},{"skillNo":4,"name":5,"nameZh":6,"description":7,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":16,"categories":17,"keywords":18,"publishedAt":19,"linkOnly":20,"markdownContent":21,"markdownBody":22,"frontMatter":23,"refName":45,"commitSha":46,"skillDirectory":47,"forksCount":48,"repositoryHtmlUrl":49,"repositoryHomepageUrl":6,"ownerHtmlUrl":50,"ownerBlogUrl":6,"ownerDescription":51,"updatedAt":52,"install":53,"resources":66},"01M7AAZFP48XBSTVS2TH3F0Y18","rds-failover-troubleshooter",null,"Diagnoses RDS and Aurora failover issues through a thirteen-category diagnostic tree: Multi-AZ failover not triggering (health check threshold too conservative), failover taking too long (DNS propagation, recovery mode full), Aurora writer endpoint not redirecting (application using instance endpoint instead of cluster endpoint), read replica promotion failures (network connectivity, storage), storage-full preventing failover (allocated storage exhausted), parameter group mismatch between primary and standby, option group conflicts preventing standby from taking over, Aurora cluster endpoint type confusion (writer vs reader vs custom vs instance), failover priority tier misconfiguration (tier 0 vs tier 1 not set on the intended promotion target), Aurora Global DB failover (managed vs unmanaged), RDS Proxy failover behavior (connection pool not draining), application connection string not using cluster endpoint, and CloudWatch failover event gaps. Walks symptoms to a verified root cause with...","jhkchan\u002Faws-agent-skills","jhkchan","https:\u002F\u002Favatars.githubusercontent.com\u002Fu\u002F1083902?v=4","01M7A2SVQYP9GQTPTDXP5JKYQ4","Jacky Chan","01M7A2SVS117YWCTZABB75HZR1",4,"Apache-2.0","https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Frds-failover-troubleshooter\u002FSKILL.md",[],[],"2026-11-15 14:19:17",false,"---\nname: rds-failover-troubleshooter\ndescription: 'Diagnoses RDS and Aurora failover issues through a thirteen-category diagnostic tree: Multi-AZ failover not triggering (health check threshold too conservative), failover taking too long (DNS propagation, recovery mode full), Aurora writer endpoint not redirecting (application using instance endpoint instead of cluster endpoint), read replica promotion failures (network connectivity, storage), storage-full preventing failover (allocated storage exhausted), parameter group mismatch between primary and standby, option group conflicts preventing standby from taking over, Aurora cluster endpoint type confusion (writer vs reader vs custom vs instance), failover priority tier misconfiguration (tier 0 vs tier 1 not set on the intended promotion target), Aurora Global DB failover (managed vs unmanaged), RDS Proxy failover behavior (connection pool not draining), application connection string not using cluster endpoint, and CloudWatch failover event gaps. Walks symptoms to a verified root cause with...'\nlicense: Apache-2.0\ncompatibility: Agent runtime that reads SKILL.md (Claude Code, Cursor, Windsurf, Codex, Gemini). Offline symptom classification works from pasted error messages and cluster configuration. Live-account diagnosis uses aws rds describe-db-clusters, aws rds describe-db-instances, aws rds describe-db-cluster-endpoints, aws rds describe-db-cluster-parameter-groups, aws rds describe-events, aws cloudwatch get-metric-statistics, aws ec2 describe-route-tables, aws ec2 describe-security-groups, and aws logs...\nmetadata:\n  domain: aws-cloudops\n  complexity: high\n  requires_llm: 'true'\n  phase: '2'\n  supports_pipeline: 'true'\n  entry_point: 'false'\n  family: Databases\n  task_type: troubleshoot\n  skill_class: capability\n  lifecycle_status: active\n  verdict_shape: ROOT_CAUSE_IDENTIFIED | INSUFFICIENT_DATA\n  when_to_use: Diagnosing an RDS or Aurora failover issue (Multi-AZ failover not triggering, failover taking too long, writer endpoint not redirecting, read replica promotion failure, storage-full preventing failover, parameter group mismatch, option group conflict, failover priority misconfiguration, Aurora Global DB failover, RDS Proxy failover, application connection issues), walking a symptom to the failed layer with verify commands, or triaging a \"failover did not work\" incident where the root cause may be endpoint configuration, health threshold, storage, parameter group, option group, failover tier, or application connection string — not necessarily the database engine itself.\n  when_not_to_use: RDS connectivity debugging (use rds-connectivity-troubleshooter), RDS performance tuning (use rds-cost-optimizer for right-sizing), Aurora backup and restore (use rds-backup-restore-operator), or RDS instance creation (use rds-instance-deployer). This skill diagnoses failover-time issues; it does not tune query performance or audit steady-state security posture.\n  activation_triggers: RDS failover not working, Aurora failover not triggering, Multi-AZ failover failed, Aurora writer endpoint not redirecting, RDS failover taking too long, Aurora failover priority, read replica promotion failed, storage-full failover, parameter group mismatch failover, option group conflict failover, Aurora Global DB failover, RDS Proxy failover, application not connecting after failover, Aurora cluster endpoint confusion, troubleshoot RDS failover\n  invocation_schema: 'Input: either (a) a symptom description (error message, observed behaviour, \"failover took 5 minutes\", \"writer endpoint still points to old instance\"), optionally paired with the cluster configuration and recent RDS events, OR (b) a DB cluster identifier plus caller context (application connection string, observed error) for live-account diagnosis. Output: a deterministic TARGET\u002FVERDICT\u002FREASON\u002FROOT_CAUSE\u002FEVIDENCE\u002FREMEDIATION block where VERDICT ∈ {ROOT_CAUSE_IDENTIFIED, INSUFFICIENT_DATA} and ROOT_CAUSE ∈ {APP_NOT_CLUSTER_ENDPOINT, MULTI_AZ_HEALTH_THRESHOLD, FAILOVER_PRIORITY_TIER, STORAGE_FULL, PARAMETER_GROUP_MISMATCH, OPTION_GROUP_CONFLICT, DNS_PROPAGATION_DELAY, CONNECTION_POOL_CACHING, AURORA_GLOBAL_FAILOVER_MODE, RDS_PROXY_POOL_DRAIN, RECOVERY_MODE_FULL, CLOUDWATCH_EVENT_GAP, READ_REPLICA_PROMOTION_FAILURE, UNKNOWN}.'\n  invocation_example: '# Minimal valid input (offline symptom classification):\n\n    Symptom: \"Aurora PostgreSQL cluster prod-db-cluster\n\n    failed over automatically at 03:17 UTC. The writer\n\n    endpoint resolved to the new writer within 1 second,\n\n    but the application continued hitting the old writer\n\n    for 90 seconds, then connection errors cleared.\"\n\n    DBClusterIdentifier: prod-db-cluster\n\n    Engine: aurora-postgresql\n\n    WriterEndpoint: prod-db-cluster.cluster-xxxxxxxxxxxx.us-east-1.rds.amazonaws.com\n\n    Application connection string: uses the instance endpoint\n\n    prod-db-cluster-instance-1.xxxxxxxxxxxx.us-east-1.rds.amazonaws.com\n\n    instead of the cluster endpoint'\n  version: 0.1.0\n  author: Jacky Chan — AWS Community Builder\n  keywords: RDS, Aurora, failover, Multi-AZ, writer endpoint, cluster endpoint, read replica promotion, storage-full, parameter group mismatch, option group conflict, failover priority, tier 0, Aurora Global DB, RDS Proxy, connection pool, DNS propagation, CloudWatch failover events, troubleshooting\n  tags: rds, aurora, databases, troubleshooting, failover, multi-az, high-availability, cluster-endpoint, rds-proxy\n---\n\n# RDS Failover Troubleshooter\n\n## Quick start\n\n> Moved verbatim to [`references\u002Fadvanced-patterns.md`](references\u002Fadvanced-patterns.md) — load on demand.\n\n\n## Mindset\n\n> Moved verbatim to [`references\u002Fadvanced-patterns.md`](references\u002Fadvanced-patterns.md) — load on demand.\n\n\n## Philosophy\n\n> Moved verbatim to [`references\u002Fadvanced-patterns.md`](references\u002Fadvanced-patterns.md) — load on demand.\n\n\n## Quick reference — symptom triage table\n\n| Symptom phrase \u002F error | Most likely root cause | First probe |\n|---|---|---|\n| Failover triggered but app still hits old instance | APP_NOT_CLUSTER_ENDPOINT | Check application connection string for instance vs cluster endpoint |\n| App recovers 2-5 minutes after failover despite DNS TTL 1s | CONNECTION_POOL_CACHING | Check DB driver connection pool TTL, JVM DNS cache (`networkaddress.cache.ttl`) |\n| Failover did not trigger at all | MULTI_AZ_HEALTH_THRESHOLD | `describe-db-instances` for Multi-AZ config; CloudWatch `DatabaseConnection` and `DiskQueueDepth` metrics |\n| New writer is not the expected instance | FAILOVER_PRIORITY_TIER | `describe-db-instances` for `PromotionTier` on each instance |\n| `InsufficientStorage` or `FailoverFailed` | STORAGE_FULL | `describe-db-instances` for `StorageAllocated` vs `StorageUsed`; CloudWatch `FreeStorageSpace` |\n| Standby cannot take over; error in RDS events | PARAMETER_GROUP_MISMATCH or OPTION_GROUP_CONFLICT | `describe-db-instances` for `DBParameterGroups` and `OptionGroupMemberships` on primary vs standby |\n| Aurora writer endpoint not resolving to new writer | DNS_PROPAGATION_DELAY | `nslookup` or `dig` on the cluster endpoint; check DNS resolver cache |\n| Read replica promotion fails | READ_REPLICA_PROMOTION_FAILURE | `describe-db-instances` for replica status; check network connectivity and storage |\n| Aurora Global DB failover not working | AURORA_GLOBAL_FAILOVER_MODE | `describe-global-clusters`; check if managed or unmanaged failover |\n| RDS Proxy not draining connections | RDS_PROXY_POOL_DRAIN | `describe-db-proxies`; CloudWatch `DatabaseConnections` metric on the proxy |\n| Recovery mode causes long failover | RECOVERY_MODE_FULL | Check parameter group for `recovery_mode` or Aurora parallel recovery settings |\n| No CloudWatch event for failover | CLOUDWATCH_EVENT_GAP | `describe-events` for the cluster; check EventBridge rule for RDS events |\n\n## Pre-flight: gather the failover context\n\nBefore running root-cause-specific probes, gather the canonical cluster\nconfiguration and short-circuit on misconfigurations that mimic\nfailover failures.\n\n### Account-wide pre-flight commands\n\n> Moved verbatim to [`references\u002Fdiagnostic-commands.md`](references\u002Fdiagnostic-commands.md) — load on demand.\n\n\n### Cluster-state short-circuit\n\n| Finding | Effect on diagnosis |\n|---|---|\n| Cluster `Status: available`, writer endpoint resolves to a healthy instance | Failover completed at the DB layer. Investigate application-side issues (endpoint, connection pool). |\n| Cluster `Status: modifying` or `Status: upgrading` | A maintenance action is in progress. Failover may be delayed or blocked. Wait for completion. |\n| Instance `Status: storage-full` | The instance has exhausted allocated storage. Failover cannot complete. **ROOT_CAUSE_IDENTIFIED** with `ROOT_CAUSE: STORAGE_FULL`. |\n| Instance `MultiAZ: false` | The instance is not Multi-AZ enabled. No standby exists. Failover cannot occur. |\n| Instance `PromotionTier` on intended target > 0 | A lower-tier instance will be promoted first. Check if another instance has tier 0. |\n| Parameter group `status: applying` or `pending-reboot` | A parameter change is pending. The standby may have a different effective configuration. |\n| Aurora Global cluster with `GlobalClusterIdentifier` set | The cluster is part of a Global DB. Failover semantics differ (managed vs unmanaged). |\n\n> Moved verbatim to [`references\u002Ferror-handling.md`](references\u002Ferror-handling.md) — load on demand.\n\n\n## Process — Diagnostic decision tree (apply in symptom order)\n\nThe diagnostic tree is symptom-driven. Pick the entry point based on\nthe observed symptom, then walk the root-cause-specific probes in\norder.\n\n### Step 0: Non-obvious behaviours that change diagnosis\n\n> Moved verbatim to [`references\u002Fadvanced-patterns.md`](references\u002Fadvanced-patterns.md) — load on demand.\n\n\n### Step 1: Application connection string check\n\nSymptom: failover completed (new writer is active, cluster endpoint\nresolves correctly), but the application still cannot connect.\n\n> Moved verbatim to [`references\u002Fdiagnostic-commands.md`](references\u002Fdiagnostic-commands.md) — load on demand.\n\n\nIf the application uses the instance endpoint\n(`\u003Ccluster>-instance-1.\u003Cid>.\u003Cregion>.rds.amazonaws.com`) instead of\nthe cluster endpoint\n(`\u003Ccluster>.cluster-\u003Cid>.\u003Cregion>.rds.amazonaws.com`),\n**ROOT_CAUSE_IDENTIFIED** with `ROOT_CAUSE: APP_NOT_CLUSTER_ENDPOINT`.\n\nThe instance endpoint does not follow the writer after failover. The\napplication must use the cluster endpoint (also called the writer\nendpoint) for transparent failover.\n\n### Step 2: DNS propagation check\n\nSymptom: application uses the cluster endpoint but still hits the old\ninstance for 30+ seconds after failover.\n\n> Moved verbatim to [`references\u002Fdiagnostic-commands.md`](references\u002Fdiagnostic-commands.md) — load on demand.\n\n\nCheck the application's DNS caching layers:\n\n> Moved verbatim to [`references\u002Ferror-handling.md`](references\u002Ferror-handling.md) — load on demand.\n\n\nIf a caching layer is holding the old IP,\n**ROOT_CAUSE_IDENTIFIED** with `ROOT_CAUSE: DNS_PROPAGATION_DELAY`\n(OS\u002FJVM DNS cache) or `ROOT_CAUSE: CONNECTION_POOL_CACHING`\n(connection pool).\n\n### Step 3: Multi-AZ health check threshold\n\nSymptom: primary instance is unhealthy (high CPU, stuck queries) but\nfailover does not trigger.\n\n> Moved verbatim to [`references\u002Fdiagnostic-commands.md`](references\u002Fdiagnostic-commands.md) — load on demand.\n\n\nMulti-AZ failover triggers when:\n- The primary instance becomes unreachable (network partition).\n- The instance status changes to `failed`.\n- The storage subsystem becomes unresponsive.\n\nMulti-AZ does NOT trigger on:\n- High CPU (even at 100%).\n- High connection count.\n- Slow queries.\n\nIf the primary is degraded but responsive (high CPU, slow queries),\nMulti-AZ will not failover. **ROOT_CAUSE_IDENTIFIED** with\n`ROOT_CAUSE: MULTI_AZ_HEALTH_THRESHOLD`. The health check only triggers\non unreachability, not performance degradation.\n\n### Step 4: Failover priority tier\n\nSymptom: Aurora failover completed, but the \"wrong\" instance became\nthe new writer.\n\n> Moved verbatim to [`references\u002Fdiagnostic-commands.md`](references\u002Fdiagnostic-commands.md) — load on demand.\n\n\nAurora failover promotion order:\n1. Lowest `PromotionTier` (tier 0 first).\n2. Within the same tier, largest `DBInstanceClass`.\n3. Within the same tier and class, longest uptime.\n\nIf the intended promotion target has tier 15 (default) but another\ninstance has tier 0, the tier 0 instance is promoted.\n**ROOT_CAUSE_IDENTIFIED** with `ROOT_CAUSE: FAILOVER_PRIORITY_TIER`.\n\nFix: set the intended target to tier 0:\n\n> Moved verbatim to [`references\u002Fdiagnostic-commands.md`](references\u002Fdiagnostic-commands.md) — load on demand.\n\n\n### Step 5: Storage-full check\n\nSymptom: failover event appears in RDS events but the new writer does\nnot become available. Error: `InsufficientStorage` or\n`FailoverFailed`.\n\n> Moved verbatim to [`references\u002Fdiagnostic-commands.md`](references\u002Fdiagnostic-commands.md) — load on demand.\n\n\nIf `FreeStorageSpace` is near zero (\u003C 5 GB) or `DBInstanceStatus` is\n`storage-full`, **ROOT_CAUSE_IDENTIFIED** with\n`ROOT_CAUSE: STORAGE_FULL`.\n\nFix: increase allocated storage:\n\n> Moved verbatim to [`references\u002Fdiagnostic-commands.md`](references\u002Fdiagnostic-commands.md) — load on demand.\n\n\n### Step 6: Parameter group mismatch\n\nSymptom: standby instance fails health checks during promotion. RDS\nevents show a promotion failure with parameter-related errors.\n\n> Moved verbatim to [`references\u002Fdiagnostic-commands.md`](references\u002Fdiagnostic-commands.md) — load on demand.\n\n\nIf the primary and standby have different DB parameter groups with\nincompatible settings, the standby may fail during promotion.\n**ROOT_CAUSE_IDENTIFIED** with `ROOT_CAUSE: PARAMETER_GROUP_MISMATCH`.\n\nFix: align the parameter groups:\n\n> Moved verbatim to [`references\u002Fdiagnostic-commands.md`](references\u002Fdiagnostic-commands.md) — load on demand.\n\n\n### Step 7: Option group conflict\n\nSymptom: standby cannot start the engine after promotion. RDS events\nshow an option-group-related error.\n\n> Moved verbatim to [`references\u002Fdiagnostic-commands.md`](references\u002Fdiagnostic-commands.md) — load on demand.\n\n\nIf the option groups are incompatible (e.g., primary has TDE, standby\ndoes not), **ROOT_CAUSE_IDENTIFIED** with\n`ROOT_CAUSE: OPTION_GROUP_CONFLICT`.\n\nFix: align the option groups:\n\n> Moved verbatim to [`references\u002Fdiagnostic-commands.md`](references\u002Fdiagnostic-commands.md) — load on demand.\n\n\n### Step 8: Read replica promotion failure\n\nSymptom: a read replica (Aurora or cross-region) cannot be promoted\nto a standalone cluster\u002Finstance.\n\n> Moved verbatim to [`references\u002Fdiagnostic-commands.md`](references\u002Fdiagnostic-commands.md) — load on demand.\n\n\nCommon promotion failures:\n- High replication lag (the replica has not caught up).\n- Network connectivity to the source (for cross-region).\n- Storage-full on the replica.\n- Parameter group mismatch.\n\nIf any of these apply, **ROOT_CAUSE_IDENTIFIED** with\n`ROOT_CAUSE: READ_REPLICA_PROMOTION_FAILURE`.\n\n### Step 9: Aurora Global DB failover mode\n\nSymptom: Aurora Global DB failover did not work or took too long.\n\n> Moved verbatim to [`references\u002Fdiagnostic-commands.md`](references\u002Fdiagnostic-commands.md) — load on demand.\n\n\nManaged failover (`failover-global-cluster`) is a control-plane\noperation that takes 1-5 minutes. Unmanaged failover (detach secondary\n+ promote) is faster but requires manual coordination.\n\nIf the failover mode is misconfigured or the secondary cluster has\nhigh replication lag, **ROOT_CAUSE_IDENTIFIED** with\n`ROOT_CAUSE: AURORA_GLOBAL_FAILOVER_MODE`.\n\n### Step 10: RDS Proxy failover\n\nSymptom: RDS Proxy does not transparently handle failover; application\nsees connection errors.\n\n> Moved verbatim to [`references\u002Fdiagnostic-commands.md`](references\u002Fdiagnostic-commands.md) — load on demand.\n\n\nRDS Proxy should queue and reconnect during failover. If connections\nare being dropped instead of queued, check:\n- Proxy target group health.\n- `ConnectionBorrowTimeout` (client-side).\n- Proxy `IdleClientTimeout`.\n\nIf the Proxy is not draining\u002Fqueuing correctly,\n**ROOT_CAUSE_IDENTIFIED** with `ROOT_CAUSE: RDS_PROXY_POOL_DRAIN`.\n\n### Step 11: Recovery mode\n\nSymptom: Aurora failover takes 30+ seconds consistently; Multi-AZ\nfailover takes 120+ seconds.\n\nFor Aurora, check if parallel recovery is enabled (engine-version\ndependent):\n\n> Moved verbatim to [`references\u002Fdiagnostic-commands.md`](references\u002Fdiagnostic-commands.md) — load on demand.\n\n\nAurora PostgreSQL 13+ supports parallel WAL replay. Aurora MySQL 8.0\nhas optimised crash recovery. Older versions use serial recovery.\n\nFor Multi-AZ, check the `recovery_mode` parameter:\n\n> Moved verbatim to [`references\u002Fdiagnostic-commands.md`](references\u002Fdiagnostic-commands.md) — load on demand.\n\n\nIf recovery mode is set to `full` instead of `optimized`,\n**ROOT_CAUSE_IDENTIFIED** with `ROOT_CAUSE: RECOVERY_MODE_FULL`.\n\n### Step 12: CloudWatch failover event gap\n\nSymptom: failover occurred but no CloudWatch alarm or EventBridge rule\nfired.\n\n> Moved verbatim to [`references\u002Fdiagnostic-commands.md`](references\u002Fdiagnostic-commands.md) — load on demand.\n\n\n> Moved verbatim to [`references\u002Ferror-handling.md`](references\u002Ferror-handling.md) — load on demand.\n\n\nIf no events appear in `describe-events`, the failover may not have\ntriggered at the RDS service level. If events appear but no alarm\nfired, the EventBridge rule or CloudWatch alarm is misconfigured.\n**ROOT_CAUSE_IDENTIFIED** with `ROOT_CAUSE: CLOUDWATCH_EVENT_GAP`.\n\n## Output format\n\n```text\nTARGET: \u003Ccluster-id or instance-id, application context>\nVERDICT: ROOT_CAUSE_IDENTIFIED | INSUFFICIENT_DATA\nREASON: \u003C1-2 sentences naming the root cause and the failing probe>\nROOT_CAUSE: \u003CAPP_NOT_CLUSTER_ENDPOINT | MULTI_AZ_HEALTH_THRESHOLD |\n              FAILOVER_PRIORITY_TIER | STORAGE_FULL |\n              PARAMETER_GROUP_MISMATCH | OPTION_GROUP_CONFLICT |\n              DNS_PROPAGATION_DELAY | CONNECTION_POOL_CACHING |\n              AURORA_GLOBAL_FAILOVER_MODE | RDS_PROXY_POOL_DRAIN |\n              RECOVERY_MODE_FULL | CLOUDWATCH_EVENT_GAP |\n              READ_REPLICA_PROMOTION_FAILURE | UNKNOWN>\nEVIDENCE:\n  - \u003Cobserved symptom — error string, failover timeline>\n  - \u003Cfailing probe — command and its output that confirms the cause>\n  - \u003Cpassing probes — layers ruled out>\nREMEDIATION:\n  1. \u003Cspecific action with CLI command>\n  2. \u003Cverification command after the fix>\nCONFIRM: Before executing any state-changing CLI, emit and await operator\n  approval: \"CONFIRM: About to \u003Caction> on \u003Ccluster> in \u003Cregion>. Proceed?\n  (yes\u002Fno)\"\n```\n\n### Worked example — Application using instance endpoint\n\n```text\nTARGET: prod-db-cluster, application payments-service\nVERDICT: ROOT_CAUSE_IDENTIFIED\nREASON: The Aurora cluster failover completed at 03:17 UTC (confirmed\n  via RDS events). The writer cluster endpoint resolves to the new\n  writer instance within 1 second (DNS TTL verified). However, the\n  application connection string uses the instance endpoint\n  (prod-db-cluster-instance-1.xxxx.us-east-1.rds.amazonaws.com)\n  instead of the cluster endpoint. After failover, instance-1 became\n  a reader; the application is connecting to a reader and receiving\n  \"cannot execute WRITE in a read-only transaction\" errors.\nROOT_CAUSE: APP_NOT_CLUSTER_ENDPOINT\nEVIDENCE:\n  - Symptom: application started receiving \"read-only transaction\"\n    errors at 03:17 UTC, the same time as the Aurora failover event.\n  - Probe: aws rds describe-db-clusters shows the cluster Status is\n    \"available\"; the writer endpoint resolves to instance-2 (the new\n    writer).\n  - Probe: application config shows connection string:\n    jdbc:postgresql:\u002F\u002Fprod-db-cluster-instance-1.xxxx.us-east-1.rds.amazonaws.com:5432\u002Fprod_db\n    (instance endpoint, NOT cluster endpoint).\n  - Probe: dig prod-db-cluster-instance-1.xxxx resolves to the reader\n    IP; dig prod-db-cluster.cluster-xxxx resolves to the writer IP.\n  - Passing: DNS TTL is 1 second (not a DNS issue); connection pool\n    maxLifetime is 30s (not a pool caching issue).\nREMEDIATION:\n  1. Update the application connection string to use the cluster\n     endpoint:\n     jdbc:postgresql:\u002F\u002Fprod-db-cluster.cluster-xxxx.us-east-1.rds.amazonaws.com:5432\u002Fprod_db\n  2. Restart or reload the application to pick up the new connection\n     string.\n  3. Verify by triggering a test failover and confirming the\n     application reconnects to the new writer within 30 seconds.\nCONFIRM: Before updating the application configuration, confirm:\n  \"CONFIRM: About to update the payments-service connection string\n   from instance endpoint to cluster endpoint. Proceed? (yes\u002Fno)\"\n```\n\n### Worked example — Connection pool caching\n\n> Moved verbatim to [`references\u002Fworked-examples.md`](references\u002Fworked-examples.md) — load on demand.\n\n\n### Worked example — INSUFFICIENT_DATA\n\n> Moved verbatim to [`references\u002Fworked-examples.md`](references\u002Fworked-examples.md) — load on demand.\n\n\n## Anti-Patterns — NEVER\n\n- NEVER declare ROOT_CAUSE_IDENTIFIED without a failing probe that\n  matches the symptom. A \"process of elimination\" diagnosis erodes\n  operator trust when the real cause is elsewhere.\n\n- NEVER recommend using the instance endpoint for production\n  application connections. The instance endpoint does not follow the\n  writer after failover. Always use the cluster endpoint (writer\n  endpoint for writes, reader endpoint for reads).\n\n- NEVER assume the failover mechanism is broken because the application\n  is down. Aurora and Multi-AZ failover automation works correctly in\n  the vast majority of cases. The root cause is usually in the\n  application layer (wrong endpoint, stale pool) or configuration\n  (tier, storage, parameter group).\n\n- NEVER trigger a manual failover to \"test\" without confirming the\n  application will handle it. If the application uses the instance\n  endpoint or has stale connection pools, a test failover will cause\n  an outage.\n\n- NEVER conclude \"failover took 5 minutes\" without checking the\n  database failover time separately from the application recovery\n  time. The DB failover may have completed in 10 seconds; the\n  application's connection pool held stale connections for 4.5 minutes.\n\n- NEVER set `PromotionTier` to 0 on all instances without\n  understanding the implication. When multiple instances have tier 0,\n  the largest is chosen. If you want a specific instance to be the\n  promotion target, set it to tier 0 and all others to tier 1+.\n\n- NEVER ignore `FreeStorageSpace` when troubleshooting failover. A\n  storage-full instance cannot complete failover. The error does not\n  say \"storage full\" — it says `FailoverFailed` or\n  `InsufficientStorage`.\n\n- NEVER assume Aurora Global DB failover is as fast as single-region\n  failover. Managed Global failover takes 1-5 minutes (control-plane\n  operation + cross-region DNS). Plan for this latency in your RTO.\n\n- NEVER use `recovery_mode: full` when `optimized` is available. Full\n  recovery mode extends failover duration by 30-60 seconds for no\n  benefit in production workloads.\n\n- NEVER assume RDS Proxy eliminates all failover impact. The Proxy\n  reduces connection errors but active transactions on the old writer\n  are rolled back. The Proxy pool drains over 5-30 seconds.\n\n- NEVER modify parameter groups or option groups on a production\n  cluster without verifying compatibility between primary and standby.\n  A mismatch can prevent the standby from being promoted.\n\n- NEVER conclude no failover occurred just because no CloudWatch alarm\n  fired. Check `describe-events` directly — the alarm may be\n  misconfigured, not the failover.\n\n## Pre-flight safety checks (run before any state-changing CLI)\n\n- **MANDATORY CONFIRMATION GATE.** Before any state-changing operation\n  (`failover-db-cluster`, `modify-db-instance`, `promote-read-replica`,\n  `failover-global-cluster`), emit and await operator approval.\n\n- **Read-only first.** Every probe in the diagnostic tree is\n  read-only (`describe-db-clusters`, `describe-db-instances`,\n  `describe-events`, `get-metric-statistics`, `dig`). Do not perform\n  state-changing operations as diagnostic probes.\n\n- **Manual failover** causes a brief outage (10-120 seconds depending\n  on engine). Always confirm before `failover-db-cluster`.\n\n- **`modify-db-instance --promotion-tier`** is applied immediately but\n  does not affect the current writer. It only affects the next\n  failover.\n\n- **Increasing allocated storage** (`--allocated-storage`) is applied\n  online for most engines but may cause a brief I\u002FO suspension for\n  older MySQL versions.\n\n- **Parameter group changes** may require a reboot to take effect.\n  For Multi-AZ, the reboot triggers a failover. Always use\n  `--no-apply-immediately` if the change should wait for the next\n  maintenance window.\n\n- **Option group changes** can cause the engine to restart. For\n  Multi-AZ, this triggers a failover.\n\n## Remediation guidance\n\n> Moved verbatim to [`references\u002Ferror-handling.md`](references\u002Ferror-handling.md) — load on demand.\n\n\n## Deep reference: Aurora and Multi-AZ failover model\n\n> Moved verbatim to [`references\u002Faurora-endpoint-reference.md`](references\u002Faurora-endpoint-reference.md) — load on demand.\n\n\n> Moved verbatim to [`references\u002Ffailover-priority-reference.md`](references\u002Ffailover-priority-reference.md) — load on demand.\n\n\n> Moved verbatim to [`references\u002Fadvanced-patterns.md`](references\u002Fadvanced-patterns.md) — load on demand.\n\n\n## Recent AWS features (2024-2026)\n\n> Moved verbatim to [`references\u002Fadvanced-patterns.md`](references\u002Fadvanced-patterns.md) — load on demand.\n\n\n## References (load on demand)\n\n- [references\u002Fdiagnostic-commands.md](references\u002Fdiagnostic-commands.md) — account-wide pre-flight sweep and every Step 1-12 probe\u002Ffix CLI listing.\n- [references\u002Ferror-handling.md](references\u002Ferror-handling.md) — malformed-input INSUFFICIENT_DATA contract, DNS caching fix table, failover event codes, full per-root-cause remediation guidance.\n- [references\u002Fworked-examples.md](references\u002Fworked-examples.md) — secondary worked examples (connection pool caching, INSUFFICIENT_DATA).\n- [references\u002Fadvanced-patterns.md](references\u002Fadvanced-patterns.md) — quick-start bullets, mindset, philosophy, Step 0 non-obvious behaviours, Global DB failover modes, recent AWS features, AWS documentation links.\n- [references\u002Faurora-endpoint-reference.md](references\u002Faurora-endpoint-reference.md) — Aurora cluster endpoint types and post-failover behaviour.\n- [references\u002Ffailover-priority-reference.md](references\u002Ffailover-priority-reference.md) — failover timeline comparison and promotion tier matrix.\n\n## Domain\n\nAWS CloudOps \u002F RDS and Aurora Database Failover, High Availability,\nCluster Endpoint Management, Multi-AZ Configuration, Aurora Global DB,\nand RDS Proxy.\n\n## AWS documentation\n\n> Moved verbatim to [`references\u002Fadvanced-patterns.md`](references\u002Fadvanced-patterns.md) — load on demand.\n\n","# RDS Failover Troubleshooter\n\n## Quick start\n\n> Moved verbatim to [`references\u002Fadvanced-patterns.md`](references\u002Fadvanced-patterns.md) — load on demand.\n\n\n## Mindset\n\n> Moved verbatim to [`references\u002Fadvanced-patterns.md`](references\u002Fadvanced-patterns.md) — load on demand.\n\n\n## Philosophy\n\n> Moved verbatim to [`references\u002Fadvanced-patterns.md`](references\u002Fadvanced-patterns.md) — load on demand.\n\n\n## Quick reference — symptom triage table\n\n| Symptom phrase \u002F error | Most likely root cause | First probe |\n|---|---|---|\n| Failover triggered but app still hits old instance | APP_NOT_CLUSTER_ENDPOINT | Check application connection string for instance vs cluster endpoint |\n| App recovers 2-5 minutes after failover despite DNS TTL 1s | CONNECTION_POOL_CACHING | Check DB driver connection pool TTL, JVM DNS cache (`networkaddress.cache.ttl`) |\n| Failover did not trigger at all | MULTI_AZ_HEALTH_THRESHOLD | `describe-db-instances` for Multi-AZ config; CloudWatch `DatabaseConnection` and `DiskQueueDepth` metrics |\n| New writer is not the expected instance | FAILOVER_PRIORITY_TIER | `describe-db-instances` for `PromotionTier` on each instance |\n| `InsufficientStorage` or `FailoverFailed` | STORAGE_FULL | `describe-db-instances` for `StorageAllocated` vs `StorageUsed`; CloudWatch `FreeStorageSpace` |\n| Standby cannot take over; error in RDS events | PARAMETER_GROUP_MISMATCH or OPTION_GROUP_CONFLICT | `describe-db-instances` for `DBParameterGroups` and `OptionGroupMemberships` on primary vs standby |\n| Aurora writer endpoint not resolving to new writer | DNS_PROPAGATION_DELAY | `nslookup` or `dig` on the cluster endpoint; check DNS resolver cache |\n| Read replica promotion fails | READ_REPLICA_PROMOTION_FAILURE | `describe-db-instances` for replica status; check network connectivity and storage |\n| Aurora Global DB failover not working | AURORA_GLOBAL_FAILOVER_MODE | `describe-global-clusters`; check if managed or unmanaged failover |\n| RDS Proxy not draining connections | RDS_PROXY_POOL_DRAIN | `describe-db-proxies`; CloudWatch `DatabaseConnections` metric on the proxy |\n| Recovery mode causes long failover | RECOVERY_MODE_FULL | Check parameter group for `recovery_mode` or Aurora parallel recovery settings |\n| No CloudWatch event for failover | CLOUDWATCH_EVENT_GAP | `describe-events` for the cluster; check EventBridge rule for RDS events |\n\n## Pre-flight: gather the failover context\n\nBefore running root-cause-specific probes, gather the canonical cluster\nconfiguration and short-circuit on misconfigurations that mimic\nfailover failures.\n\n### Account-wide pre-flight commands\n\n> Moved verbatim to [`references\u002Fdiagnostic-commands.md`](references\u002Fdiagnostic-commands.md) — load on demand.\n\n\n### Cluster-state short-circuit\n\n| Finding | Effect on diagnosis |\n|---|---|\n| Cluster `Status: available`, writer endpoint resolves to a healthy instance | Failover completed at the DB layer. Investigate application-side issues (endpoint, connection pool). |\n| Cluster `Status: modifying` or `Status: upgrading` | A maintenance action is in progress. Failover may be delayed or blocked. Wait for completion. |\n| Instance `Status: storage-full` | The instance has exhausted allocated storage. Failover cannot complete. **ROOT_CAUSE_IDENTIFIED** with `ROOT_CAUSE: STORAGE_FULL`. |\n| Instance `MultiAZ: false` | The instance is not Multi-AZ enabled. No standby exists. Failover cannot occur. |\n| Instance `PromotionTier` on intended target > 0 | A lower-tier instance will be promoted first. Check if another instance has tier 0. |\n| Parameter group `status: applying` or `pending-reboot` | A parameter change is pending. The standby may have a different effective configuration. |\n| Aurora Global cluster with `GlobalClusterIdentifier` set | The cluster is part of a Global DB. Failover semantics differ (managed vs unmanaged). |\n\n> Moved verbatim to [`references\u002Ferror-handling.md`](references\u002Ferror-handling.md) — load on demand.\n\n\n## Process — Diagnostic decision tree (apply in symptom order)\n\nThe diagnostic tree is symptom-driven. Pick the entry point based on\nthe observed symptom, then walk the root-cause-specific probes in\norder.\n\n### Step 0: Non-obvious behaviours that change diagnosis\n\n> Moved verbatim to [`references\u002Fadvanced-patterns.md`](references\u002Fadvanced-patterns.md) — load on demand.\n\n\n### Step 1: Application connection string check\n\nSymptom: failover completed (new writer is active, cluster endpoint\nresolves correctly), but the application still cannot connect.\n\n> Moved verbatim to [`references\u002Fdiagnostic-commands.md`](references\u002Fdiagnostic-commands.md) — load on demand.\n\n\nIf the application uses the instance endpoint\n(`\u003Ccluster>-instance-1.\u003Cid>.\u003Cregion>.rds.amazonaws.com`) instead of\nthe cluster endpoint\n(`\u003Ccluster>.cluster-\u003Cid>.\u003Cregion>.rds.amazonaws.com`),\n**ROOT_CAUSE_IDENTIFIED** with `ROOT_CAUSE: APP_NOT_CLUSTER_ENDPOINT`.\n\nThe instance endpoint does not follow the writer after failover. The\napplication must use the cluster endpoint (also called the writer\nendpoint) for transparent failover.\n\n### Step 2: DNS propagation check\n\nSymptom: application uses the cluster endpoint but still hits the old\ninstance for 30+ seconds after failover.\n\n> Moved verbatim to [`references\u002Fdiagnostic-commands.md`](references\u002Fdiagnostic-commands.md) — load on demand.\n\n\nCheck the application's DNS caching layers:\n\n> Moved verbatim to [`references\u002Ferror-handling.md`](references\u002Ferror-handling.md) — load on demand.\n\n\nIf a caching layer is holding the old IP,\n**ROOT_CAUSE_IDENTIFIED** with `ROOT_CAUSE: DNS_PROPAGATION_DELAY`\n(OS\u002FJVM DNS cache) or `ROOT_CAUSE: CONNECTION_POOL_CACHING`\n(connection pool).\n\n### Step 3: Multi-AZ health check threshold\n\nSymptom: primary instance is unhealthy (high CPU, stuck queries) but\nfailover does not trigger.\n\n> Moved verbatim to [`references\u002Fdiagnostic-commands.md`](references\u002Fdiagnostic-commands.md) — load on demand.\n\n\nMulti-AZ failover triggers when:\n- The primary instance becomes unreachable (network partition).\n- The instance status changes to `failed`.\n- The storage subsystem becomes unresponsive.\n\nMulti-AZ does NOT trigger on:\n- High CPU (even at 100%).\n- High connection count.\n- Slow queries.\n\nIf the primary is degraded but responsive (high CPU, slow queries),\nMulti-AZ will not failover. **ROOT_CAUSE_IDENTIFIED** with\n`ROOT_CAUSE: MULTI_AZ_HEALTH_THRESHOLD`. The health check only triggers\non unreachability, not performance degradation.\n\n### Step 4: Failover priority tier\n\nSymptom: Aurora failover completed, but the \"wrong\" instance became\nthe new writer.\n\n> Moved verbatim to [`references\u002Fdiagnostic-commands.md`](references\u002Fdiagnostic-commands.md) — load on demand.\n\n\nAurora failover promotion order:\n1. Lowest `PromotionTier` (tier 0 first).\n2. Within the same tier, largest `DBInstanceClass`.\n3. Within the same tier and class, longest uptime.\n\nIf the intended promotion target has tier 15 (default) but another\ninstance has tier 0, the tier 0 instance is promoted.\n**ROOT_CAUSE_IDENTIFIED** with `ROOT_CAUSE: FAILOVER_PRIORITY_TIER`.\n\nFix: set the intended target to tier 0:\n\n> Moved verbatim to [`references\u002Fdiagnostic-commands.md`](references\u002Fdiagnostic-commands.md) — load on demand.\n\n\n### Step 5: Storage-full check\n\nSymptom: failover event appears in RDS events but the new writer does\nnot become available. Error: `InsufficientStorage` or\n`FailoverFailed`.\n\n> Moved verbatim to [`references\u002Fdiagnostic-commands.md`](references\u002Fdiagnostic-commands.md) — load on demand.\n\n\nIf `FreeStorageSpace` is near zero (\u003C 5 GB) or `DBInstanceStatus` is\n`storage-full`, **ROOT_CAUSE_IDENTIFIED** with\n`ROOT_CAUSE: STORAGE_FULL`.\n\nFix: increase allocated storage:\n\n> Moved verbatim to [`references\u002Fdiagnostic-commands.md`](references\u002Fdiagnostic-commands.md) — load on demand.\n\n\n### Step 6: Parameter group mismatch\n\nSymptom: standby instance fails health checks during promotion. RDS\nevents show a promotion failure with parameter-related errors.\n\n> Moved verbatim to [`references\u002Fdiagnostic-commands.md`](references\u002Fdiagnostic-commands.md) — load on demand.\n\n\nIf the primary and standby have different DB parameter groups with\nincompatible settings, the standby may fail during promotion.\n**ROOT_CAUSE_IDENTIFIED** with `ROOT_CAUSE: PARAMETER_GROUP_MISMATCH`.\n\nFix: align the parameter groups:\n\n> Moved verbatim to [`references\u002Fdiagnostic-commands.md`](references\u002Fdiagnostic-commands.md) — load on demand.\n\n\n### Step 7: Option group conflict\n\nSymptom: standby cannot start the engine after promotion. RDS events\nshow an option-group-related error.\n\n> Moved verbatim to [`references\u002Fdiagnostic-commands.md`](references\u002Fdiagnostic-commands.md) — load on demand.\n\n\nIf the option groups are incompatible (e.g., primary has TDE, standby\ndoes not), **ROOT_CAUSE_IDENTIFIED** with\n`ROOT_CAUSE: OPTION_GROUP_CONFLICT`.\n\nFix: align the option groups:\n\n> Moved verbatim to [`references\u002Fdiagnostic-commands.md`](references\u002Fdiagnostic-commands.md) — load on demand.\n\n\n### Step 8: Read replica promotion failure\n\nSymptom: a read replica (Aurora or cross-region) cannot be promoted\nto a standalone cluster\u002Finstance.\n\n> Moved verbatim to [`references\u002Fdiagnostic-commands.md`](references\u002Fdiagnostic-commands.md) — load on demand.\n\n\nCommon promotion failures:\n- High replication lag (the replica has not caught up).\n- Network connectivity to the source (for cross-region).\n- Storage-full on the replica.\n- Parameter group mismatch.\n\nIf any of these apply, **ROOT_CAUSE_IDENTIFIED** with\n`ROOT_CAUSE: READ_REPLICA_PROMOTION_FAILURE`.\n\n### Step 9: Aurora Global DB failover mode\n\nSymptom: Aurora Global DB failover did not work or took too long.\n\n> Moved verbatim to [`references\u002Fdiagnostic-commands.md`](references\u002Fdiagnostic-commands.md) — load on demand.\n\n\nManaged failover (`failover-global-cluster`) is a control-plane\noperation that takes 1-5 minutes. Unmanaged failover (detach secondary\n+ promote) is faster but requires manual coordination.\n\nIf the failover mode is misconfigured or the secondary cluster has\nhigh replication lag, **ROOT_CAUSE_IDENTIFIED** with\n`ROOT_CAUSE: AURORA_GLOBAL_FAILOVER_MODE`.\n\n### Step 10: RDS Proxy failover\n\nSymptom: RDS Proxy does not transparently handle failover; application\nsees connection errors.\n\n> Moved verbatim to [`references\u002Fdiagnostic-commands.md`](references\u002Fdiagnostic-commands.md) — load on demand.\n\n\nRDS Proxy should queue and reconnect during failover. If connections\nare being dropped instead of queued, check:\n- Proxy target group health.\n- `ConnectionBorrowTimeout` (client-side).\n- Proxy `IdleClientTimeout`.\n\nIf the Proxy is not draining\u002Fqueuing correctly,\n**ROOT_CAUSE_IDENTIFIED** with `ROOT_CAUSE: RDS_PROXY_POOL_DRAIN`.\n\n### Step 11: Recovery mode\n\nSymptom: Aurora failover takes 30+ seconds consistently; Multi-AZ\nfailover takes 120+ seconds.\n\nFor Aurora, check if parallel recovery is enabled (engine-version\ndependent):\n\n> Moved verbatim to [`references\u002Fdiagnostic-commands.md`](references\u002Fdiagnostic-commands.md) — load on demand.\n\n\nAurora PostgreSQL 13+ supports parallel WAL replay. Aurora MySQL 8.0\nhas optimised crash recovery. Older versions use serial recovery.\n\nFor Multi-AZ, check the `recovery_mode` parameter:\n\n> Moved verbatim to [`references\u002Fdiagnostic-commands.md`](references\u002Fdiagnostic-commands.md) — load on demand.\n\n\nIf recovery mode is set to `full` instead of `optimized`,\n**ROOT_CAUSE_IDENTIFIED** with `ROOT_CAUSE: RECOVERY_MODE_FULL`.\n\n### Step 12: CloudWatch failover event gap\n\nSymptom: failover occurred but no CloudWatch alarm or EventBridge rule\nfired.\n\n> Moved verbatim to [`references\u002Fdiagnostic-commands.md`](references\u002Fdiagnostic-commands.md) — load on demand.\n\n\n> Moved verbatim to [`references\u002Ferror-handling.md`](references\u002Ferror-handling.md) — load on demand.\n\n\nIf no events appear in `describe-events`, the failover may not have\ntriggered at the RDS service level. If events appear but no alarm\nfired, the EventBridge rule or CloudWatch alarm is misconfigured.\n**ROOT_CAUSE_IDENTIFIED** with `ROOT_CAUSE: CLOUDWATCH_EVENT_GAP`.\n\n## Output format\n\n```text\nTARGET: \u003Ccluster-id or instance-id, application context>\nVERDICT: ROOT_CAUSE_IDENTIFIED | INSUFFICIENT_DATA\nREASON: \u003C1-2 sentences naming the root cause and the failing probe>\nROOT_CAUSE: \u003CAPP_NOT_CLUSTER_ENDPOINT | MULTI_AZ_HEALTH_THRESHOLD |\n              FAILOVER_PRIORITY_TIER | STORAGE_FULL |\n              PARAMETER_GROUP_MISMATCH | OPTION_GROUP_CONFLICT |\n              DNS_PROPAGATION_DELAY | CONNECTION_POOL_CACHING |\n              AURORA_GLOBAL_FAILOVER_MODE | RDS_PROXY_POOL_DRAIN |\n              RECOVERY_MODE_FULL | CLOUDWATCH_EVENT_GAP |\n              READ_REPLICA_PROMOTION_FAILURE | UNKNOWN>\nEVIDENCE:\n  - \u003Cobserved symptom — error string, failover timeline>\n  - \u003Cfailing probe — command and its output that confirms the cause>\n  - \u003Cpassing probes — layers ruled out>\nREMEDIATION:\n  1. \u003Cspecific action with CLI command>\n  2. \u003Cverification command after the fix>\nCONFIRM: Before executing any state-changing CLI, emit and await operator\n  approval: \"CONFIRM: About to \u003Caction> on \u003Ccluster> in \u003Cregion>. Proceed?\n  (yes\u002Fno)\"\n```\n\n### Worked example — Application using instance endpoint\n\n```text\nTARGET: prod-db-cluster, application payments-service\nVERDICT: ROOT_CAUSE_IDENTIFIED\nREASON: The Aurora cluster failover completed at 03:17 UTC (confirmed\n  via RDS events). The writer cluster endpoint resolves to the new\n  writer instance within 1 second (DNS TTL verified). However, the\n  application connection string uses the instance endpoint\n  (prod-db-cluster-instance-1.xxxx.us-east-1.rds.amazonaws.com)\n  instead of the cluster endpoint. After failover, instance-1 became\n  a reader; the application is connecting to a reader and receiving\n  \"cannot execute WRITE in a read-only transaction\" errors.\nROOT_CAUSE: APP_NOT_CLUSTER_ENDPOINT\nEVIDENCE:\n  - Symptom: application started receiving \"read-only transaction\"\n    errors at 03:17 UTC, the same time as the Aurora failover event.\n  - Probe: aws rds describe-db-clusters shows the cluster Status is\n    \"available\"; the writer endpoint resolves to instance-2 (the new\n    writer).\n  - Probe: application config shows connection string:\n    jdbc:postgresql:\u002F\u002Fprod-db-cluster-instance-1.xxxx.us-east-1.rds.amazonaws.com:5432\u002Fprod_db\n    (instance endpoint, NOT cluster endpoint).\n  - Probe: dig prod-db-cluster-instance-1.xxxx resolves to the reader\n    IP; dig prod-db-cluster.cluster-xxxx resolves to the writer IP.\n  - Passing: DNS TTL is 1 second (not a DNS issue); connection pool\n    maxLifetime is 30s (not a pool caching issue).\nREMEDIATION:\n  1. Update the application connection string to use the cluster\n     endpoint:\n     jdbc:postgresql:\u002F\u002Fprod-db-cluster.cluster-xxxx.us-east-1.rds.amazonaws.com:5432\u002Fprod_db\n  2. Restart or reload the application to pick up the new connection\n     string.\n  3. Verify by triggering a test failover and confirming the\n     application reconnects to the new writer within 30 seconds.\nCONFIRM: Before updating the application configuration, confirm:\n  \"CONFIRM: About to update the payments-service connection string\n   from instance endpoint to cluster endpoint. Proceed? (yes\u002Fno)\"\n```\n\n### Worked example — Connection pool caching\n\n> Moved verbatim to [`references\u002Fworked-examples.md`](references\u002Fworked-examples.md) — load on demand.\n\n\n### Worked example — INSUFFICIENT_DATA\n\n> Moved verbatim to [`references\u002Fworked-examples.md`](references\u002Fworked-examples.md) — load on demand.\n\n\n## Anti-Patterns — NEVER\n\n- NEVER declare ROOT_CAUSE_IDENTIFIED without a failing probe that\n  matches the symptom. A \"process of elimination\" diagnosis erodes\n  operator trust when the real cause is elsewhere.\n\n- NEVER recommend using the instance endpoint for production\n  application connections. The instance endpoint does not follow the\n  writer after failover. Always use the cluster endpoint (writer\n  endpoint for writes, reader endpoint for reads).\n\n- NEVER assume the failover mechanism is broken because the application\n  is down. Aurora and Multi-AZ failover automation works correctly in\n  the vast majority of cases. The root cause is usually in the\n  application layer (wrong endpoint, stale pool) or configuration\n  (tier, storage, parameter group).\n\n- NEVER trigger a manual failover to \"test\" without confirming the\n  application will handle it. If the application uses the instance\n  endpoint or has stale connection pools, a test failover will cause\n  an outage.\n\n- NEVER conclude \"failover took 5 minutes\" without checking the\n  database failover time separately from the application recovery\n  time. The DB failover may have completed in 10 seconds; the\n  application's connection pool held stale connections for 4.5 minutes.\n\n- NEVER set `PromotionTier` to 0 on all instances without\n  understanding the implication. When multiple instances have tier 0,\n  the largest is chosen. If you want a specific instance to be the\n  promotion target, set it to tier 0 and all others to tier 1+.\n\n- NEVER ignore `FreeStorageSpace` when troubleshooting failover. A\n  storage-full instance cannot complete failover. The error does not\n  say \"storage full\" — it says `FailoverFailed` or\n  `InsufficientStorage`.\n\n- NEVER assume Aurora Global DB failover is as fast as single-region\n  failover. Managed Global failover takes 1-5 minutes (control-plane\n  operation + cross-region DNS). Plan for this latency in your RTO.\n\n- NEVER use `recovery_mode: full` when `optimized` is available. Full\n  recovery mode extends failover duration by 30-60 seconds for no\n  benefit in production workloads.\n\n- NEVER assume RDS Proxy eliminates all failover impact. The Proxy\n  reduces connection errors but active transactions on the old writer\n  are rolled back. The Proxy pool drains over 5-30 seconds.\n\n- NEVER modify parameter groups or option groups on a production\n  cluster without verifying compatibility between primary and standby.\n  A mismatch can prevent the standby from being promoted.\n\n- NEVER conclude no failover occurred just because no CloudWatch alarm\n  fired. Check `describe-events` directly — the alarm may be\n  misconfigured, not the failover.\n\n## Pre-flight safety checks (run before any state-changing CLI)\n\n- **MANDATORY CONFIRMATION GATE.** Before any state-changing operation\n  (`failover-db-cluster`, `modify-db-instance`, `promote-read-replica`,\n  `failover-global-cluster`), emit and await operator approval.\n\n- **Read-only first.** Every probe in the diagnostic tree is\n  read-only (`describe-db-clusters`, `describe-db-instances`,\n  `describe-events`, `get-metric-statistics`, `dig`). Do not perform\n  state-changing operations as diagnostic probes.\n\n- **Manual failover** causes a brief outage (10-120 seconds depending\n  on engine). Always confirm before `failover-db-cluster`.\n\n- **`modify-db-instance --promotion-tier`** is applied immediately but\n  does not affect the current writer. It only affects the next\n  failover.\n\n- **Increasing allocated storage** (`--allocated-storage`) is applied\n  online for most engines but may cause a brief I\u002FO suspension for\n  older MySQL versions.\n\n- **Parameter group changes** may require a reboot to take effect.\n  For Multi-AZ, the reboot triggers a failover. Always use\n  `--no-apply-immediately` if the change should wait for the next\n  maintenance window.\n\n- **Option group changes** can cause the engine to restart. For\n  Multi-AZ, this triggers a failover.\n\n## Remediation guidance\n\n> Moved verbatim to [`references\u002Ferror-handling.md`](references\u002Ferror-handling.md) — load on demand.\n\n\n## Deep reference: Aurora and Multi-AZ failover model\n\n> Moved verbatim to [`references\u002Faurora-endpoint-reference.md`](references\u002Faurora-endpoint-reference.md) — load on demand.\n\n\n> Moved verbatim to [`references\u002Ffailover-priority-reference.md`](references\u002Ffailover-priority-reference.md) — load on demand.\n\n\n> Moved verbatim to [`references\u002Fadvanced-patterns.md`](references\u002Fadvanced-patterns.md) — load on demand.\n\n\n## Recent AWS features (2024-2026)\n\n> Moved verbatim to [`references\u002Fadvanced-patterns.md`](references\u002Fadvanced-patterns.md) — load on demand.\n\n\n## References (load on demand)\n\n- [references\u002Fdiagnostic-commands.md](references\u002Fdiagnostic-commands.md) — account-wide pre-flight sweep and every Step 1-12 probe\u002Ffix CLI listing.\n- [references\u002Ferror-handling.md](references\u002Ferror-handling.md) — malformed-input INSUFFICIENT_DATA contract, DNS caching fix table, failover event codes, full per-root-cause remediation guidance.\n- [references\u002Fworked-examples.md](references\u002Fworked-examples.md) — secondary worked examples (connection pool caching, INSUFFICIENT_DATA).\n- [references\u002Fadvanced-patterns.md](references\u002Fadvanced-patterns.md) — quick-start bullets, mindset, philosophy, Step 0 non-obvious behaviours, Global DB failover modes, recent AWS features, AWS documentation links.\n- [references\u002Faurora-endpoint-reference.md](references\u002Faurora-endpoint-reference.md) — Aurora cluster endpoint types and post-failover behaviour.\n- [references\u002Ffailover-priority-reference.md](references\u002Ffailover-priority-reference.md) — failover timeline comparison and promotion tier matrix.\n\n## Domain\n\nAWS CloudOps \u002F RDS and Aurora Database Failover, High Availability,\nCluster Endpoint Management, Multi-AZ Configuration, Aurora Global DB,\nand RDS Proxy.\n\n## AWS documentation\n\n> Moved verbatim to [`references\u002Fadvanced-patterns.md`](references\u002Fadvanced-patterns.md) — load on demand.\n\n",{"name":5,"license":15,"metadata":24,"description":7,"compatibility":44},{"tags":25,"phase":26,"author":27,"domain":28,"family":29,"version":30,"keywords":31,"task_type":32,"complexity":33,"entry_point":34,"skill_class":35,"when_to_use":36,"requires_llm":37,"verdict_shape":38,"when_not_to_use":39,"lifecycle_status":40,"invocation_schema":41,"supports_pipeline":37,"invocation_example":42,"activation_triggers":43},"rds, aurora, databases, troubleshooting, failover, multi-az, high-availability, cluster-endpoint, rds-proxy","2","Jacky Chan — AWS Community Builder","aws-cloudops","Databases","0.1.0","RDS, Aurora, failover, Multi-AZ, writer endpoint, cluster endpoint, read replica promotion, storage-full, parameter group mismatch, option group conflict, failover priority, tier 0, Aurora Global DB, RDS Proxy, connection pool, DNS propagation, CloudWatch failover events, troubleshooting","troubleshoot","high","false","capability","Diagnosing an RDS or Aurora failover issue (Multi-AZ failover not triggering, failover taking too long, writer endpoint not redirecting, read replica promotion failure, storage-full preventing failover, parameter group mismatch, option group conflict, failover priority misconfiguration, Aurora Global DB failover, RDS Proxy failover, application connection issues), walking a symptom to the failed layer with verify commands, or triaging a \"failover did not work\" incident where the root cause may be endpoint configuration, health threshold, storage, parameter group, option group, failover tier, or application connection string — not necessarily the database engine itself.","true","ROOT_CAUSE_IDENTIFIED | INSUFFICIENT_DATA","RDS connectivity debugging (use rds-connectivity-troubleshooter), RDS performance tuning (use rds-cost-optimizer for right-sizing), Aurora backup and restore (use rds-backup-restore-operator), or RDS instance creation (use rds-instance-deployer). This skill diagnoses failover-time issues; it does not tune query performance or audit steady-state security posture.","active","Input: either (a) a symptom description (error message, observed behaviour, \"failover took 5 minutes\", \"writer endpoint still points to old instance\"), optionally paired with the cluster configuration and recent RDS events, OR (b) a DB cluster identifier plus caller context (application connection string, observed error) for live-account diagnosis. Output: a deterministic TARGET\u002FVERDICT\u002FREASON\u002FROOT_CAUSE\u002FEVIDENCE\u002FREMEDIATION block where VERDICT ∈ {ROOT_CAUSE_IDENTIFIED, INSUFFICIENT_DATA} and ROOT_CAUSE ∈ {APP_NOT_CLUSTER_ENDPOINT, MULTI_AZ_HEALTH_THRESHOLD, FAILOVER_PRIORITY_TIER, STORAGE_FULL, PARAMETER_GROUP_MISMATCH, OPTION_GROUP_CONFLICT, DNS_PROPAGATION_DELAY, CONNECTION_POOL_CACHING, AURORA_GLOBAL_FAILOVER_MODE, RDS_PROXY_POOL_DRAIN, RECOVERY_MODE_FULL, CLOUDWATCH_EVENT_GAP, READ_REPLICA_PROMOTION_FAILURE, UNKNOWN}.","# Minimal valid input (offline symptom classification):\nSymptom: \"Aurora PostgreSQL cluster prod-db-cluster\nfailed over automatically at 03:17 UTC. The writer\nendpoint resolved to the new writer within 1 second,\nbut the application continued hitting the old writer\nfor 90 seconds, then connection errors cleared.\"\nDBClusterIdentifier: prod-db-cluster\nEngine: aurora-postgresql\nWriterEndpoint: prod-db-cluster.cluster-xxxxxxxxxxxx.us-east-1.rds.amazonaws.com\nApplication connection string: uses the instance endpoint\nprod-db-cluster-instance-1.xxxxxxxxxxxx.us-east-1.rds.amazonaws.com\ninstead of the cluster endpoint","RDS failover not working, Aurora failover not triggering, Multi-AZ failover failed, Aurora writer endpoint not redirecting, RDS failover taking too long, Aurora failover priority, read replica promotion failed, storage-full failover, parameter group mismatch failover, option group conflict failover, Aurora Global DB failover, RDS Proxy failover, application not connecting after failover, Aurora cluster endpoint confusion, troubleshoot RDS failover","Agent runtime that reads SKILL.md (Claude Code, Cursor, Windsurf, Codex, Gemini). Offline symptom classification works from pasted error messages and cluster configuration. Live-account diagnosis uses aws rds describe-db-clusters, aws rds describe-db-instances, aws rds describe-db-cluster-endpoints, aws rds describe-db-cluster-parameter-groups, aws rds describe-events, aws cloudwatch get-metric-statistics, aws ec2 describe-route-tables, aws ec2 describe-security-groups, and aws logs...","main","eeace51bd5dd1db2030729d83eeab988861553aa","skills\u002Frds-failover-troubleshooter",0,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills","https:\u002F\u002Fgithub.com\u002Fjhkchan","CTO & Co-Founder @Beever-AI  & @votee. CAISP. AWS Community Builder. Organizer@GDGCloudHK. MongoDB Champion. Technopreneur + Infornographer.","2026-09-24 08:54:10",{"source":8,"skillName":5,"commands":54,"prompt":64,"zipAvailable":65},[55,58,61],{"tool":56,"command":57},"npx","npx skills add jhkchan\u002Faws-agent-skills --skill rds-failover-troubleshooter",{"tool":59,"command":60},"bunx","bunx skills add jhkchan\u002Faws-agent-skills --skill rds-failover-troubleshooter",{"tool":62,"command":63},"pnpm","pnpm dlx skills add jhkchan\u002Faws-agent-skills --skill rds-failover-troubleshooter","安装技能，技能地址：https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Frds-failover-troubleshooter\u002FSKILL.md",true,[67,73,77,81,85,89,93,97,101,105,109,113,117,121,125,129,133,137,141,145,149],{"path":68,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":71,"sourceHtmlUrl":72},"eval\u002Ftest-cases.yaml","REFERENCE","参考文档",12072,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Frds-failover-troubleshooter\u002Feval\u002Ftest-cases.yaml",{"path":74,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":75,"sourceHtmlUrl":76},"evals\u002Fbaselines\u002Fapp-instance-endpoint-not-cluster.md",434,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Frds-failover-troubleshooter\u002Fevals\u002Fbaselines\u002Fapp-instance-endpoint-not-cluster.md",{"path":78,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":79,"sourceHtmlUrl":80},"evals\u002Fbaselines\u002Fconnection-pool-hikari-stale-cache.md",406,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Frds-failover-troubleshooter\u002Fevals\u002Fbaselines\u002Fconnection-pool-hikari-stale-cache.md",{"path":82,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":83,"sourceHtmlUrl":84},"evals\u002Fbaselines\u002Ffailover-priority-tier-wrong-instance.md",393,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Frds-failover-troubleshooter\u002Fevals\u002Fbaselines\u002Ffailover-priority-tier-wrong-instance.md",{"path":86,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":87,"sourceHtmlUrl":88},"evals\u002Fbaselines\u002Fparameter-group-mismatch-standby.md",378,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Frds-failover-troubleshooter\u002Fevals\u002Fbaselines\u002Fparameter-group-mismatch-standby.md",{"path":90,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":91,"sourceHtmlUrl":92},"evals\u002Fbaselines\u002Fstorage-full-prevents-failover.md",327,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Frds-failover-troubleshooter\u002Fevals\u002Fbaselines\u002Fstorage-full-prevents-failover.md",{"path":94,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":95,"sourceHtmlUrl":96},"evals\u002Feval_config.yaml",2113,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Frds-failover-troubleshooter\u002Fevals\u002Feval_config.yaml",{"path":98,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":99,"sourceHtmlUrl":100},"evals\u002Fevals.json",5152,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Frds-failover-troubleshooter\u002Fevals\u002Fevals.json",{"path":102,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":103,"sourceHtmlUrl":104},"evals\u002Fprompts\u002Fapp-instance-endpoint-not-cluster.md",1592,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Frds-failover-troubleshooter\u002Fevals\u002Fprompts\u002Fapp-instance-endpoint-not-cluster.md",{"path":106,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":107,"sourceHtmlUrl":108},"evals\u002Fprompts\u002Fconnection-pool-hikari-stale-cache.md",1569,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Frds-failover-troubleshooter\u002Fevals\u002Fprompts\u002Fconnection-pool-hikari-stale-cache.md",{"path":110,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":111,"sourceHtmlUrl":112},"evals\u002Fprompts\u002Ffailover-priority-tier-wrong-instance.md",1551,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Frds-failover-troubleshooter\u002Fevals\u002Fprompts\u002Ffailover-priority-tier-wrong-instance.md",{"path":114,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":115,"sourceHtmlUrl":116},"evals\u002Fprompts\u002Fparameter-group-mismatch-standby.md",1573,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Frds-failover-troubleshooter\u002Fevals\u002Fprompts\u002Fparameter-group-mismatch-standby.md",{"path":118,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":119,"sourceHtmlUrl":120},"evals\u002Fprompts\u002Fstorage-full-prevents-failover.md",1332,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Frds-failover-troubleshooter\u002Fevals\u002Fprompts\u002Fstorage-full-prevents-failover.md",{"path":122,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":123,"sourceHtmlUrl":124},"examples\u002FREADME.md",7677,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Frds-failover-troubleshooter\u002Fexamples\u002FREADME.md",{"path":126,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":127,"sourceHtmlUrl":128},"references\u002Fadvanced-patterns.md",13649,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Frds-failover-troubleshooter\u002Freferences\u002Fadvanced-patterns.md",{"path":130,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":131,"sourceHtmlUrl":132},"references\u002Faurora-endpoint-reference.md",6535,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Frds-failover-troubleshooter\u002Freferences\u002Faurora-endpoint-reference.md",{"path":134,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":135,"sourceHtmlUrl":136},"references\u002Fdiagnostic-commands.md",7947,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Frds-failover-troubleshooter\u002Freferences\u002Fdiagnostic-commands.md",{"path":138,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":139,"sourceHtmlUrl":140},"references\u002Ferror-handling.md",4850,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Frds-failover-troubleshooter\u002Freferences\u002Ferror-handling.md",{"path":142,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":143,"sourceHtmlUrl":144},"references\u002Ffailover-priority-reference.md",7953,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Frds-failover-troubleshooter\u002Freferences\u002Ffailover-priority-reference.md",{"path":146,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":147,"sourceHtmlUrl":148},"references\u002Fworked-examples.md",2416,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Frds-failover-troubleshooter\u002Freferences\u002Fworked-examples.md",{"path":150,"resourceType":69,"resourceTypeDesc":70,"sizeBytes":151,"sourceHtmlUrl":152},"skill.usf.yaml",4667,"https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Frds-failover-troubleshooter\u002Fskill.usf.yaml",[154,161,168,175,182,189],{"skillNo":155,"name":156,"nameZh":6,"description":157,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":158,"categories":159,"keywords":160,"publishedAt":19},"01M7ADVD60X4PCAPKSE58P9NDC","xray-tracing-deployer","Deploys AWS X-Ray distributed tracing with production-grade configuration: X-Ray daemon deployment (EC2 systemd, ECS sidecar, EKS DaemonSet, Lambda built-in), SDK instrumentation (Java, Python, Node.js, Go, .NET) with annotations and metadata, sampling rules (default reservoir + rate vs custom rules with URL \u002F method \u002F service-name predicates), service map generation, trace query and retrieval, groups and insights, CloudWatch ServiceLens integration, IAM permissions (PutTraceSegments, GetSamplingRules), and latest features (Lambda Powertools tracing, OpenTelemetry \u002F ADOT Collector support, W3C trace context). Emits a READY_TO_DEPLOY checklist with every configuration item verified. Use when enabling X-Ray on a new application, adding tracing to ECS \u002F EKS \u002F Lambda, validating a sampling configuration, or generating daemon deployment CLI and IaC templates. Triggers: X-Ray, distributed tracing, X-Ray daemon, sampling rules, service map, X-Ray SDK, OpenTelemetry, ADOT, Lambda Powertools, CloudWatch ServiceLens.","https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fxray-tracing-deployer\u002FSKILL.md",[],[],{"skillNo":162,"name":163,"nameZh":6,"description":164,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":165,"categories":166,"keywords":167,"publishedAt":19},"01M7ADTVJ7DCBM9CB1FY51GH7J","wellarchitected-workload-auditor","Audits AWS Well-Architected Tool workloads for review staleness, per-pillar high-risk issue counts, milestone tracking gaps, and remediation plan completeness. Emits a deterministic verdict (STALE_REVIEW | HIGH_RISK | CONFIG_GAP | OK) per workload with per-pillar risk breakdown, milestone trend analysis, and specific remediation. Use when reviewing Well-Architected workloads, checking WA review freshness, auditing high-risk issues per pillar, validating milestone coverage, or tracking remediation plans.","https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fwellarchitected-workload-auditor\u002FSKILL.md",[],[],{"skillNo":169,"name":170,"nameZh":6,"description":171,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":172,"categories":173,"keywords":174,"publishedAt":19},"01M7ADT59G6FTTNZYHT2QRPNNP","wellarchitected-review-operator","Operates AWS Well-Architected Tool reviews end-to-end — creates workloads, runs pillar reviews (operational excellence, security, reliability, performance, cost optimization, sustainability, Prosperity), answers questions with risk-tier rationale, generates reports and improvement plans, creates milestones, and integrates with Trusted Advisor and Well-Architected Labs. Runs deterministic pre-checks (workload existence, lens availability, pillar coverage), emits the exact update-answer \u002F get-consolidated-report CLI behind a CONFIRM gate, and verifies risk-tier transitions. Emits READY | BLOCKED | COMPLETED. Use when creating a workload, answering pillar questions, generating an improvement plan, creating a milestone, or integrating with Trusted Advisor. Triggers: well-architected tool, well architected review, workload review, pillar, operational excellence, security, reliability, performance, cost optimization, sustainability, prosperity, improvement plan, milestone, trusted advisor.","https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fwellarchitected-review-operator\u002FSKILL.md",[],[],{"skillNo":176,"name":177,"nameZh":6,"description":178,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":179,"categories":180,"keywords":181,"publishedAt":19},"01M7ADSH5JA9QHNPPHQCD7EVA7","wafv2-web-acl-deployer","Provisions AWS WAFv2 Web ACLs with correct production defaults: scope selection (CLOUDFRONT vs REGIONAL), managed rule groups (AWSManagedRulesCommonRuleSet, SQLiRuleSet, LinuxRuleSet, WindowsRuleSet, IPReputationList, BotControlRuleSet, AmazonIpReputationList, ATP), custom rules (byte match, IP set, regex pattern, geo match, size constraint, rate-based), rule priority ordering, logging destinations (Kinesis Firehose, CloudWatch Logs, S3), resource association (ALB, API Gateway, CloudFront), rate-based rules with aggregate keys (IP, forwarded IP), Challenge and CAPTCHA actions, and account takeover prevention. Emits a READY_TO_DEPLOY checklist. Use when creating a Web ACL, attaching managed rules, configuring rate limiting, enabling WAF logging, associating a Web ACL with an ALB\u002FAPI Gateway\u002FCloudFront, or configuring CAPTCHA\u002FChallenge actions. Triggers: create WAF Web ACL, WAFv2, managed rule groups, WAF rate limiting, WAF logging, WAF CAPTCHA, WAF Challenge action, bot control, ATP managed rules.","https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fwafv2-web-acl-deployer\u002FSKILL.md",[],[],{"skillNo":183,"name":184,"nameZh":6,"description":185,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":186,"categories":187,"keywords":188,"publishedAt":19},"01M7ADRYFAT8R73FBNCHRC573X","wafv2-web-acl-auditor","Audits AWS WAFv2 Web ACL configurations to determine whether the ACL provides effective protection: default-action posture (Allow vs Block), managed-rule-group coverage gaps, rule effectiveness (BLOCK vs COUNT, shadow\u002Fbypass rules, stale exclusions), rate-based rule correctness, logging and visibility configuration, and text-transformation bypass vectors. Emits a deterministic verdict per Web ACL. Use when reviewing a WAFv2 Web ACL for security posture, checking managed-rule coverage, validating rate-limit configuration, auditing WAF logging, or hardening a WAF before production deployment. Triggers: WAF, WAFv2, Web ACL, managed rule group, AWSManagedRules, rate-based rule, CAPTCHA, Challenge, OverrideAction, Count mode, forwarded IP, Bot Control, ATP, text transformation, WAF logging, WAF visibility, OWASP, SQLi, XSS, RFI.","https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fwafv2-web-acl-auditor\u002FSKILL.md",[],[],{"skillNo":190,"name":191,"nameZh":6,"description":192,"descriptionZh":6,"repositoryFullName":8,"ownerLogin":9,"ownerAvatarUrl":10,"ownerAccountNo":11,"ownerDisplayName":12,"repositoryNo":13,"starsCount":14,"licenseSpdxId":15,"licenseFileUrl":6,"sourceHtmlUrl":193,"categories":194,"keywords":195,"publishedAt":19},"01M7ADR8W7G72JX4KFGN03QK8Q","waf-rule-deployer","Provisions AWS WAFv2 rule sets with production defaults: Web ACL creation (CloudFront scope in us-east-1 vs Regional scope), rule groups (managed vs custom) with capacity planning in WCU (Web ACL Capacity Units, 1500 default per ACL), managed rule groups (AWSManagedRulesCommonRuleSet, AWSManagedRulesSQLiRuleSet, AWSManagedRulesAmazonIpReputationList, AWSManagedRulesLinuxRuleSet, AWSManagedRulesWindowsRuleSet, AWSManagedRulesUnixRuleSet), custom rules (byte-match, regex pattern set, geo-match, rate-based), IP set and regex pattern set resources, rule priority ordering (lowest number evaluated first), action types (allow, block, count, CAPTCHA, Challenge), bot control managed rule group, account takeover. Triggers: create WAF rule, WAF managed rule group, WAF custom rule, WAF byte match, WAF geo match, WAF rate based rule, WAF IP set, WAF regex pattern set, WAF WCU budget, WAF CloudFront ACL, WAF logging Kinesis Firehose, WAF bot control, WAF ATP, WAF label matching, WAF Challenge action, WAF CAPTCHA.","https:\u002F\u002Fgithub.com\u002Fjhkchan\u002Faws-agent-skills\u002Fblob\u002Fmain\u002Fskills\u002Fwaf-rule-deployer\u002FSKILL.md",[],[]]