chaos-study-design

chaos-study-design — what should we test, and why?

Studies often go wrong before anything is injected. Someone picks a fault because it is available, runs it, sees nothing, and concludes the system is resilient. This skill tries to steer away from that outcome: it asks for the question what would we learn? to be answered — in the customer's own words and against the customer's own code — before a scenario is ever chosen.

Principles

Read before you ask. Analysis always precedes the interview. Asking a customer "what are your dependencies?" when their repository is sitting right there wastes their time and produces worse answers than reading it. Eight areas are inspected: code, deployment, dependencies, resilience behaviour, persistence, workload, telemetry, and identity.

Unavailable is not unknown. Each area is either observed with a citation — a file, a symbol, a config key — or explicitly declared unavailable with a reason. Silence is rejected: analyze exits 24 and names every area left unaccounted for. A candidate hypothesis may never rest on evidence nobody read.

Grill, don't survey. "Test resilience" is not a goal. Nine questions are asked one at a time, in dependency order, and vague answers are refused rather than recorded. Four of them are grounded: the answer must engage with the customer's own system vocabulary — the services, dependencies and signals analysis actually found — not generic reliability language.

Never ask what you already know. A question that analysis already answered is pre-filled and marked establishedBy. The customer is asked only for what only they can supply: intent, business impact, risk appetite, abort criteria.

Candidates are code-grounded or they are not candidates. Every candidate carries eight mandatory fields — dependency edge, failure mechanism, mechanism evidence, mechanism probe, steady-state predicate, exposure inputs, collateral risks, and abort criteria. A candidate missing any of them is rejected at construction, not at review.

The platform is asked, never assumed. Candidates are intersected with live regional action discovery, workspace-discovered resources, and workspace-recommended scenarios. When Azure cannot be reached, candidates are marked provisional, limitation L15 is recorded, and the brief says plainly that platform availability is unconfirmed. No bundled catalogue, ever.

The customer chooses. Two to four ranked candidates are presented with rationale and tradeoffs. Nothing is recommended as final until the customer returns an exact confirmation phrase bound to the brief hash. On confirmation the brief seals: the phrase keeps referring to what they actually saw.

Usage

Design is the first phase, and the front door drives it for you — that is the recommended path, because it also enforces the ordering:

../chaos-study/scripts/Invoke-ChaosStudy.ps1 -System "orders-api"

The actions below are the same machine, exposed for expert use. Whichever route you take, exit 26 means the study is waiting on a person: one question is open. Ask the customer that single question with the host's interactive mechanism (ask_user where available), wait for their reply, pass it back with -Answer, and repeat. Never batch the questions into one message, never answer on the customer's behalf, and never treat 26 as a failure.

1. Start a brief

./scripts/Invoke-ChaosStudyDesign.ps1 -Action start -System "orders-api"

Prints a briefId. Every later action takes -BriefId.

2. Record what you read

./scripts/Invoke-ChaosStudyDesign.ps1 -Action analyze -BriefId <id> `
  -Observed 'code=src/Orders/PaymentGateway.cs:14 Charge() has no retry or timeout' `
  -Observed 'dependencies=orders-api -> payments-api (synchronous, in request path)' `
  -Observed 'telemetry=metrics:checkoutSuccessRate, metrics:paymentGatewayErrors' `
  -Unavailable 'identity=no access to the subscription from this session'

Repeat -Observed/-Unavailable per area. All eight must be accounted for.

3. Propose candidates

./scripts/Invoke-ChaosStudyDesign.ps1 -Action candidates -BriefId <id> -CandidateFile ./candidates.json

Each candidate needs the eight required fields. Ranking is computed: availability, then confidence, then evidence count, then whether exposure is computable. Pass -Region/-Workspace to intersect with live discovery.

4. Interview

./scripts/Invoke-ChaosStudyDesign.ps1 -Action interview -BriefId <id>
./scripts/Invoke-ChaosStudyDesign.ps1 -Action answer -BriefId <id> -QuestionId purpose -Answer "..."

interview prints the single next open question and exits 26. The question is also emitted between CHAOS-QUESTION-BEGIN / CHAOS-QUESTION-END as JSON — questionId, prompt, why, choices, grounded — so an orchestrator can ask exactly one question without parsing prose. When choices is present the answer must be one of them verbatim. answer accepts or refuses; a refusal leaves the question open and exits 24 — it is not an error, it is the skill doing its job.

5. Recommend, then confirm

./scripts/Invoke-ChaosStudyDesign.ps1 -Action recommend -BriefId <id>
./scripts/Invoke-ChaosStudyDesign.ps1 -Action confirm -BriefId <id> `
  -Select <candidateId> -ConfirmPhrase "study <candidateId> on <system> <hash8>"

recommend requires every question answered, and exits 26 because the choice is the customer's to make. confirm requires the exact case-sensitive phrase printed by recommend.

6. Hand off

../chaos-study-scope/scripts/Invoke-ChaosStudyScope.ps1 -Brief <briefPath> -Workspace ... -ResourceGroup ...

Scope reads the confirmed brief for hypothesis, steady state, failure mechanism, mechanism evidence, mechanism probe, exposure inputs, scenario parameters, blast radius and windows. Anything passed explicitly on the command line wins, so the brief is a starting point rather than a cage.

Artifacts

Briefs live at <studyRoot>/briefs/<briefId>/study-brief.v1.json, alongside but separate from studies — a brief exists before there is a workspace to hash. The brief carries briefVersion, state, the analysis with citations, candidates, the interview transcript including refusals, the recommendation, the customer confirmation, and briefHash.

States advance monotonically: DRAFT → ANALYZED → CANDIDATES → INTERVIEWING → RECOMMENDED → CONFIRMED. A confirmed brief is immutable; further writes exit 13.

Exit codes

Code	Meaning
0	Success
1	Unexpected failure
13	Brief already confirmed — start a new one
24	Design incomplete — unaccounted area, vague answer, or missing field
26	Awaiting customer input — a question is on the table. Not a failure. Ask it, then resume.

What this skill does not do

It never contacts Azure to mutate anything, never creates a workspace, never validates a configuration and never injects a fault. It reads, asks, ranks and records. Everything with a blast radius belongs to chaos-study-scope and chaos-study-run, behind their own consent gates.