resilience

Resilience Patterns

When writing or reviewing code that makes outbound calls (REST APIs, database operations, queue interactions), always consider transient failure handling.

Core Principle

Every outbound HTTP call should have a retry strategy unless there's a specific reason not to. Network blips, temporary 503s, and connection resets are normal in distributed systems. Code without retry is fragile by default.

Available Tools

This repo provides retry utilities in packages/shared/src/utils/retry.ts:

@Retry decorator (preferred for methods)

import { Retry } from "shared";

class MyWorker {
  @Retry({ maxRetries: 3, baseDelayMs: 1000, isRetryable: () => true })
  async callExternalService(): Promise<void> {
    const res = await fetch("http://api/endpoint", { method: "POST" });
    if (!res.ok) throw new Error(`HTTP ${res.status}`);
  }
}

withRetry function (for inline/conditional logic)

import { withRetry } from "shared";

const result = await withRetry(
  () => fetch("http://api/data").then(r => r.json()),
  { maxRetries: 3, baseDelayMs: 500, isRetryable: (err) => isTransient(err) }
);

Decision Checklist

When you encounter an outbound call, ask:

1. Is this call idempotent? If yes (GET, PUT, DELETE with idempotency key) -> retry aggressively.
2. Is this call non-idempotent? If yes (POST creating resources) -> retry cautiously, only on network errors (never got a response), not on 5xx (server may have processed it).
3. Is this best-effort? (e.g., triggering reports, sending notifications) -> retry with try/catch at call site so failures don't propagate.
4. Is this critical path? (e.g., saving user data) -> retry and propagate failure if exhausted.

What to Retry

Condition	Retry?	Reason
Network error (ECONNREFUSED, ETIMEDOUT)	Yes	Transient connectivity issue
HTTP 429 (Too Many Requests)	Yes	Rate limiting, respect Retry-After header
HTTP 500, 502, 503, 504	Yes	Server-side transient failures
HTTP 400, 401, 403, 404, 409	No	Client errors indicate a permanent problem
DNS resolution failure	Yes (limited)	May be temporary DNS propagation

Configuration Defaults

For service-to-service calls within the cluster:

• maxRetries: 3
• baseDelayMs: 1000 (1 second)
• maxDelayMs: 5000 (5 seconds)

For external API calls:

• maxRetries: 5
• baseDelayMs: 500
• maxDelayMs: 30000 (respect rate limits)

Anti-patterns

• No retry at all on service-to-service calls -- fragile
• Retry on 4xx -- wastes resources, will never succeed
• Retry without backoff -- can overwhelm a recovering service
• Retry without a cap -- can hang indefinitely
• Retry non-idempotent POST without checking -- can create duplicates
• Stacking retry layers -- never combine an HTTP client's built-in retry with withRetry/@Retry around the same call, or attempts multiply (maxRetries × client.limit). See below.

Single retry layer

Retry belongs to exactly one layer per call. The ky HTTP clients used by the CLI (apps/cli/src/utils/api-client.ts) and Portal (apps/portal/src/lib/api-client.ts) set retry: 0 precisely so they never stack with the shared cockatiel retry.

If a call needs transient-failure retries, pick one layer:

• Enable the HTTP client's own retry (with its own backoff), OR
• Wrap the call in withRetry / @Retry from packages/shared/src/utils/retry.ts.

Never both — combining them compounds attempts to maxRetries × client.limit.

Reference

Full API documentation: docs/architecture/retry.md