Every state-changing endpoint accepts an operator-generated idempotency key. Client generates a UUID per logical operation; server returns the original response on duplicate submission within the retry window. Retries become safe — if the operation succeeded but the response was lost, the retry sees the cached success.

Without idempotency, naive retry logic causes duplicate bets, double deposits, and chaos in reconciliation. The pattern is required for any write endpoint, especially under flaky mobile network conditions in APAC — where retries happen constantly without the frontend knowing.

Retries wait exponentially longer between attempts — 100ms, 200ms, 400ms, 800ms. Jitter adds random variance so thousands of clients don’t synchronize their retries into a thundering herd. Capped at a reasonable maximum (usually 5-8 attempts, 30s max wait).

Dumb retry loops that hammer a struggling server make incidents worse, not better. Exponential backoff with jitter lets the backend recover while clients eventually succeed. Combined with idempotency keys, retries become safe to apply aggressively.

Track failure rate per endpoint; when failures exceed a threshold (e.g., 50% in 30 seconds), open the circuit — stop sending requests, fail fast locally, periodically test if backend recovered. Prevents cascading failure when one API endpoint is down from exhausting your client resources.

Especially important for operators serving millions of requests per hour. Without circuit breakers, a dependency outage spawns millions of queued requests that pile up, exhaust connection pools, and bring down your own frontend — magnifying a vendor incident into an operator outage.

Every API call has an explicit timeout. Odds reads: 500ms-1s. Bet submissions: 3-5s. Settlement reads: 10s. Long-running queries: 30s with pagination. No request should ever hang indefinitely waiting on a backend that silently died.

Default HTTP library timeouts (sometimes measured in minutes) are catastrophically wrong for sportsbook. A 60-second hang on an odds-feed call is a player who leaves and never comes back. Explicit, aggressive timeouts fail fast and let fallback logic engage.

When settlement webhooks or batch calls fail after all retries, the failed message goes to a dead letter queue (DLQ) for manual review — never silently dropped. Settlement failures are the highest-stakes error class because they directly affect player balances and operator financial correctness.

Automated retries handle 95% of transient settlement issues. The remaining 5% — malformed payloads, business-rule violations, data mismatches — need human eyes. DLQ ensures nothing is lost; absence of DLQ guarantees eventual reconciliation disasters.

Every API call emits metrics (latency, status code, endpoint), logs (correlation ID, request snapshot), and distributed traces (full path across services). When something breaks at 2am during live NBA, your on-call engineer needs this data instantly — not after hours of code archaeology.

Observability is the pattern that makes all other patterns debuggable. A circuit breaker that opens for mysterious reasons is useless; a circuit breaker whose state transitions emit metrics with specific failure reasons is diagnostic gold. The best operators we work with on WSGaming deployments instrument everything from day one.