Chapter 20: Building Production AI Systems¶

Reliability engineering for AI services: rate limiting, circuit breakers, backoff, canary rollouts, observability, and cost control under real traffic.

Metadata¶

Learning Objectives¶

• Rate-limit requests with a token-bucket and reason about burst vs sustained load
• Contain failures with a circuit breaker (closed / open / half-open)
• Retry safely with exponential backoff and jitter
• Roll out safely with deterministic canary routing and rollback
• Instrument services with structured logs and the golden signals
• Reason about SLOs — latency percentiles, error budgets, and availability
• Control cost — caching, batching, and right-sizing inference
• Navigate the human side: on-call, incident review, and technical debt

What's Included¶

Notebooks¶

Scripts¶

• config.py — Chapter config: SLO targets, limiter and breaker defaults
• reliability.py — Token bucket, circuit breaker, backoff, canary routing
• observability.py — Structured JSON logging and golden-signal aggregation

Exercises¶

• Problem Set 1: Limiting & Breaking — Verify token-bucket burst behaviour and circuit-breaker tripping
• Problem Set 2: Rollouts & Signals — Check canary split stability and compute p99 latency
• Solutions — in exercises/solutions/ (notebooks and solutions.py for CI)

Diagrams (Mermaid)¶

• request_path.mermaid, circuit_breaker.mermaid

Read Online¶

• 20.1 Introduction — Token-bucket rate limiting, circuit breakers, and backoff
• 20.2 Intermediate — Deterministic canary routing and rollback
• 20.3 Advanced — Golden signals, structured logs, SLOs, and cost control

Or try the code in the Playground.

How to Use This Chapter¶

git clone https://github.com/luigipascal/berta-chapters.git
cd berta-chapters/chapters/chapter-20-building-production-ai-systems
pip install -r requirements.txt
jupyter notebook notebooks/01_reliability_patterns.ipynb

Created by Luigi Pascal Rondanini | Generated by Berta AI