Ch 19: Model Optimization & Inference - Advanced¶
Track: Advanced | Try code in Playground | Back to chapter overview
Read online or run locally
You can read this content here on the web. To run the code interactively, either use the Playground or clone the repo and open chapters/chapter-19-model-optimization-and-inference/notebooks/03_inference_optimization.ipynb in Jupyter.
Chapter 19: Model Optimization — Notebook 03 (Inference Optimization)¶
Optimization pays off at inference. We account for model size across precisions and chart the latency vs throughput trade-off that batching creates.
What you'll learn¶
| Topic | Section |
|---|---|
| Bytes per parameter across precisions | §1 |
| Latency vs throughput under batching | §2 |
| Choosing an optimization for the target | §3 |
Time estimate: 2.5 hours
Key concepts¶
- Memory footprint — params x bytes/param; int8 is 4x smaller than float32.
- Latency — time for one request; throughput — requests per second.
- Batching — amortizes fixed overhead, raising throughput at the cost of latency.
Inference economics come down to memory (precision x params) and the latency/throughput trade-off of batching. The right optimization depends on whether you're serving edge, online, or batch workloads.
Run the full notebook in the chapter folder for code and outputs.
Generated by Berta AI