Chapter 19: Model Optimization & Inference¶

Make models small and fast: int8 quantization, magnitude pruning, knowledge distillation, and the latency/throughput/memory trade-offs of production inference.

Metadata¶

Learning Objectives¶

• Quantize weights to int8 with a scale factor and bound the reconstruction error
• Prune weights by magnitude and measure the accuracy/sparsity trade-off
• Distill a small student from a large teacher with temperature-scaled soft targets
• Reason about precision — float32 vs float16 vs int8 and where each is safe
• Estimate model size and memory footprint before and after optimization
• Trade off latency vs throughput with batching
• Profile an inference path and find the bottleneck
• Choose an optimization appropriate to the deployment target (edge, server, batch)

What's Included¶

Notebooks¶

Scripts¶

• config.py — Chapter config: seeds, precision byte sizes
• optimize.py — int8 quantization, magnitude pruning, distillation softmax, size estimation
• inference.py — Latency/throughput model for batching and a simple profiler

Exercises¶

• Problem Set 1: Quantize & Prune — Bound int8 error and prune to an exact sparsity
• Problem Set 2: Distill & Serve — Generate soft targets and reason about size and batching
• Solutions — in exercises/solutions/ (notebooks and solutions.py for CI)

Diagrams (Mermaid)¶

• optimization_pipeline.mermaid, precision_tradeoff.mermaid

Read Online¶

• 19.1 Introduction — int8 quantization and magnitude pruning with measured error
• 19.2 Intermediate — Temperature-scaled soft targets to train a small student
• 19.3 Advanced — Precision/size accounting and the latency-throughput trade-off

Or try the code in the Playground.

How to Use This Chapter¶

git clone https://github.com/luigipascal/berta-chapters.git
cd berta-chapters/chapters/chapter-19-model-optimization-and-inference
pip install -r requirements.txt
jupyter notebook notebooks/01_quantization_pruning.ipynb

Created by Luigi Pascal Rondanini | Generated by Berta AI