developer-roadmap/roadmaps/inference-engineering/content/quantization@DaSqcLHfnJ4_n9utVVTSk.md

Quantization

Quantization converts model weights and other values from their native precision to a lower-precision format. Cutting precision in half doubles memory bandwidth for decode and enables twice the FLOPS for prefill on compatible Tensor Cores. The tradeoff is potential quality loss from reduced numerical precision, which can compound across the many calculations in a forward pass.

Visit the following resources to learn more:

• @opensource@AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
• @opensource@bitsandbytes
• @opensource@SmoothQuant
• @article@What is quantization?
• @video@How LLMs survive in low precision | Quantization Fundamentals