In plain English

A trained model is billions of numbers (weights), normally stored as 16 bit floats. Quantization stores those numbers with fewer bits, for example 8 or 4 instead of 16. Each number becomes a little less exact, but the model gets smaller and faster.

Think of a ruler. A 16 bit ruler has 65,536 marks, so any length lands very close to a mark. An 8 bit ruler has 256 marks and a 4 bit ruler only 16. Quantization picks where to put those few marks (the scale) and then snaps every number to the nearest mark (rounding). If the marks are placed well, the snapped numbers are close enough that the model still gives nearly the same answers.

Why bother: half the bits means half the memory, half the data to move from HBM on every token, and on NVIDIA Tensor Cores roughly double the math throughput. For LLM inference, where token generation is limited by memory bandwidth, that translates almost directly into more tokens per second and lower cost per token.

Remember

Quantization = fewer bits per number: FP16 (2 bytes) to FP8 or INT8 (1 byte) to FP4 or INT4 (half a byte).
Two steps: pick a scale so the values fit the small format, then round each value to the nearest representable number.
Gains: memory and bandwidth drop in proportion to bits; Tensor Core throughput roughly doubles each time bits halve (on GPUs that support that format).
Cost: rounding error. Good scaling (per channel or per block) and methods like QAT keep accuracy close to the original.

Memory math: what the sizes mean

Bytes per parameter is just bits divided by 8. FP32 is 32 bits, so 4 bytes. FP16 and BF16 are 16 bits, so 2 bytes. FP8 and INT8 are 1 byte. FP4 and INT4 are 4 bits, so half a byte.

Weight size = number of parameters x bytes per parameter. The shortcut: 1 billion parameters at 1 byte is 1 GB. So a 7B model in FP16 is 7 x 2 = 14 GB, and a 70B model in FP16 is 70 x 2 = 140 GB. Quantize that 70B model to FP8 and it becomes 70 GB; to FP4 about 35 GB (a bit more with the scales, about 39 GB for NVFP4).

Why this matters: 140 GB does not fit on one 80 GB H100, so FP16 needs at least two GPUs. At FP8 (70 GB) the weights fit on one H100, though with little room left for the KV cache, and fit comfortably on one H200 (141 GB). A 405B model at FP8 is 405 GB: one 8 GPU DGX H100 (640 GB total) holds it.

Model	FP32 (4 B)	FP16 / BF16 (2 B)	FP8 / INT8 (1 B)	FP4 / INT4 (0.5 B)
7B	28 GB	14 GB	7 GB	3.5 GB
70B	280 GB	140 GB	70 GB	35 GB
405B	1,620 GB	810 GB	405 GB	about 203 GB

Inference needs more than the weights. The KV cache grows with every token of every active request. For a 70B Llama 3 style model (80 layers, 8 KV heads of 128 dims) each token stores 2 x 80 x 8 x 128 values, about 0.33 MB in FP16. A single 128K token conversation is about 43 GB of KV cache on top of the weights. That is why FP8 KV cache (half the size) matters so much for serving many users.

Training needs far more. With mixed precision and the Adam optimizer, every parameter carries five things:

What is stored per parameter	Precision	Bytes
Weights used in the forward and backward pass	FP16 / BF16	2
Gradients	FP16 / BF16	2
Master copy of the weights (updates are too small for 16 bits)	FP32	4
Adam first moment (momentum)	FP32	4
Adam second moment (variance)	FP32	4
Total		16

So a 7B model needs 7 x 16 = 112 GB just for this state, before activations (which depend on batch size and sequence length). That is 8x the 14 GB needed to serve it in FP16. A 70B model needs about 1,120 GB, more than a whole DGX H100, which is why large training runs split these states across GPUs (ZeRO / FSDP sharding) and why training clusters are so much bigger than inference clusters.

Parameters (billions) Format

Remember

GB of weights = billions of parameters x bytes per parameter.
Inference memory = weights + KV cache + a little overhead. Training memory = about 16 bytes per parameter + activations.
Halving the bits halves the weights and the bandwidth needed per token; it does not shrink the KV cache unless you quantize that too.

How a number is stored

A floating point number has three parts: a sign bit, an exponent that sets how big the number is (its range), and a mantissa that sets how many digits of detail it keeps (its precision). More exponent bits means a wider range; more mantissa bits means finer steps. An integer format has no exponent: its values are evenly spaced whole numbers, and a separate scale turns them back into real values.

Bit layouts. Each square is one bit: sign, exponent, mantissa.

signexponent (range)mantissa (precision)integer value bits

Format	Bits (S/E/M)	Largest value	What it trades	Tensor Core support
FP32	32 (1/8/23)	about 3.4 x 10³⁸	Baseline, full range and precision	CUDA cores; TF32 mode on Tensor Cores
TF32	19 used (1/8/10)	same as FP32	FP32 range, FP16 like precision. A math mode, data stays FP32 in memory	Ampere and later
FP16	16 (1/5/10)	65,504	Good precision, narrow range; training needs loss scaling	Volta and later
BF16	16 (1/8/7)	same as FP32	FP32 range, less precision; no loss scaling needed	Ampere and later
FP8 E4M3	8 (1/4/3)	448	More precision, less range: weights and activations	Hopper, Ada, Blackwell
FP8 E5M2	8 (1/5/2)	57,344	More range, less precision: gradients in training	Hopper, Ada, Blackwell
INT8	8	127 (times scale)	Evenly spaced levels; needs a good scale and calibration	Turing and later
FP6 (E3M2 / E2M3)	6	28 / 7.5	Between FP8 and FP4	Blackwell
FP4 E2M1	4 (1/2/1)	6	Only 15 distinct values; needs block scaling	Blackwell and later
INT4	4	7 (times scale)	Common for weight only quantization (AWQ, GPTQ)	Turing, Ampere, Ada (not Hopper Tensor Cores)

The whole FP4 E2M1 number line is just: 0, 0.5, 1, 1.5, 2, 3, 4, 6 (and the negatives). Notice the steps get wider as values grow. That is the key difference between float and integer grids: floats put more levels near zero, where most neural network weights live, while integers space levels evenly.

Where the representable values sit between 0 and the format maximum (scaled to the same width).

How quantization works, step by step

Take four weights: -2.1, 0.3, 1.0, 3.2. Quantize them to INT8 with a symmetric scale:

Find the range. The largest absolute value (amax) is 3.2.
Pick the scale. INT8 goes up to 127, so scale = amax / 127 = 3.2 / 127 = 0.0252. One integer step is now worth 0.0252.
Divide and round. -2.1 / 0.0252 = -83.3 becomes -83; 0.3 becomes 12; 1.0 becomes 40; 3.2 becomes 127. These small integers are what gets stored, 1 byte each, plus one scale for the whole group.
Dequantize when needed. Multiply back by the scale: -83 x 0.0252 = -2.092, 12 x 0.0252 = 0.302, 40 x 0.0252 = 1.008, 127 x 0.0252 = 3.2. The error is at most half a step (0.0126).

In practice the GPU usually does not even dequantize: Tensor Cores multiply the low precision numbers directly and apply the scales to the result, which is where the speedup comes from.

Clipping. If you choose a smaller amax than the true maximum, values above it are clipped to the top level. You lose the outlier but every other value gets finer steps. Calibration methods (max, percentile, entropy/KL) are different ways of picking that trade off.

Symmetric vs asymmetric. Symmetric maps zero to zero and uses the same scale on both sides (common for weights). Asymmetric adds a zero point offset so a range like 0 to 6 uses all levels (useful for activations after ReLU).

The same four weights before and after INT8 and FP4 quantization.

Granularity: how many values share one scale

One scale for a whole tensor is cheap, but a single large value (an outlier) stretches the scale and wastes levels for everything else. LLM activations are famous for a few channels with values 100x bigger than the rest. The fix is to use more scales, each covering fewer values.

Granularity	One scale per	Overhead	Used by
Per tensor	Whole weight matrix or activation	Negligible	FP8 inference and FP8 training (Transformer Engine, Hopper)
Per channel	Each output row of a weight matrix	Small	INT8 weights in TensorRT
Per group	Every 64 or 128 weights	About 0.1 to 0.25 bit per value	INT4 weight only (AWQ, GPTQ, group size 128)
Per block, MXFP4 / MXFP8	Every 32 values, power of two scale (E8M0)	0.25 bit per value	OCP microscaling standard, Blackwell
Per block, NVFP4	Every 16 values, FP8 E4M3 scale, plus one FP32 scale per tensor	0.5 bit per value (about 4.5 bits total)	NVIDIA's FP4 format, Blackwell and later

Why NVFP4 beats MXFP4 on accuracy: its blocks are smaller (16 vs 32 values, so an outlier affects fewer neighbors) and its block scale is a real FP8 number instead of a power of two, so the scale can match the block's range closely instead of rounding up to the next power of two. NVIDIA reports NVFP4 within about 1% of FP8 accuracy on large models such as DeepSeek R1, at roughly 3.5x less memory than FP16.

NVFP4 layout: every block of 16 FP4 values carries its own FP8 scale; one FP32 scale covers the whole tensor.

Try it: quantization lab

64 sample weights from a normal distribution (like real layer weights). Pick a format and watch how far each value moves. Add an outlier to see why per tensor scaling struggles and block scaling helps.

Format Add an outlier (value 25x larger)

Dots: original values. Bars: where each value landed after quantization. The shorter the gap, the better. Error is shown relative to the average weight size.

What gets quantized

Target	What it is	What you gain	Notes
Weights only (for example W4A16)	Store weights in INT4/FP4, compute in FP16/BF16	Memory and bandwidth. Big speedup for decode, which is memory bound	Works on GPUs without FP4 math (Hopper, Ada): weights are expanded back before the matmul. AWQ and GPTQ are the common methods
Weights and activations (W8A8, W4A4)	Both inputs to the matmul are low precision	Memory plus faster Tensor Core math, helps prefill (compute bound) too	FP8 on Hopper, FP4/NVFP4 on Blackwell; INT8 uses SmoothQuant to tame activation outliers
KV cache	Stored keys and values for past tokens	More concurrent users and longer contexts in the same memory	FP8 KV cache is common in TensorRT-LLM; NVFP4 KV cache on Blackwell
Gradients and optimizer (training)	Backward pass values	Faster training, less memory	FP8 training on Hopper uses E5M2 for gradients; NVFP4 training on Blackwell

Label format: W4A16 means 4 bit weights, 16 bit activations. W8A8 means both 8 bit. The exam may describe the goal instead of the label: "fit a bigger model in the same GPU memory" points to weight quantization; "raise Tensor Core throughput" points to quantizing both.

The methods: PTQ, QAT, QAD and friends

Method	How it works	Cost	Accuracy	When to pick it
PTQ (post-training quantization)	Take the finished model, run a few hundred calibration samples to measure value ranges, set scales, convert. No training	Minutes to hours on one GPU	Usually near lossless for FP8 and INT8; can drop for 4 bit	Default first step. No training data or pipeline needed
SmoothQuant (a PTQ technique)	Moves the difficulty of activation outliers into the weights with a per channel scaling, so both can be INT8	Same as PTQ	Good W8A8 on models with activation outliers	INT8 on GPUs without FP8
AWQ (activation-aware weight quantization)	Protects the small set of weights that matter most (found from activations) by scaling them before INT4 quantization	Same as PTQ	Strong INT4 weight only results	W4A16 on Ampere, Ada, Hopper
GPTQ	Quantizes weights one column at a time and adjusts the rest to cancel the error	PTQ, a bit slower	Strong 3 to 4 bit weight only	Weight only, often from the open source ecosystem
QAT (quantization-aware training)	A short fine tune with fake quantization inserted, so the model learns weights that survive rounding	A training run (short, a fraction of original)	Recovers most of what PTQ lost	PTQ accuracy is not good enough, and you can fine tune
QAD (quantization-aware distillation)	Like QAT, but the quantized student also learns to match the full precision teacher's outputs	Training run plus teacher inference	Highest recovery of the three	Aggressive formats like NVFP4 where every point of accuracy matters
Low precision training (FP8, NVFP4)	Train in low precision from the start; Transformer Engine keeps per tensor scales from the history of max values	Built into training	Matches BF16 when done right	Pretraining on Hopper (FP8) or Blackwell (NVFP4)

All of PTQ, QAT and QAD, plus pruning, distillation and speculative decoding, live in NVIDIA Model Optimizer (formerly TensorRT Model Optimizer). Its quantized checkpoints feed TensorRT-LLM, TensorRT, vLLM and SGLang, and NIM picks a quantized profile that matches the detected GPU. Transformer Engine handles FP8 (Hopper) and FP4 (Blackwell) automatically during training and inference.

Choosing a format

Format	Bytes per param	70B model weights	Speed vs FP16	Accuracy risk	Runs fast on
FP16 / BF16	2	140 GB	1x	None (reference)	Volta / Ampere and later
FP8	1	70 GB	About 2x math, 2x less memory traffic	Very low with per tensor scaling	H100, H200, L40S, L4, Blackwell
INT8 (W8A8)	1	70 GB	About 2x	Low with calibration and SmoothQuant	Turing and later (A100, T4)
INT4 weight only	0.5 + scales	about 37 GB	Faster decode (memory bound), math still FP16	Low to moderate (AWQ/GPTQ help)	Most GPUs
NVFP4	about 0.56	about 39 GB	About 2x over FP8 math on Blackwell	About 1% vs FP8 with good PTQ, less with QAT/QAD	B200, B300, GB200, GB300, RTX PRO 6000 Blackwell

Quick rule: on Hopper pick FP8 (it has no FP4 Tensor Cores). On Blackwell pick NVFP4 for the biggest speed and memory win, FP8 when accuracy is paramount. On Ampere (A100) there is no FP8: use INT8 or INT4 weight only.

Exam traps

Quantization changes how numbers are stored; it does not remove weights. Removing weights is pruning; training a smaller model to copy a big one is distillation.
PTQ needs only a small calibration set and no retraining. QAT and QAD need a (short) training run.
FP8 E4M3 is for weights and activations (precision); E5M2 is for gradients (range). Not the other way round.
TF32 is a Tensor Core math mode, not a storage format: data stays FP32 in memory.
BF16 has the same range as FP32 (8 exponent bits); FP16 has more precision but a narrow range and needs loss scaling in training.
FP8 needs Hopper, Ada or Blackwell. A100 (Ampere) has no FP8 Tensor Cores. FP4/NVFP4 needs Blackwell. Hopper Tensor Cores dropped INT4.
NVFP4 = 16 value blocks with an FP8 E4M3 scale plus an FP32 tensor scale. MXFP4 = 32 value blocks with a power of two (E8M0) scale. Both store E2M1 values.
Weight only quantization speeds up decode (memory bound) even when the math stays in FP16; it helps prefill much less.
Datasheet TFLOPS for low precision are often quoted with 2:4 sparsity, which doubles the number. Sparsity is a different trick (zeros), not quantization.
A quantized TensorRT engine is still tied to the GPU model and TensorRT version it was built for.

Check yourself