LM8-T4 — Quantization Concepts and Trade-Offs

Learning objectives

After this lesson, you should be able to:

• Quantise and dequantise a tensor with a per-tensor and a group-wise scale, and bound the error.
• Explain why outliers hurt and how group-wise scales and non-uniform grids help.
• Compute the memory for different bit widths, including the overhead of scales.
• Distinguish post-training quantisation from quantisation-aware training, and explain QLoRA.

The idea: a coarser ruler

A float stores a number with 16 or 32 bits. An 8-bit integer can hold 256 values; a 4-bit integer only 16. To store weights in 4 bits we choose 16 evenly spaced levels covering the range of the weights, and replace each weight by the index of the nearest level. Alongside the integers we keep a single floating-point number, the scale, which says how far apart the levels are. To use the weight we multiply the integer by the scale (dequantise).

Absmax quantisation

The simplest scheme sets the scale from the largest absolute value, so that it maps to the largest integer. The error of each weight is at most half a step. The step is proportional to the largest weight in the group: one large weight makes the step big for everybody.

Outliers

Weights are mostly small with a few large values. In a tensor with one weight of 1.0 among values of size 0.02, a per-tensor 4-bit scale is 1.0/7 = 0.14, larger than almost every ordinary weight: nearly all of them round to zero and the layer is destroyed. In the experiment below the relative error is 0.98 for per-tensor and 0.11 for groups of 64. Group-wise (block-wise) quantisation gives each block of 32 to 128 weights its own scale, so an outlier only damages its own block. The scales cost memory: 16 bits per group of 64 adds 0.25 bits per weight.

Better grids

Evenly spaced levels are not the best use of 16 values for weights that follow a bell-shaped distribution. The NormalFloat (NF4) grid of QLoRA places levels at quantiles of a normal distribution, with more levels near zero where most weights are. Double quantisation also quantises the scales to save a little more.

When to quantise

• Post-training quantisation (PTQ): take a trained model and round it; no training. Works well at 8 bits and often at 4 bits for large models; at lower bit widths, methods that use calibration data to choose the rounding (such as GPTQ and AWQ) reduce the damage.
• Quantisation-aware training (QAT): simulate the rounding during training so that the model learns to be robust to it; better quality at very low bit widths, much more expensive.
• QLoRA: freeze a 4-bit base and train LoRA adapters in 16-bit on top. Gradients flow through the dequantised weights; only the adapters are updated. This is what lets a 7B model be fine-tuned on a single consumer GPU.

The mathematics: scale, rounding and error

For a group of weights w and a signed integer range [−q_max, q_max] with q_max = 2^b−1 − 1 (127 for 8 bits, 7 for 4 bits):

b

number of bits per weight.

s

the scale: the distance between neighbouring levels; stored in 16 or 32 bits per group.

q_i

the stored integer for weight i.

ŵ_i

the dequantised weight used in computation.

Error bound. Rounding to the nearest level gives

For the weights [0.5, −1.2, 0.03, 0.9]: with 8 bits, s = 1.2/127 = 0.00945, the integers are [53, −127, 3, 95] and the dequantised weights [0.5008, −1.2, 0.0283, 0.8976]; the largest error is 0.0024. With 4 bits, s = 1.2/7 = 0.1714, the integers are [3, −7, 0, 5] and the weights [0.514, −1.2, 0, 0.857]; the largest error is 0.043 and 0.03 became 0. Going from 8 to 4 bits multiplies the step by 127/7 = 18.

Memory. A model with P parameters in b bits with a 16-bit scale per group of g weights needs

For P = 6.7 · 10⁹: 16 bits 13.4 GB; 8 bits 6.7 GB; 4 bits with groups of 64: 6.7 · 10⁹ · 4.25/8 = 3.56 GB. Ignoring the scales (3.35 GB) understates it by 6%.

Effect on the output. A matrix product with weights perturbed by independent errors of standard deviation about s/√12 changes the output by an amount that grows like √(in) times that, relative to a signal that grows like in for aligned inputs: networks are robust because errors are independent and average out, but systematic errors (everything rounding to zero in one layer) are not.

Merging adapters into quantised weights. The merged weight W + (α/r)BA is no longer on the quantisation grid; re-quantising it re-introduces rounding error, which can erase a small adapter update. Common practice is to merge into the 16-bit original and quantise the result, or to keep the adapter separate.

A worked example: measuring the error

The code implements absmax quantisation with optional groups, reproduces the numbers above, and measures the relative error on a random matrix with and without an outlier.

import torch

def quantize(w, bits, group=None):
    qmax = 2 ** (bits - 1) - 1
    w2 = w.reshape(-1, group) if group else w.reshape(1, -1)
    scale = w2.abs().amax(dim=1, keepdim=True) / qmax
    q = torch.round(w2 / scale).clamp(-qmax, qmax)
    return q.to(torch.int8), scale

def dequantize(q, scale, shape):
    return (q.float() * scale).reshape(shape)

torch.manual_seed(0)
w = torch.tensor([0.5, -1.2, 0.03, 0.9])
q, s = quantize(w, 8)
print(q.flatten().tolist(), round(s.item(), 5), [round(v, 4) for v in dequantize(q, s, w.shape).tolist()])
# [53, -127, 3, 95] 0.00945 [0.5008, -1.2, 0.0283, 0.8976]
q4, s4 = quantize(w, 4)
print(q4.flatten().tolist(), round(s4.item(), 4), [round(v, 3) for v in dequantize(q4, s4, w.shape).tolist()])
# [3, -7, 0, 5] 0.1714 [0.514, -1.2, 0.0, 0.857]

W = torch.randn(256, 256) * 0.02
for bits in (8, 4):
    q, s = quantize(W, bits)
    err = (W - dequantize(q, s, W.shape)).pow(2).mean().sqrt().item()
    print(bits, "bits: relative error", round(err / W.std().item(), 4))      # 8 -> 0.0104, 4 -> 0.188

W2 = W.clone()
W2[0, 0] = 1.0                                                              # one outlier among 65,536 weights
for g in (None, 64):
    q, s = quantize(W2, 4, g)
    err = (W2 - dequantize(q, s, W2.shape)).pow(2).mean().sqrt().item()
    print("4 bits, group", g, "relative error", round(err / W2.std().item(), 3))   # None -> 0.981, 64 -> 0.11

P = 6.7e9
for bits, g in ((16, None), (8, None), (4, 64)):
    extra = 16 / g if g else 0
    print(bits, round(P * (bits + extra) / 8 / 1e9, 2), "GB")             # 13.4, 6.7, 3.56

The jump from 0.188 to 0.981 shows how a single large weight spoils a per-tensor scale, and the 0.11 shows that groups contain the damage to the block that holds the outlier.

What it does to a model

Smaller models usually tolerate quantisation less well than larger ones, and some layers (embeddings, the output head, layer norms) are often kept in higher precision.

Engineering implications and limitations

Evaluate after quantising. Compare the quantised and original model on your task and on a general benchmark. Aggregate scores can hide failures on specific skills, such as long-context or arithmetic.

Use calibrated methods for 4 bits. Plain rounding is fine at 8 bits. For 4 bits and below, use a method designed for it (NF4 with group scales, GPTQ, AWQ) rather than naive rounding.

Keep sensitive parts in higher precision. Embeddings, output layer and norms are small and sensitive; quantising them saves little and costs quality.

Speed is not guaranteed. Quantised weights must be dequantised on the fly or used in dedicated kernels. Memory-bound decoding speeds up with fewer bytes to read; compute-bound training may not. Measure on your hardware.

Adapters in 16-bit. In QLoRA the adapter parameters stay in 16- or 32-bit floating point; only the frozen base is quantised.

Record the scheme. Store the bit width, group size, method and library version with the checkpoint; a quantised model cannot be interpreted without them.

Check licence terms and integrity. Quantised copies of models distributed by third parties should be verified against the original’s hash or provenance before use.

For robots. On-board inference on embedded hardware is memory-limited; 4-bit weights let a 3–7B model fit on an edge GPU. Test the quantised model on the robot’s own commands and in the simulator, not only on text benchmarks.

Common misconceptions

Engineering takeaways

1. Absmax quantisation: s = max|w|/q_max, q = round(w/s), ŵ = s·q; the error is at most s/2.
2. Going from 8 to 4 bits multiplies the step by about 18 (127/7); one outlier can force a large step on a whole tensor.
3. Group-wise scales contain outliers (error 0.98 → 0.11 in the experiment) at a cost of 16/g extra bits per weight.
4. Memory ≈ P·(b + 16/g)/8 bytes: 13.4 GB, 6.7 GB and 3.56 GB for a 6.7-billion-parameter model at 16, 8 and 4 bits (g = 64).
5. QLoRA trains LoRA adapters on a frozen 4-bit base; merging adapters into quantised weights needs care.

References and further reading

See Appendix E of the book for LoRA; the quantisation arithmetic here can be reproduced with the code above, and the quality statements are general guidance, not measurements on your model.

• Dettmers, T. et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. arXiv:2305.14314 (NF4 and double quantisation).
• Dettmers, T. et al. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. arXiv:2208.07339.
• Dettmers et al. (2023), QLoRA (research paper)
• Build a Large Language Model (From Scratch), Appendix E: Parameter-efficient Finetuning with LoRA (primary book reference)

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: choosing a base model for a task and a budget, and managing checkpoints so that experiments stay reproducible.

Related content

Nothing below is required to finish this lesson.

• Before
  LM8-T3: Memory and Compute for Training

  Estimate weights, gradients, optimizer state and activations in bytes.

• Lab
  LM8-L1: Estimate Trainable Parameters and Memory

  Write a calculator for parameters, optimizer state and memory under full fine-tuning, LoRA and quantization.