LM8-T1 — Full Versus Parameter-Efficient Fine-Tuning

Learning objectives

After this lesson, you should be able to:

• List the main parameter-efficient methods and what each trains.
• Compare full fine-tuning and PEFT on trainable parameters, memory, storage per task and serving.
• Compute the storage of several task-specific models under both approaches.
• Decide between the approaches from data size, task distance, hardware and deployment needs.

The idea: most of the model can stay as it is

A pretrained model contains general knowledge in billions of weights. Adapting it to a task rarely needs to change all of them. The change that fine-tuning makes to the weights tends to be small and structured, so it can be represented with far fewer numbers than the weights themselves. PEFT methods exploit this: freeze the base model and learn a compact description of the change.

The family

Why it works so well

The reported results for LoRA on large models are striking: Hu et al. (2022) report that for GPT-3 175B it cuts the number of trainable parameters by a factor of 10,000 and the GPU memory by a factor of three, with quality on the tasks they tested comparable to full fine-tuning. One reason is that the big model already represents the necessary features; the adaptation only has to redirect them. Another is that the low-capacity update acts as a regulariser against overfitting small data sets.

What it costs

PEFT is not free quality. On tasks far from the pretraining data, or when much new behaviour must be learned, full fine-tuning can reach higher quality, because it can change everything. PEFT also does not remove the memory of the base weights themselves: the frozen model must still be loaded for the forward and backward pass. What it removes is the gradients and optimizer state for the frozen weights, and the need to store a full copy per task.

The mathematics: what is trained and what is stored

Let the model have P parameters and a method train T of them, with T ≪ P for PEFT. The memory during training (Lesson LM8-T3) is approximately

b_w

bytes per stored base weight (4 for float32, 2 for 16-bit, 1 or 0.5 for quantised).

b_g, b_o

bytes per trainable parameter for its gradient and its optimizer state (for Adam in float32: 4 and 8).

T

number of trainable parameters; for full fine-tuning T = P.

Only the second term shrinks with PEFT; the first stays unless the base is quantised.

Storage per task. With K tasks, full fine-tuning stores K · P · b bytes. PEFT stores one base model and K adapters:

For GPT-2 small (P = 124,412,160, b = 4) and K = 3: S_full = 1,493 MB. A LoRA of rank 16 on the four weight matrices of every block has T = 12 · 16 · 16 · 768 = 2,359,296 parameters (9.4 MB). Then S_peft = 497.6 + 3 · 9.4 = 525.9 MB. For K = 100 tasks the figures are 49.8 GB and 1.44 GB. A service with many customers, each with a private fine-tune, is practical only with adapters.

Serving. Adapters can be merged into the base weights (no overhead, one model per task) or kept separate and swapped per request on a shared base (a small overhead per request, many tasks per GPU). Which is better depends on how many tasks share a GPU and how often they change.

Trainable share. T/P = 2,359,296 / 124,412,160 = 1.9% for the example. For a 7-billion-parameter model with rank 8 on the same matrices (32 layers, d = 4,096) it is 16.8 million, 0.25%.

A worked example: the storage of many tasks

The code compares the storage of 1, 3 and 100 task-specific models with full fine-tuning and with LoRA adapters, and the trainable shares for two models.

P = 124_412_160                         # GPT-2 small, tied head
b = 4                                   # bytes per parameter (float32)
L, d, r = 12, 768, 16
T = L * r * 16 * d                      # LoRA on qkv, proj, ff up, ff down: 16*d*r per layer
print(T, round(T * b / 1e6, 1), round(100 * T / P, 2))          # 2359296 9.4 1.9

def storage_mb(K, peft):
    return (P * b + K * T * b) / 1e6 if peft else K * P * b / 1e6

for K in (1, 3, 100):
    print(K, round(storage_mb(K, False)), round(storage_mb(K, True)))
# 1 498 507
# 3 1493 526
# 100 49765 1441

# a 7B-class model, rank 8
T7 = 32 * 8 * 16 * 4096
print(T7, round(100 * T7 / 6.7e9, 2))                           # 16777216 0.25

For one task, the adapter approach stores slightly more than full fine-tuning (the base plus an adapter); the saving begins with the second task. With 100 tasks the adapters need 3% of the storage.

Choosing

Engineering implications and limitations

Compare against full fine-tuning once. On your task and data, run both on a small scale and compare on a validation set. The quality gap, if any, is what you pay for the savings.

Match the learning rate to the method. LoRA and adapters usually need a higher learning rate than full fine-tuning (10⁻⁴ to 10⁻³ against 10⁻⁵).

Record the base model version with every adapter. An adapter is meaningless without the exact base weights it was trained against. Store the base model identifier and hash in the adapter’s metadata.

Do not stack adapters carelessly. Averaging or adding adapters trained for different tasks can work or can interfere; test any combination.

Check forgetting. PEFT often forgets less than full fine-tuning, but not always; measure general performance before and after.

Licence the pair. The adapter inherits obligations from the base model’s licence and from your data.

For robots. A fleet with different platforms or sites can share one base language model and carry a small adapter per platform; updating an adapter over a slow link is practical where re-sending a full model is not.

Common misconceptions

Engineering takeaways

1. PEFT trains a small set of parameters (adapters, low-rank updates, prompts, biases) on a frozen base; full fine-tuning trains everything.
2. Training memory ≈ P·b_w + T·(b_g + b_o): only the second term shrinks, unless the base is also quantised.
3. Storage for K tasks: K·P·b for full fine-tuning against P·b + K·T·b for adapters (1,493 MB against 526 MB for GPT-2 small, K = 3).
4. LoRA can be merged for no inference cost or kept separate for swapping; adapters need the exact base model.
5. Choose by data size, task distance from pretraining, hardware and the number of tasks; verify the quality gap on validation data.

References and further reading

See Appendix E of the book (parameter-efficient finetuning with LoRA).

• Hu, E. J. et al. (2022). LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685.
• Dettmers, T. et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. arXiv:2305.14314.
• Build a Large Language Model (From Scratch), Appendix E: Parameter-efficient Finetuning with LoRA (primary book reference)

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: low-rank adaptation (LoRA), derived step by step.

Related content

Nothing below is required to finish this lesson.

• Before
  LM7-T5: Evaluating Instruction Following and Data Contamination

  Define criteria, build a held-out set and spot contamination between training and evaluation data.

• Lab
  LM8-L1: Estimate Trainable Parameters and Memory

  Write a calculator for parameters, optimizer state and memory under full fine-tuning, LoRA and quantization.