M4.4 — LLM Fundamentals

Learning objectives

After this lesson, you should be able to:

• Define a large language model as a decoder-style transformer trained for next-token prediction.
• Explain logits, softmax and the cross-entropy training loss.
• Describe the autoregressive generation loop.
• Explain greedy decoding, temperature and top-k sampling.
• Describe instruction tuning at a high level.
• List the main limits of a text-only LLM for robotics.

The one-sentence idea

A language model predicts the next token from the tokens so far, and generating text, or later robot actions, is just repeating that prediction and feeding each result back in.

Predict, choose, append, repeat.

What a large language model is

A large language model (LLM) is a very large decoder-style transformer (M3.4) trained to predict the next token (M3.5) on an enormous amount of text. That single objective, applied at scale, produces a model that can continue, summarise, translate, answer and follow instructions, because doing well at predicting text requires modelling the facts, patterns and reasoning the text contains.

For this roadmap an LLM matters in two ways. First, it supplies the language understanding half of a VLA: the instruction “put the red cup in the bin” has to be read. Second, many VLAs reuse an LLM’s architecture and pretrained weights as the backbone that produces actions, so understanding how an LLM generates text explains how such a model can generate actions too.

Predicting the next token

Given the tokens so far (M3.2), an LLM outputs one logit per vocabulary entry, a score for each possible next token. Softmax (M3.1) turns the scores into probabilities. The training loss is the cross-entropy between that distribution and the token that actually came next, and, thanks to the causal mask, one forward pass trains a prediction at every position at once: the inputs are the tokens [:-1] and the targets are the tokens [1:].

import math
import torch
from torch import nn

torch.manual_seed(0)
V, d, T_ = 50, 32, 8                          # vocabulary size, model width, sequence length

class TinyLM(nn.Module):
    def __init__(self):
        super().__init__()
        self.tok = nn.Embedding(V, d)
        self.pos = nn.Embedding(T_, d)
        layer = nn.TransformerEncoderLayer(d, 4, 64, dropout=0.0, batch_first=True, norm_first=True)
        self.blocks = nn.TransformerEncoder(layer, 2, enable_nested_tensor=False)
        self.head = nn.Linear(d, V)           # one logit per vocabulary entry

    def forward(self, ids):
        h = self.tok(ids) + self.pos(torch.arange(ids.shape[1]))
        mask = nn.Transformer.generate_square_subsequent_mask(ids.shape[1])    # causal
        return self.head(self.blocks(h, mask=mask))

model = TinyLM()
tokens = torch.randint(0, V, (4, T_ + 1))     # random token ids, standing in for text
logits = model(tokens[:, :-1])                # predict from all but the last token
loss = nn.functional.cross_entropy(logits.reshape(-1, V), tokens[:, 1:].reshape(-1))
print(logits.shape)
print(abs(loss.item() - math.log(V)) < 0.5)   # an untrained model is about as unsure as chance

torch.Size([4, 8, 50])
True

The logits have one row of 50 scores for each of the 8 positions in each of the 4 sequences. An untrained model knows nothing, so its loss is close to ln 50 ≈ 3.9, the loss of guessing uniformly among 50 tokens. Training lowers it by making the right next token more probable.

Generating text one token at a time

Generation repeats one step: run the model on the tokens so far, take the distribution for the last position, choose a token, append it, and go again until an end token appears. This is called autoregressive generation, because the model’s own output becomes its next input. A tiny hand-written next-token table shows the loop without any training:

import numpy as np

# P(next token | previous token), written by hand for illustration
table = {
    "<start>": {"pick": 0.7, "place": 0.3},
    "pick":    {"up": 0.8, "the": 0.2},
    "up":      {"the": 1.0},
    "the":     {"cup": 0.6, "bowl": 0.4},
    "cup":     {"<end>": 1.0},
    "bowl":    {"<end>": 1.0},
    "place":   {"the": 1.0},
}

seq = ["<start>"]
while seq[-1] != "<end>":
    options = table[seq[-1]]
    seq.append(max(options, key=options.get))       # greedy: take the most likely token
print(" ".join(seq[1:-1]))

pick up the cup

Choosing the most likely token each time is greedy decoding. It is deterministic but can be repetitive. Real models instead sample from the distribution, which gives varied output, and control how adventurous it is with a temperature: the logits are divided by it before softmax. Low temperature sharpens the distribution toward the top choice, and high temperature flattens it:

def softmax(z):
    e = np.exp(z - z.max())
    return e / e.sum()

logits = np.array([2.0, 1.0, 0.1])
for temp in (0.5, 1.0, 2.0):
    print(temp, softmax(logits / temp).round(3))

0.5 [0.864 0.117 0.019]
1.0 [0.659 0.242 0.099]
2.0 [0.502 0.304 0.194]

At temperature 0.5 the top token takes 86% of the probability, at 1.0 about 66%, and at 2.0 only about 50%. As the temperature approaches zero, sampling becomes greedy. Two other common controls cut off the unlikely tail before sampling: top-k keeps only the k most likely tokens, and top-p keeps the smallest set whose probability adds to p. After cutting, the remaining probabilities are renormalised:

p = softmax(logits)
k = 2
keep = np.argsort(-p)[:k]
q = np.zeros_like(p)
q[keep] = p[keep]
print((q / q.sum()).round(3))

[0.731 0.269 0.   ]

The third token is now impossible, and the other two share its 10% in proportion. Because each generated token needs a pass through the model, generation is far slower than reading a prompt, which is why real systems cache the attention keys and values of earlier tokens and compute only the new one.

From a text predictor to an assistant

The model produced by next-token pretraining is a powerful text continuer, not yet a helpful assistant. Two further stages are common. Instruction tuning fine-tunes it (M3.5) on examples of instructions and good responses. Preference-based methods then adjust its behaviour using human or automated judgements of which responses are better. The details vary between models, and the roadmap does not depend on them. What matters is that the same transformer can be steered toward different behaviour by further training.

That steerability is what a VLA builds on: instead of text responses, the training examples pair a camera image and an instruction with robot actions, and the model is trained to produce those (Phases 4 to 6).

Limits to keep in mind for robotics

• An LLM only sees tokens. A text-only LLM has no direct access to the camera, the robot’s state or the physical consequences of what it says. That gap is what vision-language-action models are meant to close.
• It can be confidently wrong. Generating plausible text is not the same as being correct, and the softmax probabilities are not guaranteed to reflect real certainty (M3.1). For a robot, a plausible but infeasible plan is dangerous, so outputs need checking.
• It has a finite context window. Only a limited number of recent tokens can be attended to, and attention cost grows with the square of that length (M3.3).
• It is slow and large. Autoregressive decoding, one token per pass, limits how often a robot can be commanded, a point Phase 8 returns to when it discusses latency.

Common misconceptions

Engineering takeaways

1. An LLM is a large decoder-style transformer trained to predict the next token.
2. Its output is one logit per vocabulary entry, and softmax makes them probabilities.
3. Training minimises cross-entropy with inputs tokens[:-1] and targets tokens[1:].
4. Generation is autoregressive: predict, choose a token, append it, repeat.
5. Greedy picks the top token. Temperature, top-k and top-p control sampling.
6. Instruction tuning and preference training turn a text predictor into an assistant.
7. A VLA reuses this machinery but is trained to output actions.
8. LLMs can be confidently wrong, have a finite context, and decode slowly.

Everything a language model writes is one more answer to the question: what token comes next?

Knowledge check

Try each question first, then reveal the explanation.

What’s next

You now understand the language side. Next is how a model learns a shared space where images and text can be compared.

Back to the VLA roadmap prerequisites