LM1-T1 — What a Language Model Is

Learning objectives

After this lesson, you should be able to:

• Define a language model as a probability distribution over the next token given a context.
• Write the probability of a sentence with the chain rule and compute it from counts.
• Explain why counting works for short contexts and fails for long ones.
• Relate the counting model to a neural network that computes the same kind of distribution.

The idea: autocomplete with probabilities

Your phone suggests a next word as you type. Behind the suggestion is a ranking: some words are more likely than others after “see you”. A language model is that ranking made precise. For any piece of text (the context), it gives each possible next token a probability, and the probabilities add to one.

Tokens, not just words

Models do not read letters or whole words; they read tokens, pieces of text such as “robot”, “ing” or “,”. Module 2 builds a tokenizer. For now a token is a word, and the vocabulary is the list of all tokens the model knows. A model with a vocabulary of 50,000 tokens gives 50,000 probabilities at every step.

Why probabilities and not one answer?

Language is not deterministic. After “the robot picks up the” many continuations are reasonable. A model that returned only its single best guess would throw away information that is useful: how confident it is, which alternatives are close, and what happens if you choose differently. Keeping the whole distribution lets you decode in different ways (Lesson LM1-T2), measure how well the model fits text (Module 5), and tell when it is guessing.

The simplest model: count

Take a corpus. For a context, count how often each token followed it, and divide by the total. In the corpus of the figure, “the” occurs 20 times: 5 times followed by “robot” and 5 by “red”, so each has probability 5/20 = 25%. That table, built for every context, is a complete (if tiny) language model, and it is exactly what you compute in Lab LM1-L1.

Where it breaks, and what replaces it

Counting needs the context to have been seen. With one word of context there are only as many contexts as the vocabulary; with the last 100 tokens there are more possible contexts than atoms in the universe, and almost none appears in any corpus. A neural network fixes this by computing the distribution from the context with learned parameters, so that similar contexts give similar distributions even if this exact context was never seen. The network is the same kind of object as the table (context in, distribution out) but it generalises. Building one is the goal of this track.

The mathematics: the chain rule and conditional probabilities

A text is a sequence of tokens w₁, w₂, …, w_n. A language model assigns a probability to the whole sequence. The chain rule of probability writes it as a product of next-token probabilities:

w_t

the token at position t, an element of the vocabulary.

P(w_t | w₁, ..., w_t-1)

the conditional probability of token t given all the tokens before it (the context). This is what a language model computes, one position at a time.

n

the length of the text in tokens.

Every factor is a distribution over the vocabulary: for each context, the probabilities of all possible next tokens are non-negative and sum to one. An autoregressive model (Lesson LM1-T2) is one that is built exactly this way.

Estimating by counting. With a one-token context (a bigram model) the maximum-likelihood estimate is a ratio of counts:

count(a, b)

the number of times token b immediately followed token a in the corpus.

count(a, anything)

the number of times a occurred with any token after it; the denominator makes the probabilities sum to one.

Longer contexts use the same ratio with longer prefixes, but the number of distinct contexts grows as V^k for vocabulary size V and context length k. For V = 50,000 and k = 3 that is already 1.25 × 10¹⁴ possible contexts, far more than any text provides examples of.

A neural model. A neural network with parameters θ computes the same conditional distribution, written P_θ(w_t | w₁, …, w_t−1). Training (Module 5) adjusts θ so that the real next tokens in a large corpus get high probability.

A worked example: scoring a sentence

The corpus of Lab LM1-L1 has ten sentences, such as “the robot picks up the red box”. Counting gives these probabilities (each is a ratio of counts):

By the chain rule the probability of the sentence (given that it starts after the previous sentence ended) is the product:

About 2.3%. Probabilities of whole sentences are always small, so models work with logarithms: log₂(0.0225) = −5.47 bits, or about 0.78 bits per token over the 7 tokens. A sentence such as “the robot picks up the box” contains a step with probability zero, because “box” never directly follows “the” in this corpus (only a colour or a name does). The counting model calls it impossible, although it is perfectly good English: a symptom of counting’s inability to generalise. Lower scores mean the text is more surprising to the model; Lesson LM5-T1 turns this into the training loss.

The same computation in Python

from collections import Counter

tokens = "the robot picks up the red box <end> the robot picks up the green bin <end>".split()
pair = Counter(zip(tokens, tokens[1:]))      # count(a, b)
single = Counter(tokens[:-1])                # count(a, anything)

def p(next_tok, prev):
    return pair[(prev, next_tok)] / single[prev]

print(p("robot", "the"))   # 2 of 4 occurrences of "the" -> 0.5

zip(tokens, tokens[1:]) pairs every token with the one after it; Counter counts the pairs. The last token has no successor, so tokens[:-1] is used for the denominator. The probability is a pair count divided by the context count, exactly the formula above.

Engineering implications and limitations

Everything is a distribution over the vocabulary. The output of a real model is a vector with one number per token (the logits, converted to probabilities by a softmax, Lesson LM5-T5). Remember this shape; every later module works with it.

Counting models are still useful. They are fast baselines, they are easy to inspect, and they are good for sanity checks. If a neural model cannot beat a bigram model on your data, something is wrong with the pipeline.

A language model knows text, not truth. It is trained to match the statistics of its corpus. Probabilities say how likely a continuation is under that corpus, not whether a statement is correct; Module 9 returns to this.

The training data define the model. A model fitted to robotics manuals assigns different probabilities from one fitted to poetry. Data quality and coverage matter as much as architecture (Module 2 and Module 5).

Smoothing in counting models. A pure count model assigns probability zero to anything unseen, which makes a whole sentence impossible. Real count-based systems smooth the estimates; neural models avoid the problem by generalising.

Robot interfaces. A language model attached to a robot produces a distribution over text, not over actions. Converting that text into safe, validated robot commands is an engineering problem handled by structured outputs, validation and a safety layer; the Physical AI and VLA tracks cover robot-specific designs.

Common misconceptions

Engineering takeaways

1. A language model gives P(next token | context) for every token in the vocabulary; the probabilities sum to 1.
2. The chain rule writes the probability of a text as a product of these next-token probabilities.
3. Counting estimates P(b | a) = count(a, b) / count(a, anything), and fails for long contexts because the contexts are almost never repeated.
4. A neural network computes the same distribution with learned parameters and generalises across similar contexts.
5. Model probabilities describe the training text, not the truth of a statement.

References and further reading

The core learning path is Chapter 1 of the book; the textbook below is optional background on count-based models.

• Jurafsky, D. and Martin, J. H. Speech and Language Processing (3rd ed. draft), the chapter on n-gram language models: a standard textbook treatment of the counting model and smoothing.
• Build a Large Language Model (From Scratch), Chapter 1: Understanding Large Language Models (primary book reference)
• Companion site for the book (primary book reference)

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: how a distribution becomes text, one token at a time: autoregressive generation and decoding.

Related content

Nothing below is required to finish this lesson.

• Lab
  LM1-L1: Explore Next-Token Probabilities and Decoding

  Build a count-based next-token model and compare greedy decoding with sampling.