4.8 — FAST Action Tokenization

Learning objectives

After this lesson, you should be able to:

• Explain why per-step binning is wasteful for smooth, high-frequency actions.
• Explain what a DCT does to a smooth motion.
• Explain the scale-and-round trade-off.
• Describe byte-pair encoding and the decoding limitation.

The one-sentence idea

Describe a motion by its slow and fast components, keep the important ones, and compress.

Smooth motions need only a few numbers.

The problem with one token per number

Lesson 4.2 showed that binning every number of every time step gives long token sequences: 700 tokens for 50 steps of 14 numbers. The FAST paper states the deeper problem: per-dimension, per-timestep binning schemes typically perform poorly when learning dexterous skills from high-frequency robot data.^[1] Neighbouring time steps of a smooth motion are nearly the same, so the tokens are highly redundant and consecutive tokens are very predictable, which makes learning from them harder.

FAST (Frequency-space Action Sequence Tokenization) borrows an idea from image and audio compression: describe the motion not by its values at each time but by how much slow and fast wiggle it contains.^[1] Combined with π0 it can scale to training on 10,000 hours of robot data and match the performance of diffusion VLAs while reducing training time by up to 5×.^[1]

The recipe

As described for the paper, the steps are: normalise the actions, apply a discrete cosine transform (DCT) along time to each action dimension, scale and round the coefficients to integers (high-frequency ones often become zero), flatten them from low to high frequency, and compress the resulting integer sequence with byte-pair encoding.^[1] Let us walk through the middle steps on a smooth chunk of motion.

import numpy as np

n = 32                                                     # time steps in the chunk
t = np.arange(n)
motion = 0.5 * np.sin(2 * np.pi * t / 64) + 0.2 * np.sin(2 * np.pi * t / 32)      # a smooth movement

# DCT-II as a matrix: row k asks "how much of cosine number k is in the signal?"
k = np.arange(n)[:, None]
dct = np.sqrt(2 / n) * np.cos(np.pi * (t + 0.5) * k / n)
dct[0] /= np.sqrt(2)
coeffs = dct @ motion

print("largest coefficients by size:", np.argsort(-np.abs(coeffs))[:5].tolist(), "(index 0 = slowest)")
print("energy in the first 6 of 32 coefficients:", round(float(np.sum(coeffs[:6] ** 2) / np.sum(coeffs ** 2)), 4))
print("transform keeps all information:", bool(np.allclose(dct.T @ coeffs, motion)))

largest coefficients by size: [0, 2, 1, 3, 4] (index 0 = slowest)
energy in the first 6 of 32 coefficients: 0.9976
transform keeps all information: True

Toy model. A hand-built DCT and a made-up motion. FAST applies the transform to each action dimension of real, normalised chunks.

Almost all the energy sits in the first few coefficients, which is why the rest can be thrown away at a small cost.

Rounding and counting

Multiplying the coefficients by a scale and rounding them to integers makes most of the high-frequency ones exactly zero. The scale is a trade-off: a bigger scale keeps more detail and gives more non-zero numbers. Compare with one token per time step:

import numpy as np

n = 32
t = np.arange(n)
motion = 0.5 * np.sin(2 * np.pi * t / 64) + 0.2 * np.sin(2 * np.pi * t / 32)
k = np.arange(n)[:, None]
dct = np.sqrt(2 / n) * np.cos(np.pi * (t + 0.5) * k / n); dct[0] /= np.sqrt(2)
coeffs = dct @ motion

for scale in (5, 20, 100):
    q = np.round(coeffs * scale).astype(int)
    rebuilt = dct.T @ (q / scale)
    nonzero = int(np.count_nonzero(q))
    last = int(np.max(np.nonzero(q)[0])) + 1 if nonzero else 0
    print(f"scale {scale:>3}: {nonzero:>2} non-zero of 32, keep the first {last:>2}, largest reconstruction error {np.abs(rebuilt - motion).max():.4f}")
print("per-step binning would need 32 tokens for this one dimension")

scale   5:  5 non-zero of 32, keep the first  5, largest reconstruction error 0.0713
scale  20: 10 non-zero of 32, keep the first 10, largest reconstruction error 0.0251
scale 100: 19 non-zero of 32, keep the first 19, largest reconstruction error 0.0077
per-step binning would need 32 tokens for this one dimension

A small scale gives few numbers and a rough reconstruction; a large scale gives precision with more numbers. After flattening, trailing zeros are cheap and byte-pair encoding merges frequent patterns into single tokens.

Byte-pair encoding: merging common pairs

Byte-pair encoding (BPE) is the same trick language models use for words: repeatedly find the most frequent adjacent pair of symbols and replace it with a new symbol. Here is the mechanism on a short integer sequence of the kind quantised coefficients produce (lots of zeros):

from collections import Counter

seq = [3, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0]
next_symbol = 100                                          # new symbols get ids from 100 up

for step in range(3):
    pairs = Counter(zip(seq, seq[1:]))
    (a, b), count = pairs.most_common(1)[0]                # the most frequent neighbouring pair
    merged, i = [], 0
    while i < len(seq):
        if i + 1 < len(seq) and seq[i] == a and seq[i + 1] == b:
            merged.append(next_symbol); i += 2
        else:
            merged.append(seq[i]); i += 1
    print(f"merge ({a}, {b}) seen {count}x -> symbol {next_symbol}; length {len(seq)} -> {len(merged)}")
    seq, next_symbol = merged, next_symbol + 1
print("final sequence:", seq)

merge (0, 0) seen 14x -> symbol 100; length 20 -> 12
merge (100, 100) seen 5x -> symbol 101; length 12 -> 9
merge (101, 100) seen 2x -> symbol 102; length 9 -> 7
final sequence: [3, 101, 1, 102, 2, 102, 0]

Toy model. A tiny BPE written for illustration; FAST trains its vocabulary on real quantised action data.

The sequence shrinks from 20 symbols to 7 in three merges because zeros repeat. The same effect makes real action chunks compact.

A limitation to know

Because BPE gives variable-length sequences, a token sequence generated by the policy is not guaranteed to decode into a valid coefficient matrix of the right size.^[1] A deployed system therefore needs a check that decoding succeeded, and a fallback when it did not (lesson 9.7), just like the parse check for text actions in lesson 1.2.

Common misconceptions

Engineering takeaways

1. DCT turns a smooth motion into a few large coefficients.
2. Scale controls the precision.
3. BPE merges frequent patterns.
4. Check that decoding succeeded.

Compress the motion, not the numbers.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

That completes action representation. Next, the research that established VLAs, starting with RT-1.

Back to the VLA roadmap