4.5 — Action Chunking with Transformers (ACT)

Learning objectives

After this lesson, you should be able to:

• State ACT’s inputs and outputs.
• Explain why predicting chunks helps.
• Explain the role of the style variable z in a CVAE.
• Describe what is different at training and run time.

The one-sentence idea

ACT predicts a chunk of joint targets, with a learned style variable to absorb differences between demonstrations.

Chunks shorten the horizon; z absorbs style.

What ACT is

ACT (Action Chunking with Transformers) comes from a paper on teaching a low-cost two-arm robot fine-grained tasks. The system learns end-to-end from real demonstrations collected through a custom teleoperation interface, and ACT is the learning method, which shortens the effective horizon by predicting actions in chunks.^[1] The headline result: difficult tasks such as opening a translucent condiment cup or slotting a battery with 80–90% success from only 10 minutes of demonstration data.^[1] ACT is not a VLA (it does not take language) but it introduced ideas that VLAs reuse, mainly action chunking (lesson 4.4).

The inputs and outputs, as the paper describes them: observations are the follower arms’ current joint positions plus camera images; actions are the joint positions of the human operator’s leader arms, 14 numbers for two arms with six joints and a gripper each.^[1] The policy is trained as a conditional variational autoencoder, with a transformer encoder-decoder that maps images and joint positions to a chunk of future actions.^[1]

The shapes involved

Chunk length is a hyperparameter; secondary descriptions of the original setup mention 100 steps but the exact value for a given task should be read from the paper or code. Here is how the pieces fit, with a chunk length we choose:

import numpy as np

n_cameras, height, width = 4, 480, 640
joints = 14
chunk = 100                                              # our choice for illustration

images = np.zeros((n_cameras, height, width, 3), dtype=np.uint8)
state = np.zeros(joints)                                 # follower arms' current joint positions
action_chunk = np.zeros((chunk, joints))                 # target joint positions for the next steps

print("images in:", images.shape)
print("joint state in:", state.shape)
print("action chunk out:", action_chunk.shape, "=", action_chunk.size, "numbers per decision")

images in: (4, 480, 640, 3)
joint state in: (14,)
action chunk out: (100, 14) = 1400 numbers per decision

Toy model. The camera resolution and chunk length are placeholders. The count of four cameras and the 14 joint values follow the paper.

One model call produces 1,400 numbers. Executing them over 100 control steps means the model runs once per 100 steps (or once per step with ensembling, lesson 4.4).

Why a variational autoencoder? The “style” variable

Different people demonstrate the same task differently: faster, slower, in a slightly different path. A policy trained to copy them with a plain loss learns the average style (recall lesson 3.4). The CVAE gives the model an extra hidden number, a “style variable” z. During training an encoder looks at the demonstrated action sequence and summarises its style in z; the decoder gets z together with the observation. Because the style is handed over, the decoder no longer has to guess it. At run time you do not have a demonstration to encode, so z is set to a neutral value (the mean of its prior); implementation notes say it is typically zero.

import numpy as np
rng = np.random.default_rng(0)

t = np.linspace(0, 1, 20)
speeds = rng.choice([0.5, 1.5], size=200)                           # half the demonstrators are slow, half are fast
demos = np.array([np.clip(s * t, 0, 1) for s in speeds])            # reach 1.0 either late or early

# a policy that does not know the style can only learn the average motion
average = demos.mean(axis=0)
loss_without = float(np.mean((demos - average) ** 2))

# a decoder that is told the style (here z = the speed itself) can follow each demonstration
loss_with = float(np.mean([(np.clip(s * t, 0, 1) - d) ** 2 for s, d in zip(speeds, demos)]))

print("error without a style variable:", round(loss_without, 4))
print("error with the style variable:  ", round(loss_with, 4))

error without a style variable: 0.0514
error with the style variable:   0.0

Toy model. The “style” is literally the speed here and is given directly. In ACT, z is learned and summarises whatever varies between demonstrations; this only shows why handing over the variation helps.

In training the model is helped by z; at test time it has to do without. That is why the CVAE keeps z close to a simple distribution (a penalty in the loss), so that using the neutral value still gives sensible motions.

The recipe in short

Collect teleoperated demonstrations with a leader-follower rig; record camera images and joint positions with the leader’s joint positions as the actions;^[1] train the CVAE-transformer to predict action chunks;^[1] at run time predict a chunk from the current images and joint positions, and blend overlapping chunks with temporal ensembling.^[1] The ideas of predicting a sequence and executing a part of it carry straight into the larger VLAs of phase 6.

Common misconceptions

Engineering takeaways

1. Inputs: images + joint positions. Output: a chunk of joint targets.
2. Actions are leader-arm joint positions.
3. CVAE with a style variable.
4. Chunk + ensembling reduces compounding error.

ACT: plan a chunk, let z carry the style.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

Next: Diffusion Policy, which generates actions by denoising.

Back to the VLA roadmap