4.6 — Diffusion Policy

Learning objectives

After this lesson, you should be able to:

• Describe the forward (noising) and reverse (denoising) directions.
• Explain why denoising can produce one valid option out of several.
• Compute the latency of iterative denoising.
• Explain why Diffusion Policy executes chunks.

The one-sentence idea

Learn to clean noise into good actions, then clean random noise to act.

Sampling gives a real option; the cost is many network passes.

Actions from noise

Diffusion models, made famous by image generators, create things by starting from pure noise and removing the noise a little at a time. Denoising diffusion probabilistic models (DDPM) are latent variable models whose best results come from training on a weighted variational bound linked to denoising score matching with Langevin dynamics.^[1] Diffusion Policy applies the idea to robot control. It represents a visuomotor policy as a conditional denoising diffusion process: the policy learns the gradient of the action-distribution score function and refines actions at inference through stochastic Langevin dynamics steps, conditioned on what the robot sees.^[2] The authors report beating existing state-of-the-art methods by an average of 46.9% across their benchmarks and credit the formulation with handling multimodal action distributions, scaling to high-dimensional action spaces and training stably.^[2]

In plain words: instead of predicting the action, the model learns what a good action looks like and “cleans up” a random guess until it looks like one.

The easy direction: adding noise

Training needs a recipe that turns a clean action into a noisy one. You choose a schedule that says how much of the original survives after t noising steps. At step t the noisy action is a mix of the clean action and fresh random noise. After enough steps nothing of the original remains:

import numpy as np
rng = np.random.default_rng(0)

T = 50
betas = np.linspace(1e-3, 0.2, T)                      # how much noise is added at each step
alpha_bar = np.cumprod(1 - betas)                       # how much of the original survives after t steps

clean = 1.0                                             # a clean action
for t in (0, 10, 25, 49):
    noisy = np.sqrt(alpha_bar[t]) * clean + np.sqrt(1 - alpha_bar[t]) * rng.normal()
    print(f"after {t+1:>2} steps: {100*alpha_bar[t]:5.1f}% of the signal survives, noisy action = {noisy:6.2f}")

after  1 steps:  99.9% of the signal survives, noisy action =   1.00
after 11 steps:  78.8% of the signal survives, noisy action =   0.83
after 26 steps:  24.8% of the signal survives, noisy action =   1.05
after 50 steps:   0.5% of the signal survives, noisy action =   0.17

Toy model. A made-up noise schedule on a single number. Real policies use schedules tuned for their data and noise whole action sequences.

The training task for the network is: given a noisy action, the step number and the camera view, predict the noise (or equivalently the clean action). That is ordinary supervised learning; the generation comes from running the learned cleaner repeatedly.

The generating direction: removing noise

To act, start from random noise and apply the cleaner step by step. To see how this handles two valid options, take the “left or right” case from lesson 3.4: the demonstrations contain only −1 and +1. For this simple distribution the ideal cleaner can be written down exactly (a trained network learns to approximate such a function). We then run the usual denoising loop many times:

import numpy as np
rng = np.random.default_rng(0)

T = 50
betas = np.linspace(1e-3, 0.2, T)
alphas = 1 - betas
alpha_bar = np.cumprod(alphas)

def best_guess_of_clean(x, t):                          # the ideal cleaner for data that is -1 or +1 with equal chance
    return np.tanh(np.sqrt(alpha_bar[t]) * x / (1 - alpha_bar[t]))

def sample():
    x = rng.normal()                                    # start from pure noise
    for t in reversed(range(T)):
        x0 = best_guess_of_clean(x, t)
        eps = (x - np.sqrt(alpha_bar[t]) * x0) / np.sqrt(1 - alpha_bar[t])
        x = (x - betas[t] / np.sqrt(1 - alpha_bar[t]) * eps) / np.sqrt(alphas[t])
        if t > 0:
            x += np.sqrt(betas[t]) * rng.normal()       # a little fresh randomness except at the last step
    return x

samples = np.array([sample() for _ in range(2000)])
print("share near -1:", round(float(np.mean(samples < -0.5)), 2))
print("share near +1:", round(float(np.mean(samples > 0.5)), 2))
print("share stuck in the middle (the 'average'):", round(float(np.mean(np.abs(samples) <= 0.5)), 3))

share near -1: 0.51
share near +1: 0.49
share stuck in the middle (the 'average'): 0.0

Toy model. A one-number action and an analytic cleaner standing in for a trained network. It illustrates how denoising picks one valid option each time instead of the average, which squared-error regression would output.

Each run ends at one of the real options, and both options appear about equally often. A regression would always answer 0. That is the practical meaning of “handles multimodal action distributions”.

The price: many network passes per action

Every denoising step is a forward pass through the network. If a policy needs 50 steps for each decision and a pass takes 10 ms, one decision takes half a second:

pass_ms = 10                                            # time of one network pass (an assumed number)
for steps in (1, 10, 50, 100):
    total = steps * pass_ms
    print(f"{steps:>3} denoising steps -> {total:>4} ms per decision -> at most {1000/total:5.1f} decisions per second")

  1 denoising steps ->   10 ms per decision -> at most 100.0 decisions per second
 10 denoising steps ->  100 ms per decision -> at most  10.0 decisions per second
 50 denoising steps ->  500 ms per decision -> at most   2.0 decisions per second
100 denoising steps -> 1000 ms per decision -> at most   1.0 decisions per second

Toy model. The 10 ms per pass is an assumption, not a measurement.

That is one reason Diffusion Policy predicts a chunk of actions and executes part of it before querying again (receding-horizon control):^[2] the cost of denoising is shared across many control steps. Reducing the number of steps is an active topic, and the next lesson, flow matching, is one related approach.

Common misconceptions

Engineering takeaways

1. Training: predict the noise in noisy actions.
2. Acting: start from noise and denoise, conditioned on observations.
3. Can represent several valid options.
4. Costs many passes per decision.

Diffusion Policy draws an action instead of calculating one.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

Next: flow matching, a related generative approach that learns a smooth path from noise to actions.

Back to the VLA roadmap