M6.5 — Policy, Observation, State and Action

Learning objectives

After this lesson, you should be able to:

• Define policy, observation, state, action, episode, horizon and trajectory.
• Distinguish deterministic and stochastic policies.
• Write the reset-and-step rollout loop.
• Explain a goal- or language-conditioned policy and place a VLA in this picture.
• Explain why partial observability can make history or memory necessary.
• Compute how action chunking changes how often a policy is called.

The one-sentence idea

A policy maps what the robot observes, and what it has been asked to do, to an action, and it only has meaning inside a loop with the world.

Policy: observation and goal in, action out.

What a policy is

A policy is a rule for choosing actions. It is the thing every method in this module trains: behavior cloning (M6.2) fits one to demonstrations, and reinforcement learning (M6.6) improves one from reward. Written as a function it takes what the robot knows and returns an action: a = π(o). The same word covers a hand-written controller, a small neural network, and a billion-parameter VLA. M5.1 introduced the surrounding vocabulary, and this lesson puts the pieces together.

Observation o

What the policy is given at this moment: camera images, joint readings, and for a VLA the language instruction.

State s

The full underlying situation, which the policy usually cannot see. The robot’s true pose, the exact position of every object.

Action a

What the policy outputs: a joint or end-effector command, and a gripper value.

Episode and horizon

One attempt at the task, from the start until it ends, with a maximum length called the horizon.

Trajectory τ

The sequence that results: o₀, a₀, o₁, a₁, …

A policy can be deterministic (one action for each observation) or stochastic, giving a probability distribution π(a | o) from which an action is sampled. Stochastic policies matter because they can represent the several valid choices of M6.2 and because exploration in reinforcement learning needs them.

import numpy as np

rng = np.random.default_rng(0)

def deterministic(obs):
    return -0.5 * obs                             # always the same action for the same observation

def stochastic(obs, sigma=0.2):
    return -0.5 * obs + rng.normal(0, sigma)      # a Gaussian around the same action

print(deterministic(0.8), deterministic(0.8))
print([round(stochastic(0.8), 2) for _ in range(4)])

-0.4 -0.4
[-0.37, -0.43, -0.27, -0.38]

The deterministic policy repeats itself exactly. The stochastic one centres on the same action, −0.4, but varies from call to call, with the spread set by sigma. At deployment a stochastic policy is often run by taking its mean, or with a small amount of sampling, depending on the method.

The interaction loop

A policy only means something inside a loop with an environment, real or simulated. The standard interface has two calls: reset() starts an episode and returns the first observation, and step(action) applies an action and returns the next observation and whether the episode is over. Almost all robot-learning software uses some version of this:

class Reach1D:
    """The robot's hand must get to the goal position. It sees only its position, plus a small sensor noise."""
    def __init__(self, goal=1.0, seed=0, horizon=50):
        self.goal, self.horizon = goal, horizon
        self.rng = np.random.default_rng(seed)

    def reset(self):
        self.pos, self.t = 0.0, 0
        return self._obs()

    def _obs(self):
        return self.pos + self.rng.normal(0, 0.01)

    def step(self, action):
        self.pos += float(np.clip(action, -0.2, 0.2))          # the motion is limited per step
        self.t += 1
        success = abs(self.pos - self.goal) < 0.05
        return self._obs(), success or self.t >= self.horizon

def rollout(env, policy):
    obs, done, trajectory = env.reset(), False, []
    while not done:
        action = policy(obs)
        trajectory.append((obs, action))
        obs, done = env.step(action)
    return trajectory, abs(env.pos - env.goal) < 0.05

env = Reach1D()
traj, success = rollout(env, lambda obs: 0.5 * (env.goal - obs))     # a simple policy that steps toward the goal
print(len(traj), success)

7 True

The policy steps toward the goal using only its (noisy) position reading, and succeeds after 7 steps. The list traj is the trajectory of the episode. Every dataset in M6.3 is a collection of such trajectories recorded from an expert policy, and every policy evaluation in M6.4 is a set of rollouts like this one, with the success rate counted.

Policies that depend on a goal or instruction

A policy that always does the same thing is of little use. Robots are asked to do different tasks, so the policy is told what to do and the action depends on both the observation and a goal g: a = π(o, g). For a language-conditioned policy the goal is an instruction such as “put the red cup in the bin”, embedded as in M3.2. A VLA is exactly this: a policy whose observation includes camera images and whose goal is language, a = π(image, state, instruction).

policy_goal = lambda obs, goal: 0.5 * (goal - obs)
print(policy_goal(0.2, 1.0), policy_goal(0.2, -1.0))

0.4 -0.6

With the same observation, 0.2, the policy moves forward (+0.4) for the goal 1.0 and backward (−0.6) for the goal −1.0. In a VLA, changing the instruction changes the behaviour in the same way, just in a far richer space of tasks. This is also why a VLA’s evaluation has to include instruction variation as well as object and position variation (Phase 8).

Observations are not states: why memory can matter

If the observation contained the full state, the policy could decide from the present alone. A real observation hides things, such as velocity (M5.1), objects behind others, or what was done earlier in the task. The decision process is then partially observable, and a policy that sees only the latest observation may be unable to act well. Giving it a history of recent observations, or an internal memory, recovers some of what is hidden. M5.6 gave the clearest case: a position-only controller on a mass overshoots, while one that also knows the velocity settles:

def run(use_history, Kp=16.0, Kd=8.0, dt=0.01, steps=500, target=1.0):
    q, v, prev, peak = 0.0, 0.0, 0.0, 0.0
    for _ in range(steps):
        velocity_estimate = (q - prev) / dt if use_history else 0.0       # from two successive observations
        prev = q
        u = Kp * (target - q) - Kd * velocity_estimate
        v += u * dt
        q += v * dt
        peak = max(peak, q)
    return round(max(0.0, peak - target), 3)

print(run(use_history=False), run(use_history=True))

1.0 0.0

Using only the current position the mass overshoots by 1.0. With the previous observation also available, the policy can estimate velocity and damp the motion, overshooting by 0. This is why many robot policies take a few recent frames or joint readings, or a recurrent or transformer memory, and not just the latest frame. The more is hidden in the task, such as which of several identical-looking drawers has already been checked, the more memory the policy needs.

Acting on a chunk of actions

A policy need not output a single action per observation. Some predict a short chunk of future actions (M6.3, and Phase 4) and the robot executes some or all of them before asking the policy again. Executing several actions from each prediction means the policy runs less often, which suits slow models, at the price of reacting later to what happens. This pattern of planning a stretch ahead and re-planning is called receding-horizon control:

def policy_calls(total_steps, chunk, executed):
    calls, t = 0, 0
    while t < total_steps:
        calls += 1                       # predict a chunk of actions
        t += min(executed, chunk)        # run the first few before predicting again
    return calls

print(policy_calls(100, chunk=1, executed=1))
print(policy_calls(100, chunk=8, executed=4))
print(policy_calls(100, chunk=8, executed=8))

100
25
13

For a 100-step task, a one-action policy runs 100 times. A policy predicting chunks of 8 and executing 4 runs 25 times, and one that executes all 8 runs 13 times. Executing more of each chunk saves computation but means 8 steps pass between looks at the world. How best to choose these numbers is a design question that later phases return to.

Common misconceptions

Engineering takeaways

1. A policy maps observations (and goals) to actions.
2. Policies can be deterministic or stochastic.
3. An episode is one attempt, the horizon is its maximum length, and a trajectory is the sequence of observations and actions.
4. The rollout loop is reset() then repeated step(action).
5. A language-conditioned policy takes an instruction as its goal, and a VLA is one.
6. Observations hide the state, so a history or memory can be needed.
7. Action chunks reduce how often the policy must run.
8. Imitation learning and reinforcement learning both train policies, in different ways.

Everything in this roadmap is a way of building a better function from what the robot sees and is told to what it does.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

You have the vocabulary of policies. The last lesson of this module covers the other main way to train one: reinforcement learning from reward.

Back to the VLA roadmap prerequisites