M5.1 — Robot State, Observations and Actions

Learning objectives

After this lesson, you should be able to:

• Distinguish state, observation and action.
• Roll a simple state forward under an action.
• Explain proprioception and exteroception.
• Explain why sensor noise gets amplified when computing quantities such as velocity.
• Name common action types and explain normalised, clipped actions.
• Read the array shapes of a robot demonstration.

The one-sentence idea

A policy sees noisy, partial observations of a hidden state and outputs actions, and each action changes the state it will next observe.

The robot never gets the state, only observations of it.

Three words that structure all of robot learning

Every robot-learning system is described with the same three ideas, and the VLA roadmap uses them constantly:

State

The full description of the robot and its world that is needed to predict what will happen next: joint positions and velocities, where the objects are, whether the gripper holds something. The truth about the situation.

Observation

What the robot’s sensors actually report: camera images, joint encoder readings, force measurements. A noisy, partial view of the state.

Action

What the robot is told to do next: target joint angles, a gripper command, a motion of the hand. The thing a policy outputs.

A policy is the function that turns observations into actions. Training a policy is the subject of Module 6, and a VLA is a policy whose observations include an image and an instruction. The Physical AI theory lesson on state, observation and belief covers the distinction between state and observation in more depth. This lesson fixes the vocabulary and shows what these things look like as arrays.

State, and how it changes

For a robot arm, the core of the state is the joint positions q (angles for revolute joints) and the joint velocities q̇. Both are needed: position alone does not say which way the arm is moving. The state is written as a vector, and actions move it forward in time. The simplest model of one joint, driven by an acceleration command a over a small time step dt, is:

import numpy as np

dt = 0.1
state = np.array([0.0, 0.0])        # [position q, velocity q_dot]
action = 1.0                        # a constant acceleration command

for step in range(3):
    q, dq = state
    state = np.array([q + dq * dt, dq + action * dt])
    print(step + 1, state.round(3))

1 [0.  0.1]
2 [0.01 0.2 ]
3 [0.03 0.3 ]

Each step takes the current state and an action and produces the next state. That relationship, s' = f(s, a), is what “dynamics” means. If the state really contains everything needed to predict the future, so that the past adds nothing, it is said to have the Markov property, and many learning methods assume it. A real arm’s state also includes joint torques and temperatures, and the world’s state includes every object, which is why the true state is rarely known exactly.

Observations: noisy and partial

The robot never sees the state directly. A joint encoder gives the angle with a little noise. There may be no velocity sensor at all. A camera sees objects from one viewpoint, and cannot see behind them. A robot’s observation is usually a dictionary of several readings:

rng = np.random.default_rng(0)
obs = {
    "image": rng.integers(0, 256, size=(3, 224, 224), dtype=np.uint8),   # a camera frame (stand-in)
    "joint_pos": rng.normal(size=7),                                      # 7-joint arm angles
    "gripper": np.array([0.04]),                                          # gripper opening in metres
}
print({k: v.shape for k, v in obs.items()})

{'image': (3, 224, 224), 'joint_pos': (7,), 'gripper': (1,)}

The joint readings and gripper opening, the robot’s sense of itself, are called proprioception. The image is exteroception, a view of the outside world. Real datasets use the same structure with real values. The camera image is the large part, which is why the cost of image tokens (M4.3) dominates such models.

Noise matters because it is amplified by anything computed from the readings. Velocity is not measured here, so it has to be estimated, for example from the change in position between steps. That estimate divides by a small dt, which magnifies the noise:

dt, true_velocity = 0.1, 0.3
true_pos = true_velocity * dt * np.arange(200)              # a joint moving at 0.3 rad/s
noisy_pos = true_pos + rng.normal(0, 0.01, size=200)        # encoder noise of 0.01 rad
estimate = np.diff(noisy_pos) / dt                          # finite-difference velocity estimate
print(round(estimate.mean(), 2), round(estimate.std(), 2))

0.3 0.14

On average the estimate is right (0.30), but individual values scatter by about 0.14, which is nearly half the size of the quantity being measured. The noise of 0.01 rad in position became a noise of roughly 0.14 rad/s in velocity, since dividing the difference of two noisy numbers by 0.1 multiplies the noise by about 14. This is one reason state estimation and filtering exist, and why a policy given only a position reading may struggle to know how fast it is moving.

Actions: what a policy outputs

An action is a command the robot’s lower-level controllers can carry out. Common forms for an arm are:

• Joint position targets: “go to these angles”.
• Joint velocities or torques: lower-level, with more direct control and more difficulty.
• End-effector commands: “move the hand here”, in space rather than in joints, covered in M5.5.
• Gripper commands: open and close, a continuous value or a switch.
• Deltas: a small change from the current position, rather than an absolute target.

Policies usually output numbers in a fixed range such as −1 to 1, and a fixed mapping converts them to physical commands within the robot’s limits. This keeps the network’s outputs well-scaled (M2.2’s normalisation idea) and prevents it commanding impossible motions:

vel_limit = np.array([1.0, 1.0, 1.5, 2.0])          # maximum speed of each joint in rad/s
raw = np.array([0.5, -2.0, 0.0, 1.0])               # what a network might output
action = np.clip(raw, -1.0, 1.0)                    # keep to the allowed range
command = action * vel_limit                        # scale to physical units
print(action)
print(command)

[ 0.5 -1.   0.   1. ]
[ 0.5 -1.   0.   2. ]

The second value, −2.0, was out of range and was clipped to −1.0 before being scaled. A control frequency goes with every action: a policy might produce a new action 10 times a second, while the controllers below it run at hundreds or thousands of times a second. Phase 9 of the roadmap returns to this boundary when it covers deployment. Because a VLA predicts an action from the current observation, and its observation is large, the rate at which it can produce actions is limited by how quickly it can run.

How this looks in a dataset

A robot demonstration (Module 6) is a sequence of time steps, and each step stores an observation and the action taken. Stacked over time, the data is arrays with a leading time axis:

T_ = 50                                             # 50 time steps in one demonstration
episode = {
    "joint_pos": rng.normal(size=(T_, 7)),          # observation: proprioception at every step
    "action":    rng.normal(size=(T_, 8)),          # 7 joint targets and 1 gripper command
}
print({k: v.shape for k, v in episode.items()})
history = episode["joint_pos"][-3:]                 # the last three observations, sometimes given to a policy
print(history.shape)

{'joint_pos': (50, 7), 'action': (50, 8)}
(3, 7)

A policy may see only the latest observation, or a short history of them, which helps with velocity and with partial observability. The values above are random stand-ins. In a real dataset they would be recorded from a robot, and the action at each step is what the human or controller did at that moment. Learning to predict that action from the observation is behaviour cloning, the topic of M6.2.

Common misconceptions

Engineering takeaways

1. State is the true situation, an observation is the sensor view of it, and an action is the command.
2. Dynamics map a state and an action to the next state.
3. A state is Markov if it holds everything needed to predict the future.
4. Proprioception is the robot’s sense of itself, and images are exteroception.
5. Computing velocity from position readings amplifies noise.
6. Actions are often normalised to −1…1, clipped, then scaled to physical limits.
7. A policy typically runs much slower than the controllers beneath it.
8. Demonstrations store observations and actions as arrays with a time axis.

Every robot-learning method is a way of choosing actions from observations of a state you cannot fully see.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

You now have the vocabulary of state, observation and action. Next is how a robot’s configuration can be described in joint space and in Cartesian space.

Back to the VLA roadmap prerequisites