9.6 — Policy Execution and Low-Level Controller Interfaces

Learning objectives

After this lesson, you should be able to:

• Explain the outer (policy) loop and inner (controller) loop.
• Simulate a PD controller tracking a target.
• Compare hold and interpolate between policy updates.
• Compare position and effort commands.

The one-sentence idea

The policy sets destinations; a fast controller does the driving.

Position targets are the forgiving default.

Two loops at two speeds

A VLA outputs targets a few times per second (RT-1 ran at 3 Hz^[1]), but motors need updated commands far more often to move smoothly. So there are two loops: a slow outer loop where the policy decides, and a fast inner loop where a low-level controller drives the joints toward the latest target. The inner loop is classical control, the subject of the feedback prerequisites, and it does not change when you swap the policy. In ROS 2 this inner part is the domain of ros2_control, whose controllers are managed in three states (unconfigured, inactive, active) and talk to hardware through command and state interfaces.^[2]

The inner loop: a controller chasing a target

The simplest joint controller is a PD (proportional-derivative) law: push toward the target in proportion to the error and damp with the velocity. A simulated 1-DoF joint with unit mass:

import numpy as np

def simulate(target_of_time, kp=100.0, kd=20.0, dt=0.01, seconds=2.0):
    pos, vel, trace = 0.0, 0.0, []
    for k in range(int(seconds / dt)):
        target = target_of_time(k * dt)
        force = kp * (target - pos) - kd * vel
        vel += force * dt
        pos += vel * dt
        trace.append(pos)
    return np.array(trace)

trace = simulate(lambda t: 1.0)                               # a constant target of 1.0
print("position after 0.5 s:", round(float(trace[49]), 3))
print("position after 2 s:  ", round(float(trace[-1]), 3))
print("largest overshoot past the target:", round(float(max(0.0, trace.max() - 1.0)), 4))

position after 0.5 s: 0.956
position after 2 s:   1.0
largest overshoot past the target: 0.0

Toy model. A unit-mass joint with chosen gains. Real joints have friction, limits and their own tuned controllers, often built into the robot.

With a constant target the joint settles on it. The interesting case is when the target changes in jumps, which is exactly what a 10 Hz policy produces.

Hold, or interpolate, between policy updates?

Between two policy outputs the inner loop can either hold the last target (a staircase) or move smoothly between the old and new targets (interpolation). Same policy, same controller; only the target signal differs:

import numpy as np

policy_targets = [0.0, 0.3, 0.9, 1.0, 1.0]                   # one target per 0.1 s from the policy
policy_dt = 0.1

def staircase(t):
    return policy_targets[min(int(t / policy_dt), len(policy_targets) - 1)]
def interpolated(t):
    k = min(int(t / policy_dt), len(policy_targets) - 2)
    frac = (t - k * policy_dt) / policy_dt
    return policy_targets[k] + frac * (policy_targets[k + 1] - policy_targets[k])

def simulate(target_of_time, kp=400.0, kd=40.0, dt=0.005, seconds=0.5):
    pos, vel, accel = 0.0, 0.0, []
    for k in range(int(seconds / dt)):
        force = kp * (target_of_time(k * dt) - pos) - kd * vel
        vel += force * dt; pos += vel * dt
        accel.append(force)
    return np.array(accel)

for name, fn in (("hold last target (staircase)", staircase), ("interpolate between targets", interpolated)):
    a = simulate(fn)
    print(f"{name:<30} peak force {np.abs(a).max():7.1f} | largest change of force between steps {np.abs(np.diff(a)).max():6.1f}")

hold last target (staircase)   peak force   223.4 | largest change of force between steps  240.2
interpolate between targets    peak force    32.1 | largest change of force between steps   11.1

Interpolation lowers both the peak force and the abrupt force changes: gentler on the hardware and the load. It also delays the arrival at each target by up to one policy period, because the controller is always chasing a point that has not yet reached the new target. Smoothing trades a little responsiveness for a lot of gentleness; lesson 4.4 (chunk blending) is the same trade-off at the policy level.

What kind of command: position, velocity or effort?

Controllers can accept different commands. The Joint Trajectory Controller in ros2_control supports hardware command interfaces such as position, velocity, acceleration and effort in several combinations, needs at least position feedback from the hardware, and can be driven through a follow_joint_trajectory action interface when execution monitoring is desired.^[2] For policies the common choice is position targets (or small position deltas), because they are the most forgiving: a target is a place to go, not a force to apply (lesson 4.1). Effort commands give more control for contact-rich tasks, but a wrong effort command is much more dangerous than a wrong position target:

import numpy as np

def simulate(command_type, wrong_value, dt=0.01, seconds=1.0):
    pos, vel = 0.0, 0.0
    for _ in range(int(seconds / dt)):
        if command_type == "position target":
            force = 100.0 * (wrong_value - pos) - 20.0 * vel          # the controller drives toward the (wrong) target
        else:
            force = wrong_value                                      # the command is the force itself
        vel += force * dt; pos += vel * dt
    return pos

for seconds in (1.0, 5.0):
    print(f"after {seconds:.0f} s: position target 0.5 -> joint at {simulate('position target', 0.5, seconds=seconds):.2f} | force 0.5 -> joint at {simulate('effort', 0.5, seconds=seconds):.2f}")

after 1 s: position target 0.5 -> joint at 0.50 | force 0.5 -> joint at 0.25
after 5 s: position target 0.5 -> joint at 0.50 | force 0.5 -> joint at 6.26

The position target settles at a bounded place; the force keeps accelerating, and its position grows with the square of time until something stops it. Position interfaces with limits are the safer default for a learned policy.

Common misconceptions

Engineering takeaways

1. Slow outer loop, fast inner loop.
2. PD tracks targets.
3. Interpolate between updates.
4. Prefer position targets.

The policy picks the place; the controller makes the move.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

Last in this phase: safety monitors and fallback behaviour.

Back to the VLA roadmap