1.1 — What Is Physical AI?

Learning objectives

After this lesson, you should be able to:

• State what makes an AI system “physical” using the perceive–decide–act loop.
• Explain how noisy sensing and imperfect actuation make a fixed plan fail.
• Compute how far the world moves between two decisions of a slow model.
• Explain why robot data is slow and expensive to collect.
• Explain the sim-to-real gap and the idea behind domain randomization.
• Name the main open problems that foundation models for robotics face.

The one-sentence idea

A physical AI system closes its loop through the real world, where sensing and acting are uncertain, time keeps running, data is costly and mistakes have consequences.

The world does not follow the model’s plan, so the system has to cope.

A working definition

“Physical AI” is a recent label and different sources draw its boundary in different places, so this course states the meaning it uses instead of assuming one. The idea underneath is much older: an agent is anything that perceives its environment through sensors and acts on that environment through actuators.^[1] In this course, physical AI means an AI system whose perceive–decide–act loop runs through the physical world: its inputs are measurements from sensors such as cameras and joint encoders, its outputs are commands to motors, and what it does changes the world it will measure next.

A robot arm, a self-driving car and a drone are examples. A chatbot is not, because its input and output are text and its mistakes do not move anything. Recent surveys describe how large pretrained “foundation” models are being used to improve the perception, decision-making and control parts of robots,^[2] and the models you will study in this roadmap, vision-language-action (VLA) models, are one result of that work.

The loop, in code

Here is the whole idea as a few lines of Python: an environment with a hidden true state, a sensor that reports it imperfectly, a decision rule, and an actuator that carries out the command imperfectly. The task is to move a hand along one axis to a goal at 1.0.

import numpy as np

rng = np.random.default_rng(0)

class World:
    """The hidden truth: where the hand really is."""
    def __init__(self):
        self.pos = 0.0

def sense(world, noise):
    return world.pos + rng.normal(0, noise)          # a sensor reports the position imperfectly

def act(world, command, noise):
    world.pos += command + rng.normal(0, noise)       # an actuator moves the hand imperfectly

world = World()
for step in range(5):
    seen = sense(world, noise=0.0)                    # perceive (a perfect sensor, for now)
    command = 0.5 * (1.0 - seen)                      # decide: step halfway to the goal
    act(world, command, noise=0.0)                    # act (a perfect actuator, for now)
    print(step, round(world.pos, 3))

0 0.5
1 0.75
2 0.875
3 0.938
4 0.969

Toy model. The world, sensor and decision rule here are made up so the loop is easy to see. They are not any real robot.

With perfect sensing and perfect actuation the hand closes in on the goal, halving the remaining distance each step. This is the situation a purely software system is in: what it computes is what happens. The rest of the lesson is about what changes when the sensor and the actuator are not perfect.

Why it is harder (1): sensing and acting are uncertain

A physical system never sees the true state, only measurements of it, and it never executes a command exactly. Probabilistic robotics treats both as uncertain: sensor measurements are noisy, and robot motion is uncertain, so the robot has to estimate its state from imperfect evidence.^[3] The formal setting for choosing actions when the state is only partly observable is the partially observable Markov decision process (POMDP).^[4]

The effect is easy to measure. Below, the same plan, take twenty equal steps to the goal, is run many times, once with a perfect actuator and then with actuator noise. “Success” means ending within 0.05 of the goal:

def run_plan(actuator_noise, steps=20, goal=1.0, tol=0.05):
    pos = 0.0
    for _ in range(steps):
        pos += goal / steps + rng.normal(0, actuator_noise)     # the plan: equal steps, with no looking
    return abs(pos - goal) < tol

for noise in (0.0, 0.01, 0.03, 0.05):
    rate = np.mean([run_plan(noise) for _ in range(2000)])
    print(noise, round(float(rate), 2))

0.0 1.0
0.01 0.73
0.03 0.3
0.05 0.17

Toy model. This is a made-up one-dimensional task with Gaussian actuator noise. The numbers show a mechanism, not the error rate of any real robot.

With a perfect actuator the plan always works. Each step’s small random error accumulates, so the final error grows roughly with the square root of the number of steps. Noise of 0.03 per step, small next to the 0.05 step length, already makes the plan fail about seven times in ten. Try changing noise, steps and tol and re-running. Lesson 1.5 shows what happens when the robot is allowed to look at the result while it moves.

Why it is harder (2): the world does not wait

A text model can take as long as it needs, because the conversation does not change while it computes. A robot’s world keeps moving. RT-1, an early large robot model, ran at 3 Hz,^[5] which means it chose a new action about every third of a second. Whatever the world does in that interval, it does without the model’s attention. The standard tool for coping, feedback, is the central subject of control theory.^[6]

rate_hz = 3                                    # RT-1 produced actions at 3 Hz
interval = 1 / rate_hz
for speed in (0.1, 0.3, 1.0):                  # speed of something in the scene, in metres per second
    print(speed, "m/s ->", round(speed * interval * 100, 1), "cm between decisions")

0.1 m/s -> 3.3 cm between decisions
0.3 m/s -> 10.0 cm between decisions
1.0 m/s -> 33.3 cm between decisions

This is plain arithmetic, distance = speed × time. An object drifting at 0.3 m/s moves 10 cm between two decisions at 3 Hz, which is larger than many things a robot has to grasp. This is why the speed of a model, and not only its accuracy, matters in physical AI, a point that returns in lesson 1.5 and in the evaluation phase.

Why it is harder (3): data comes from slow, costly machines

Language models learn from text that already exists. A robot model needs recordings of robots acting, and those have to be made. A large grasping study collected about 800,000 grasp attempts over two months, using between 6 and 14 robots at a time.^[7] RT-1 was trained on about 130,000 episodes covering more than 700 tasks, collected by 13 robots over 17 months.^[5] Partly for this reason, labs have started pooling data: the Open X-Embodiment collaboration assembled more than a million trajectories from 22 robot types contributed by 21 institutions.^[8]

attempts = 800_000
days = 60                                       # "two months", taken as 60 days
for robots in (6, 14):
    print(robots, "robots ->", round(attempts / (robots * days)), "attempts per robot per day")

6 robots -> 2222 attempts per robot per day
14 robots -> 952 attempts per robot per day

This is our own arithmetic from the figures in the abstract, so treat “60 days” as approximate. It shows that even a fleet of robots working around the clock produces on the order of a thousand attempts per robot per day, and that figure is for a simple, repeatable task. Compare that with the web text a language model reads.

Why it is harder (4): simulation is not the real world

Training in simulation is cheap and safe, but a policy that works in simulation can perform worse on real hardware because the two environments differ. This is the “reality gap”, and surveys catalogue the main responses: domain randomization, domain adaptation, imitation learning, meta-learning and knowledge distillation.^[9] Domain randomization randomises the simulator so that, as its authors argue, with enough variability the real world looks like just another variation. In one early study a detector trained only on simulated images with random, non-realistic textures localised real objects to about 1.5 cm.^[10]

The toy below shows the logic. A classifier must tell whether an object is present from two brightness readings, one over the object’s location and one over empty background. If the simulator always uses the same lighting, the background reading carries no information, so the classifier ignores it and learns a fixed brightness threshold. Real lighting then varies and that threshold fails. If the simulator randomises the lighting, the classifier is forced to use the background reading as a reference, and it still works when lighting changes:

def make_data(n, lighting_low, lighting_high):
    present = rng.integers(0, 2, n)
    light = rng.uniform(lighting_low, lighting_high, n)
    region = light + 0.3 * present + rng.normal(0, 0.02, n)   # brightness over the object's location
    background = light + rng.normal(0, 0.02, n)               # brightness over empty background
    return np.column_stack([region, background, np.ones(n)]), 2 * present - 1.0

def fit(X, y):
    w, *_ = np.linalg.lstsq(X, y, rcond=None)                  # a linear classifier fitted by least squares
    return w

def accuracy(w, X, y):
    return float(np.mean(np.sign(X @ w) == y))

X_real, y_real = make_data(5000, 0.2, 0.8)                     # "the real world": lighting varies a lot
for name, lo, hi in (("fixed lighting in simulation", 0.5, 0.5), ("randomised lighting in simulation", 0.2, 0.8)):
    X_sim, y_sim = make_data(5000, lo, hi)
    w = fit(X_sim, y_sim)
    print(name, "| simulation:", round(accuracy(w, X_sim, y_sim), 2), "| real:", round(accuracy(w, X_real, y_real), 2))

fixed lighting in simulation | simulation: 1.0 | real: 0.75
randomised lighting in simulation | simulation: 1.0 | real: 1.0

Toy model. The two “cameras”, the lighting and the classifier are invented for this example. Real domain randomization changes the appearance and physics of a full simulator, as in the cited work.

Both versions are perfect inside the simulation, which is the trap: a perfect simulation score says little about the real world. Only the version trained on varied lighting keeps working when the lighting changes.

Why it is harder (5): mistakes have consequences

A wrong answer from a chatbot can be re-generated. A wrong motion can break an object, a robot or a person. A survey of foundation models in robotics names, among the main open problems, the scarcity of robot-relevant training data, safety guarantees and uncertainty quantification, and real-time execution.^[2] Each of the reasons above feeds into these three. Nothing in this roadmap removes the problem, so the later phases keep returning to evaluation, safety monitors and fallback behaviour.

The difficulty is not a new discovery. The roboticist Hans Moravec observed in 1988 that it is comparatively easy to make computers exhibit adult-level performance on intelligence tests or at checkers, and difficult or impossible to give them the perception and mobility skills of a one-year-old.^[11] Rodney Brooks argued in 1990 that intelligence should be built on, and tested against, sensing and acting in the physical world, instead of on symbolic models of it alone.^[12] Today’s VLA models are a recent attempt to meet that challenge with large-scale learning.^[13]

Common misconceptions

Engineering takeaways

1. An agent perceives through sensors and acts through actuators. Physical AI closes that loop through the real world.
2. Sensing is noisy and partial, and actions are imprecise, so errors accumulate unless something corrects them.
3. The world keeps moving while a slow model decides: distance equals speed times time.
4. Robot data comes from slow, costly machines, which is why datasets are pooled.
5. A simulator that differs from reality can give a policy that fails on hardware, and randomizing the simulator is one response.
6. Mistakes can break things, so safety and evaluation are part of the problem.
7. Surveys list data scarcity, safety and uncertainty, and real-time execution as the main open problems.
8. VLA models are a recent attempt to address these problems with large-scale learning.

Physical AI is AI that has to get it right in a world that does not wait and does not repeat itself exactly.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

You now have a working picture of physical AI and why it is hard. Next is one specific kind of model built for it: the vision-language-action model.

Back to the VLA roadmap