T3 — The Physical AI Stack at a Glance

Learning objectives

After this lesson, you should be able to:

• Describe the major layers of a Physical AI system.
• Explain the purpose of each layer.
• Distinguish task planning from motion planning and control.
• Explain how information flows from sensors to decisions and actions.
• Explain why Physical AI is a closed-loop system.
• Identify where perception, SLAM, navigation and control fit in the larger architecture.

The goal is the map, not the details. Each layer gets its own lessons or tracks, and none is taught in depth here.

The one-sentence idea

A Physical AI system connects high-level goals to physical action through a hierarchy of perception, representation, planning, execution, control, and feedback.

A hierarchy here means that each layer works at one level of detail. The top layers think in goals (“the bottle is with the user”). The bottom layers think in motor commands. Each layer takes a decision from above and makes it more concrete for the layer below, while information about the world flows back up. Feedback ties the two directions into a loop.

Why we need a stack

Suppose a person says: “Bring the red bottle from the kitchen to me.” A robot cannot solve this with a single operation. It has to settle a series of different questions:

• What does the command mean?
• Which object is the red bottle?
• Where is it?
• Where is the robot?
• How should it get there?
• How should it grasp the bottle?
• How should it carry it?
• What if the grasp fails?
• What if the environment changes?
• How does it know the task is complete?

These are different kinds of problems, solved at different levels of detail and at different speeds. The engineering answer is a stack: split the job into layers, give each layer one responsibility, and define what passes between them.

The Physical AI stack

Read the diagram from the top: it shows how a goal becomes motion. Then follow the return path at the bottom, where the robot’s sensors read the result and feed it back.

This is a conceptual architecture, not a mandatory one. Real robots merge layers, skip some, or reorder them. A learned policy might fold several layers into one neural network, and many layers run at once rather than in sequence. The map is still useful because it lets you ask, for any problem, “which layer is responsible for this?”

Layer 1: Human goal / task

The question it answers: What does the human want?

The stack starts with an objective, usually in words: “Bring the red bottle to the user.” Three terms are easy to blur. A command is what the human says. A goal is the world state that would satisfy them (the bottle is with the user). A task is the work needed to reach that goal. The system must turn the first into a machine-understandable form of the second. A later lesson covers this in detail.

Layer 2: Grounding and task understanding

The question it answers: Which real things and places do these words mean?

Language is not tied to the physical world until something ties it. “Red bottle” must refer to an actual object, and “to me” must refer to a person or a place. This takes four ideas: object grounding (which object?), spatial grounding (which place?), task interpretation (what does “bring” involve?) and ambiguity (what if two bottles are red, so the robot should ask?). Language models can help here, but they are one tool, and a later lesson looks at them properly.

Layer 3: Perception

The question it answers: What is out there, according to the sensors?

Perception converts sensor measurements into useful observations. A camera yields objects, LiDAR (a laser scanner) yields geometry, an IMU (inertial measurement unit) yields motion measurements, and encoders yield wheel or joint measurements.

Go deeper: the Sensors track covers how robots measure the world. The Perception track is coming soon, and T4 follows this step in detail.

Layer 4: World model / belief

The question it answers: What does the robot currently believe?

This layer connects directly to T2. Observations pass through state estimation and reasoning to produce the robot’s current understanding of the world:

robot = kitchen
red_bottle = table
gripper = empty
door = open

This picture may carry uncertainty: the bottle is probably on the table, not certainly. Everything above this layer reads the world model, not the raw sensors, so it is the information that higher-level reasoning actually runs on.

Layer 5: Task planning

The question it answers: What should the robot do?

Given a goal and the current belief, task planning produces a sequence: find the bottle, navigate to it, grasp it, verify the grasp, navigate to the user, place the bottle, verify completion. It works with goals, actions, preconditions (what must be true before an action, such as “the gripper is empty” before grasping), effects (what is true after) and task sequences. Planning algorithms are not taught here.

Layer 6: Skills / behavior

The question it answers: How is each step carried out?

A skill is a reusable robot capability with a clear interface, such as navigate(), grasp(), place(), open() or close(). A high-level planner should not command individual motors. It calls skills, and each skill uses the lower-level system.

Skills are often organized with behavior trees, a way to arrange skills, conditions and fallbacks so that a failed step triggers a defined response. We only name them here; a later lesson explains how they work.

Layer 7: Motion planning

The question it answers: What physical movement should the robot make?

Task planning decides what to do: “Navigate to the kitchen.” Motion planning decides how the body should move: which collision-free path to take. For the task “reach the table,” the motion planner chooses a route around the chair. The two are separate jobs, solved separately and at different speeds.

Go deeper: path-planning methods belong to the Navigation track, which is coming soon.

Layer 8: Control

The question it answers: How do I make the actuators do it?

Planning produces a desired motion. Control makes the robot physically follow it, correcting for friction, load and error using feedback.

For example, a planner says “move forward at 0.5 m/s.” The controller works out the motor commands needed to actually achieve that speed, and adjusts them as the wheels slip or the load changes.

Go deeper: the Control labs teach how controllers are tuned and made safe.

Layer 9: Physical robot

The question it answers: What actually happens?

At the bottom, the system meets hardware: motors, wheels, joints, grippers and sensors. Physical execution brings noise, delay, friction, dynamics, actuator limits and unexpected events. This is where the idealized plan meets reality, and why the layers above must keep checking what really happened.

Feedback: the stack is a loop

This is the most important idea in the lesson. The stack is not simply input, then AI, then robot. It is a loop in which every action produces new observations.

Inside the loop, each cycle looks like this:

Physical AI is closed-loop intelligence.

This is the loop from T1, now placed on the stack. The observation step feeds the belief you met in T2, and the verify step is exactly what Lab 09 asks you to build.

One complete example

Follow “Bring the red bottle from the kitchen to the user” through the whole stack.

Steps 10 to 13 are the ones a plan-then-execute design leaves out, and they are what make the system robust.

Where existing robotics topics fit

Many of these layers are already subjects on FixTheRobot. They form a pipeline of capabilities:

Physical AI sits above and across these capabilities. They are not isolated subjects. Physical AI is where the system learns to coordinate them toward goals. Here is how the tracks map to the stack, with nothing duplicated:

Robot Foundations

Start here: sensing, moving, localizing and planning for beginners.

Control

Layer 8: making motion follow a command.

Sensors

The measurements that feed Layer 3.

Perception, Localization & SLAM, Navigation

Layers 3, 4 and 7 in depth. All three are coming soon.

Physical AI

The coordination across all layers, toward goals.

Common misconceptions

Engineering takeaways

1. Physical AI is a complete system, not a single model.
2. Sensors provide observations.
3. Perception extracts useful information.
4. A world model represents the robot’s current understanding.
5. Task planning determines what should happen.
6. Skills and behaviors execute tasks.
7. Motion planning determines physically feasible movement.
8. Control makes the robot follow the desired behavior.
9. Feedback continuously updates the system.
10. Verification, recovery and replanning make the system robust.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

You now have the map of the Physical AI system. The next lessons will zoom into each part of this architecture.