Skip to the lesson
Robotics Lab Explore Robotics / Physical AI / Theory / T4

Physical AI · Theory

T4 — From Sensors to Observations

How robots turn physical measurements into information they can reason about.

  • Lesson 4
  • Physical AI Theory
  • Sensing and perception
  • About 10 minutes to read

In T2 we saw what a robot is trying to know, and in T3 we saw the whole stack. This lesson zooms into the first step: how the physical world becomes something the stack can use.

Learning objectives

After this lesson, you should be able to:

  • Explain what a sensor measures.
  • Distinguish raw measurements from observations.
  • Explain why sensor measurements are incomplete and noisy.
  • Identify common sensor failure modes.
  • Explain the role of preprocessing.
  • Distinguish sensing from perception.
  • Explain why multiple sensors may complement each other.
  • Explain how observations feed state estimation and Physical AI decisions.

The question throughout is simple: what information does the robot actually receive from the physical world, and how does it become useful? This is not a full Sensors or Perception course; those tracks go deeper.

The one-sentence idea

A sensor gives the robot measurements; the Physical AI system must turn those measurements into useful observations about the world.

Five terms carry this sentence, and they are not interchangeable:

Sensor
A device that responds to a physical quantity. Example: a camera responds to light.
Measurement
The raw, or nearly raw, output of a sensor. Example: an image, or a list of distances.
Processing
The cleaning and interpretation applied to measurements. Example: removing invalid readings, finding objects.
Observation
Information derived from measurements that is useful for reasoning. Example: “red bottle detected, here.”
State / belief
What the robot concludes from observations (T2). Example: “the bottle is probably on the table.”

What a sensor actually does

A sensor never delivers “the red bottle is on the table.” It measures a physical quantity and nothing more:

Camera
Light becomes an image.
LiDAR
Reflected laser pulses become ranges, or points. (LiDAR is a laser scanner that measures distance.)
IMU
Acceleration and angular velocity. (An IMU, or inertial measurement unit, senses motion.)
Wheel encoder
Wheel rotation.
Force sensor
Force and torque.
GPS
Position-related measurements from satellite signals.

The physical world contains states and events. Sensors provide measurements that give us evidence about those states and events. This is the same chain you met in T2:

  1. True world
  2. Sensor
  3. Measurement
  4. Observation
The sensor sits between the world and everything the robot knows about it.

Raw measurement vs observation

This is the central distinction. A LiDAR produces thousands of range measurements per scan. Those are not yet “there is a wall 4 meters ahead.” Processing can turn them into a point cloud, a filtered point cloud, clusters, obstacles and geometric features. A camera works the same way:

  1. Pixels
  2. Image processing / perception
  3. Object detection
  4. “red bottle detected”
A camera’s pixels become an observation only after processing.

The exact boundary between measurement, observation and perception depends on the system architecture, and there is no single universally accepted definition. For this curriculum:

  • Measurement: raw or relatively direct sensor output.
  • Observation: information derived from measurements that is useful to the robot’s reasoning and state-estimation system.

Examples across sensors

SensorRaw measurementPossible observationDoes not directly tell you
CameraPixelsObject detected at image coordinatesExact 3D position, without more information
LiDARRanges, pointsObstacle geometryWhat an object is, automatically
IMUAcceleration, angular velocityA motion-related estimateGlobal position
Wheel encoderWheel rotationWheel displacement, odometryExact global robot position
Force sensorForce, torqueContact informationObject identity
GPSA position measurementApproximate global locationExact indoor pose

From measurement to observation

  1. Physical worldThe source of everything.
  2. SensorResponds to a physical quantity.
  3. Raw measurementRaw or nearly raw output.
  4. PreprocessingCleans and aligns the data.
  5. Feature / signal extractionPulls out edges, clusters, motion.
  6. ObservationUseful information for reasoning.
  7. State estimation / beliefCombines observations over time (T2).
  8. PlanningDecides what to do (T3).
  9. ActionChanges the world.
From the physical world to action. Not every system has every intermediate stage explicitly.

Preprocessing is the unglamorous step that decides data quality. Typical operations:

  • Filtering
  • Removing invalid values
  • Coordinate transformation
  • Calibration
  • Synchronization
  • Normalization
  • Segmentation

Why does it matter? Because errors travel. Bad measurements produce bad observations, bad observations produce bad state estimates, and bad state estimates produce bad decisions.

Sensor limitations

Every sensor gives only a limited view of reality. The main limits, in plain terms:

Range
The sensor only works within a certain distance.
Resolution
Small objects or details cannot be told apart.
Field of view
The sensor sees only part of the environment.
Accuracy
How close a measurement is to reality.
Precision
How much repeated measurements vary.
Update rate
The sensor may not refresh fast enough for the task.
Environmental sensitivity
Lighting, rain, dust, reflective or transparent surfaces, magnetic interference.

Accuracy and precision are different. A sensor can give very consistent readings that are all wrong in the same direction. The Sensors track covers these in depth.

Noise, missing data and outliers

Three problems are normal engineering conditions, not rare accidents.

Noise

Measurements fluctuate around the actual value. The true range is 5.0 m, and the readings are:

4.9   5.1   5.0   4.8   5.2

Missing data

A sensor may temporarily fail to provide useful information. A camera loses an object because something blocks it.

Outliers

A measurement is wildly inconsistent with its neighbors. One LiDAR reading among its neighbors:

5.0   5.1   5.0   27.4   5.1

Downstream systems must be designed for all three. A planner that trusts every reading will swerve around a 27.4 m ghost, or freeze when a reading disappears.

Field of view and occlusion

A sensor can only observe what it can physically measure. An object behind another object, outside the camera’s field of view, around a corner, or hidden by something blocking the LiDAR will not appear in the measurements.

Visible

  1. Robot
  2. Sensor
  3. Object: observed

Occluded

  1. Robot
  2. Sensor
  3. Obstacle
  4. Hidden object: not observed
An obstacle between the sensor and an object removes the object from the measurements.

Not observed does not mean does not exist.

This is the hidden state from T2: the world holds information the robot currently lacks.

Timing and latency

Observations have times attached. Suppose at t = 1.0 s the camera sees an obstacle at position A. At t = 1.2 s the robot receives and processes that observation, and the obstacle may now be at position B. Four things create the gap:

  • Sensor timestamp: when the measurement was actually taken.
  • Processing delay: the time spent turning measurements into observations.
  • Communication delay: the time data spends crossing buses and networks.
  • Stale observations: information that was true once but is old now.

An observation is information about what was observed at a particular time.

This matters most in dynamic environments, where the world keeps moving while the data travels.

Sensing vs perception

Sensing is measuring the physical world. Perception is extracting meaningful information from measurements.

SensorSensing givesPerception gives
CameraPixels“Red bottle detected at this location, with this confidence.”
LiDARRanges, points“Obstacle cluster detected about 3 m ahead.”

The boundary depends on the architecture. Some systems do heavy processing before exposing anything; others expose low-level measurements directly. The distinction to keep is between a measurement and the meaningful information extracted from it. The Perception track (coming soon) takes this step in depth.

Multiple sensors

One sensor is rarely enough, because each is strong where another is weak:

Camera
Rich visual information, but depth may be uncertain.
LiDAR
Strong geometry, but limited understanding of what things are.
IMU
High-rate motion information, but it drifts.
Wheel encoders
Useful motion information, but affected by wheel slip.
  • Camera
  • LiDAR
  • IMU
  • Encoders
  1. Better observations
Combining sensors can give a better picture than any one alone.

Combining them has costs: calibration (knowing exactly where each sensor sits), synchronization (aligning their clocks), coordinate frames (expressing everything in one frame), conflicting measurements, and extra computation. We do not teach fusion methods here.

Observation quality and decision making

Suppose the robot detects a red bottle with confidence 0.98, and in another case a red bottle with confidence 0.42. It should not necessarily behave the same way. With a doubtful observation it has choices:

  • act anyway,
  • observe again,
  • use another sensor,
  • move to get a better view, or
  • ask for clarification.

Observation quality affects what the robot should do next. Later lessons on belief, grounding, verification and deciding when to act, inspect or ask build on this idea.

One complete example

Take “Bring the red bottle from the kitchen to the user,” and look only at the sensing side.

  1. Step 1 — The camera looks at the kitchen

    The raw measurement is an image.

  2. Step 2 — Perception detects objects

    The observation: red bottle detected, with a location estimate and a confidence.

  3. Step 3 — LiDAR provides geometry

    The observation: where obstacles are and where space is free.

  4. Step 4 — Encoders and IMU report motion

    The observation: how the robot itself has probably moved.

  5. Step 5 — The observations feed the world model

    The state or belief is updated with all of it.

  6. Step 6 — The bottle becomes occluded

    There is no current visual observation of it. That does not mean the bottle disappeared, and the robot must keep appropriate uncertainty about where it is.

Where this fits in Physical AI

  1. World
  2. Sensors
  3. Measurements
  4. Processing
  5. Observations
  6. State / belief
  7. Planning
  8. Action
The highlighted part of the stack is T4’s subject: the world through to observations.

T4 covers the transition from sensors to measurements to observations. T2 covered observations to state and belief, and T3 showed state and belief to planning and action. Together they trace one path from the world to action.

Common misconceptions

Misconception 1 “A camera tells the robot what objects exist.”

Correction A camera provides measurements. A perception system interprets them.

Misconception 2 “If a sensor doesn’t detect something, it isn’t there.”

Correction The object may be outside the field of view, occluded, too far away, or missed by the sensing or perception system.

Misconception 3 “More sensors automatically solve perception.”

Correction Multiple sensors provide complementary information but bring calibration, synchronization and fusion challenges.

Misconception 4 “An observation is always the current state.”

Correction Observations can be noisy, delayed, incomplete or ambiguous.

Misconception 5 “Sensor data is either correct or incorrect.”

Correction Real measurements have varying degrees of uncertainty and quality.

Engineering takeaways

  1. Sensors measure physical quantities.
  2. Measurements are evidence about the world.
  3. Observations are useful information derived from measurements.
  4. Sensors have limited range, resolution, timing and reliability.
  5. Noise, missing data and outliers are normal engineering problems.
  6. Not observing something does not mean it does not exist.
  7. Perception extracts useful meaning from sensor measurements.
  8. Multiple sensors can provide complementary information.
  9. Observation quality affects state estimation and decisions.
  10. Physical AI depends on continuously turning physical measurements into useful information.

Bad observations propagate upward: poor sensing can become poor state estimation, poor planning and eventually poor physical action.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

Question 1

A LiDAR returns a point cloud. Is the point cloud itself the complete state of the world?

Explanation

No. A point cloud is a sensor measurement, and it holds only partial information about the environment. It shows only what the laser could reach, from one viewpoint, at one moment, with noise. It says nothing directly about what the objects are, what is hidden behind them, or what is out of range. The robot has to interpret and combine it to form a state estimate.

Question 2

A camera stops detecting a red bottle because another object blocks it. What should the robot conclude?

Explanation

The bottle is currently unobserved. The robot should not automatically conclude that it no longer exists. The detection stopped because the view was blocked, which says nothing about where the bottle is. The robot should keep a belief that it is probably still there, perhaps with growing uncertainty, and can move for a better view.

Question 3

Why might a robot use both a camera and LiDAR?

Explanation

They provide complementary information, and combining them can improve the robot’s understanding. The camera is rich in visual detail, so it can tell what something looks like, but its depth can be uncertain. LiDAR gives strong geometry, but little sense of identity. Using both requires calibration, synchronized timing and a shared coordinate frame.

What’s next

We now know how physical measurements become observations. The next question is: how does the robot represent what those observations mean about the world?

Next lesson

T5 — World Models and Symbolic State

T5 introduces how a robot represents the world in a form it can reason over: objects, their properties, the relationships between them, predicates, and the state of the task.

Nothing below is required to finish this lesson. Each goes deeper than T4 does.