P1-T1 — What Robot Perception Is

Learning objectives

After this lesson, you should be able to:

• Define robot perception and name the kinds of output it produces.
• Distinguish perception from sensing, localization, mapping and planning.
• Explain why perception is an inverse problem and why its answers are uncertain.
• State what a downstream consumer needs from a perception output besides the answer itself.

The idea: working backwards from the measurements

Imagine standing in a dark room and touching a few points on a surface with a stick. You get a list of contact positions. From that list you must say whether you touched a table, a chair or a wall. The list does not contain the word “table”; you have to infer it from the pattern, from what you know about tables, and from what else is around. A robot is in the same position, with a camera or a laser instead of a stick.

The process that produces measurements runs forward: the world has objects, light bounces off them, a lens focuses it, a chip counts photons. The physics of that direction is well understood. Perception runs backward: given the numbers, say what produced them. Running a process backwards is harder than running it forwards, because information is thrown away on the way. A photograph loses depth. A range scan loses colour. A bright pixel might be a white wall in shade or a grey wall in sunshine.

Sensing is what the world does to a sensor. Perception is what the robot concludes about the world from the sensor.

The outputs of perception are of several kinds, and later modules build each of them:

• Geometry: where surfaces and free space are, in a metric frame (Modules 3 and 5).
• Features: distinctive local patterns that can be found again in another image (Module 4).
• Objects: a class, a position, a size and a confidence for each thing (Module 6).
• Identity over time: the same object recognised in the next frame, with a velocity (Module 7).
• Scene understanding: what is where, in a form a task can use, often by combining several sensors (Module 8).

How perception differs from its neighbours

The robotics tracks on this site are separated by the question they answer. They depend on each other, but they are not the same job.

The boundaries blur in practice. Visual SLAM uses image features, which are perception, to estimate pose. A planner needs the objects that perception found. The distinction is useful because it tells you where a failure belongs: if a robot drives into a person, you ask whether the sensor saw the person (sensing), whether the system concluded “person” (perception), whether the robot knew where it was relative to the person (localization), and whether the planner chose to stop (planning).

The mathematics: a forward model and its inverse

Write the state of the world that matters for the robot as x (for example, the position and class of every object near the robot). A sensor produces a measurement z. The forward model is:

x

the true state of the world that the robot cares about: object positions, shapes, classes.

z

the measurement: an image, a range scan, a list of points. It is a vector, often with millions of entries.

h(·)

the sensor model: the physics that turns a world state into an ideal measurement (for a camera, projection of the scene through a lens).

n

noise: the part of the measurement that is not explained by x, treated as random with a known or estimated distribution.

Perception wants x from z. Because h usually loses information, several different states can produce nearly the same measurement. A large object far away and a small object nearby can fill the same pixels; a flat poster of a corridor and a corridor produce similar images. The inverse of h is therefore not a function. This is the sense in which the problem is ill-posed: the measurement alone does not determine the answer.

The usual remedy is to add what you know about the world. In a probabilistic formulation this is Bayes’ rule, the same rule that Localization & SLAM uses for pose:

p(x | z)

the posterior: how plausible each world state is after seeing the measurement. The robot should be able to give this, or a summary of it.

p(z | x)

the likelihood: how probable the measurement is if the world were x. This comes from the sensor model h and the noise n.

p(x)

the prior: what is plausible before looking: objects rest on the floor, people are about 1.5 to 2 m tall, a car is not on the ceiling.

Not every perception method is written this way. A neural network that maps an image to detections does not evaluate this formula, and it can be regarded as learning a function that has absorbed the priors from its training data. The formula is still the right way to think about why a perception system needs assumptions, and why its output should be an estimate with uncertainty, not a fact.

A perception output, written out

What a robot consumes is a structured summary, not the posterior itself. A useful perception output for one object has at least these fields:

The last two fields are easy to forget and are the cause of many integration failures: a position without a frame is ambiguous (relative to what?), and an object without a timestamp cannot be compared with anything that moved since.

A worked example: how much is thrown away

A robot arm has a camera with a resolution of 1280 × 720 pixels and three colour channels of 8 bits each. One frame therefore contains

The robot’s task is to pick up a mug. What the gripper needs is a position and an orientation of the handle, about six numbers, and a decision that “this is the mug” and not the jar next to it. Perception reduced about 2.8 million numbers to about 6 numbers plus a label and a confidence, for example:

Now suppose the lamp in the room flickers, and the frame is darker by half. Every one of the 2.8 million numbers changed, and the right answer did not. Perception is, in large part, the art of keeping what matters and discarding what does not. It is also why perception systems can fail on changes that are trivial for a person: the system learned or was told to rely on cues that happened to be correlated with the answer in the data it was built on, and the lamp changed one of those cues.

The same measurement, two different worlds

A range sensor reports 1.2 m straight ahead. Two worlds produce this reading: a pallet that stays where it is, and a person walking towards the robot who happens to be at 1.2 m now. The single measurement cannot separate them, so the robot needs a second measurement (does the distance change?) or knowledge about the scene. This is why the temporal modules (Module 7) exist, and why a single reading is a poor basis for a safety decision.

Engineering implications and limitations

An output is a contract. Whatever uses the perception result (a planner, a controller, a person) needs to know the frame, the time, the units and the uncertainty. A perception stack that outputs positions without covariance leaves the planner unable to distinguish a confident answer from a guess.

Perception is judged by the task, not by the picture. A system that outlines every object beautifully but gives the wrong distance to an obstacle has failed for a mobile robot. A segmentation that is two pixels off is fine for navigation and not for grasping a screw. The metric must follow the need (Lesson P1-T6).

Latency is part of correctness. A result that arrives 300 ms late describes the world as it was. A person walking at 1.4 m/s moves about 0.4 m in that time. The timestamp matters as much as the value.

Failure is normal. Every perception component has conditions under which it breaks: glare, rain, dust, glass, unusual objects, unfamiliar scenes. The engineering task is to know the boundaries, detect when you are outside them and degrade safely. The cost of a missed obstacle is very different from the cost of a false alarm; the thresholds must reflect that.

Do not duplicate the neighbours. If you need a pose, use localization; if you need a persistent map, use mapping. Perception supplies the observations they consume, such as features and detected objects, and consumes their output as a prior (for example, a known map tells you where walls are not worth detecting). The interfaces between them are where integration bugs hide.

Common misconceptions

Engineering takeaways

1. Perception turns measurements into structured information a robot can act on: geometry, features, objects, identities over time and scene understanding.
2. It differs from sensing (raw data), localization (pose), mapping (persistent model) and planning (actions); each question has its own failure modes.
3. Perception is an inverse problem: measurements underdetermine the world, so it needs assumptions, and its outputs are estimates.
4. A perception output is a contract: class, position, size, confidence, timestamp and frame, plus an honest “unknown”.
5. Judge perception by whether the downstream task works, and design for the failures you expect.

References and further reading

Standard textbooks; the examples and numbers in this lesson are illustrative.

• Thrun, S., Burgard, W. and Fox, D. (2005). Probabilistic Robotics. MIT Press: the measurement model and the Bayes filter used throughout robot perception and estimation.
• Szeliski, R. (2022). Computer Vision: Algorithms and Applications, 2nd ed. Springer: the forward and inverse view of image formation and recognition.
• Siegwart, R., Nourbakhsh, I. R. and Scaramuzza, D. (2011). Introduction to Autonomous Mobile Robots, 2nd ed. MIT Press: sensors, perception and their role in the navigation loop.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: the three levels at which a robot can hold information about a scene: raw measurements, representations and semantics, and what each one keeps and loses.

Related content

Nothing below is required to finish this lesson.

• Lab
  P1-L1: Inspect and Interpret Sensor Observations

  Read a camera frame, a range scan and a depth map of the same scene and say what each shows and hides.