P1-T2 — Sensors, Measurements, Representations and Semantics

Learning objectives

After this lesson, you should be able to:

• Distinguish a measurement, a representation and semantic information, with an example of each.
• Name common perception representations and say what each is good and bad at.
• Convert a range-and-bearing measurement into Cartesian coordinates and explain what the conversion assumes.
• Explain why conversions between representations are lossy and cannot be freely reversed.

The idea: re-describing the same scene

A photograph of a kitchen can be described in many ways. As a list of colours at each pixel. As “bright rectangle on the left, dark region below it”. As “a window above a sink”. All three describe the same scene, at different levels, and each answers different questions. The pixel list answers “what colour is the pixel at row 400, column 200?” and nothing else easily. “A window above a sink” answers “where can I wash a cup?” and nothing about exact colours.

Robots do the same thing, in a pipeline of descriptions:

• Measurement is what the sensor delivers, with its own noise and its own coordinates: pixel intensities, a range for every angle, a timestamped acceleration.
• Representation is a re-organisation of measurements into a form where geometry or structure is easier to use: a point cloud in metres, a grid of free and occupied cells, a set of corner features, an edge map, a mask that marks the pixels that belong to one object.
• Semantics attaches meaning: a class label, an instance identity, a category such as “graspable” or “drivable”.

Going up the stack adds meaning and assumptions; going down the stack recovers detail. You cannot rebuild the measurement from the meaning.

Common representations and what they are good at

No representation is best. Each makes some questions cheap and others impossible.

Notice the “empty space” entry of the point cloud and the “unknown” state of the occupancy grid. A set of points records only where something was hit. It does not distinguish “free” from “not measured”. The occupancy grid adds that distinction, which is one reason it is the usual input to planners.

The mathematics: converting a measurement into a representation

Most conversions are changes of coordinates. The most common one is from the native form of a range sensor (distance and direction) to Cartesian points. For a scanner that returns a range r at azimuth φ and elevation θ:

r

measured range in metres, from the sensor origin to the reflecting surface.

φ

azimuth: the horizontal angle from the sensor’s forward axis, positive to the left (counter-clockwise seen from above).

θ

elevation: the vertical angle above the sensor’s horizontal plane.

(x, y, z)

the point in the sensor frame: x forward, y left, z up.

For a camera, the conversion that creates a representation with metric content needs one more number than the image gives: the depth. Given a depth d for a pixel (u, v) and the camera constants, the 3D point is

(u, v)

pixel column and row.

d

depth along the optical axis in metres, supplied by a depth sensor or a stereo pair.

f_x, f_y

focal lengths expressed in pixels: they say how many pixels a unit of angle covers.

(c_x, c_y)

the principal point: the pixel where the optical axis meets the image.

Module 3 derives this model. The point for now is that every one of these formulas contains assumptions: that the sensor origin and axes are as stated, that the measurement is where the light came from, that the angles are in radians. A conversion is only as right as the assumptions behind it.

Why conversions lose information

Binning points into a voxel grid of side s replaces every point inside a cell by a single bit, “occupied”. If a cell holds n points, the position of each point inside the cell is discarded, so the position uncertainty becomes about s/√12 per axis even if the sensor was more precise. A threshold on an image turns 256 levels into 2. Detecting a box removes the shape inside it. These are not bugs; they are the reason the representation is cheap and useful. But a step that discards information cannot be undone: from an occupancy grid you cannot recover the points.

This inequality, from information theory, says that a processing step cannot increase what the data tell you about the scene. What a step can do is add assumptions: a classifier adds knowledge from its training data, and that knowledge can be wrong for the scene at hand.

A worked example: one scan, three representations

A planar LiDAR sweeps 360° at 1° steps and returns a range for each direction: 360 numbers. A wall 3 m away on the left, and a box on the right. Take two measurements:

• φ = 90° (to the left), r = 3.00 m → x = 3.00 × cos 90° = 0.00 m, y = 3.00 × sin 90° = 3.00 m. The wall point is 3 m to the left.
• φ = −30° (forward and to the right), r = 2.00 m → x = 2.00 × cos(−30°) = 1.73 m, y = 2.00 × sin(−30°) = −1.00 m. A point on the box is 1.73 m ahead and 1.00 m to the right.

Doing this for all 360 ranges gives a point cloud of at most 360 points. It keeps every measured position and loses the order, and it says nothing about where there was no return.

Now put the points in an occupancy grid with 0.10 m cells covering 10 m × 10 m, which has 100 × 100 = 10 000 cells. Suppose the scan puts about 360 points into 140 distinct cells. Each of the 140 cells is “occupied”, the cells along each ray before its hit can be marked “free”, and everything else is “unknown”. The grid is bigger than the cloud (10 000 cells against 360 points), but its size does not change as more points arrive, and it answers “is the cell at (1.7, −1.0) free?” directly, which the cloud cannot.

Finally, semantics: a clustering step finds that 18 of the points form a compact group about 0.6 m wide and labels it “box”, while the 150 points on a straight line become “wall”. From 360 numbers we reach two statements. They are compact and decision-ready, and they are only as good as the clustering rule: a person standing against the wall becomes part of “wall” unless the rule says otherwise.

What did each step cost?

Engineering implications and limitations

Choose the representation for the consumer. A planner wants free space, so give it a grid. A grasp planner wants the shape of one object, so give it points on that object. A tracker wants one position per object per frame. Handing every consumer the raw measurement pushes the same conversion into each of them and makes them inconsistent.

Convert once, in one place, with the frame and time attached. The most common integration bug is two modules converting the same measurement with different assumptions about the sensor’s mounting or axes. Lesson P1-T4 treats the frames.

Keep the raw data you may need. Logging only the high-level result makes failures impossible to diagnose, because you cannot recompute the lower levels. Many teams log raw measurements and every intermediate representation for exactly this reason.

Mind the cost. A 64-ray, 10 Hz LiDAR with 2 000 returns per ray produces over a million points per second. Memory and bandwidth set limits on what you can keep, and voxel downsampling is often the first step. Choose the cell size from the accuracy you need, not from habit.

Know where the meaning came from. A semantic label comes from a rule or a model with its own training data and failure modes. Report it with a confidence, and keep the geometry that supports it so that a downstream check can say “this label disagrees with the shape”.

Common misconceptions

Engineering takeaways

1. A measurement is the sensor’s native output; a representation re-organises it for a task; semantics attach meaning. Each level adds assumptions and discards detail.
2. Images, depth images, point clouds, grids, features, boxes and masks each make some questions cheap and others impossible.
3. Conversions such as range to Cartesian points are changes of coordinates that assume known sensor axes, units and time.
4. Information lost in a conversion cannot be recovered; keep raw data when you may need it.
5. Choose each representation for its consumer, convert once with frame and time attached, and report semantics with a confidence.

References and further reading

Textbook references; the scan numbers in the example are illustrative.

• Thrun, S., Burgard, W. and Fox, D. (2005). Probabilistic Robotics. MIT Press: range-sensor measurement models and occupancy grid maps.
• Szeliski, R. (2022). Computer Vision: Algorithms and Applications, 2nd ed. Springer: image formation and representations.
• Cover, T. M. and Thomas, J. A. (2006). Elements of Information Theory, 2nd ed. Wiley: the data-processing inequality.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: the whole chain at once, from raw data to a decision, and how an error in one stage propagates to the end.

Related content

Nothing below is required to finish this lesson.

• Before
  P1-T1: What Robot Perception Is

  Define perception, distinguish it from sensing, localization, mapping and planning, and name its outputs.

• Lab
  P1-L1: Inspect and Interpret Sensor Observations

  Read a camera frame, a range scan and a depth map of the same scene and say what each shows and hides.