P1-T3 — The Perception Pipeline

Learning objectives

After this lesson, you should be able to:

• List the stages of the perception pipeline and the data each one passes on.
• Describe how the quality of the final result depends on every stage, using a simple error-propagation model.
• Compute a latency budget for a pipeline and judge whether it fits a robot task.
• Explain feedback in a pipeline and where learned end-to-end systems blur the stages.

The idea: a chain, and the weakest link

Think of a factory line. Raw material comes in, each station does one job, and the product leaves at the end. A defect introduced at one station is carried to the end, and a station cannot repair what an earlier one destroyed. A perception pipeline behaves the same way.

The six stages

1. Acquire. The sensors produce frames, scans or points, with timestamps. Quality here is set by physics and hardware: exposure, range, field of view, rate (Sensors track).
2. Preprocess. Clean up and normalise: remove noise and invalid readings, undistort the image, align the data to a frame, adjust for lighting. Module 2 covers images; Module 5 covers point clouds.
3. Represent. Convert into a form the next stages can use: a mask, features, a point cloud, a depth image, a grid (Lesson P1-T2).
4. Detect and estimate. Find things and measure them: objects with boxes and classes, planes, poses, distances.
5. Track and understand. Link detections over time into tracks with velocity, combine sensors, and assemble a scene a task can use (Modules 7 and 8).
6. Decide. The consumer, such as a planner or a controller, turns the result into an action. It belongs to other tracks, but it defines what the pipeline must deliver.

The stage boundaries are a design choice, not a law. A system may merge two stages, skip one, or run several in parallel for different sensors. What stays constant is the set of questions: was the data good, was it re-organised sensibly, was the right thing found, was it followed correctly over time, and was the decision justified?

The mathematics: errors and delays add up

Suppose the stages are independent and each one succeeds with probability p_i (it passes on a usable result). The chance that the whole chain works is the product:

p_i

the probability that stage i delivers a result good enough for the next one, under the conditions of interest.

n

the number of stages in the chain.

P(success)

the probability that the final output is correct, assuming independence. If errors are correlated (bad lighting hurts several stages at once) the real chain is usually worse than this.

Six stages that each work 95% of the time give 0.95⁶ ≈ 0.735. A system whose parts each look good can be wrong one time in four. This is why individual component metrics can be misleading and why the end-to-end result must be measured (Lesson P1-T6).

Delays add too. If stage i takes time t_i per frame, the time from the moment the world is sampled to the moment the decision is available is

t_i

the processing time of stage i for one frame, in seconds.

latency

the delay between the world being sampled and the decision: it determines how old the information is when it is used.

throughput

frames per second the pipeline can sustain: it is limited by its slowest stage if the stages run concurrently.

The distinction matters. A pipeline that takes 40 ms per stage over five stages has a latency of 200 ms, even though it can output a new result every 40 ms. The robot reacts to the world as it was 200 ms ago.

How much delay a robot can tolerate

A robot moving at speed v with a total latency L travels vL before it can react. To brake from speed v with deceleration a it needs a further distance v²/(2a). The minimum distance at which an obstacle must be seen is therefore

A worked example: can this pipeline stop the robot?

A delivery robot drives at v = 1.5 m/s and can decelerate at a = 1.0 m/s². Its camera pipeline has these processing times:

The latency is 33 + 12 + 45 + 10 + 20 = 120 ms = 0.12 s. During that time the robot travels 1.5 × 0.12 = 0.18 m. Braking takes 1.5² / (2 × 1.0) = 1.125 m. So the minimum distance is

If the detector only finds a person reliably within 3 m, the margin is 3.0 − 1.305 ≈ 1.7 m. But if the detector’s reliable range drops to 1.2 m in rain, the margin becomes negative: the robot cannot stop in time even though every stage ran correctly. The fault is not in any stage; it is in the product of range, latency and speed. The remedy is a lower speed in poor conditions, which means the robot must know when conditions are poor.

Error flow in the first lab

In the lab for this module, the robot must pick up the red block. The pipeline is: image → colour mask → blob → position in the robot frame → action. Suppose the colour test has the wrong threshold, so the blob includes part of a yellow block. The blob centre moves by several pixels; at 5 cm per pixel, 6 pixels is 30 cm. The position, the nearest-target choice and the grasp decision are all wrong, and none of the later stages did anything wrong. The first stage’s error is carried to the end. Debugging means looking at each interface in turn, starting from the first one.

Engineering implications and limitations

Define the interface at each stage. Say exactly what is passed: the type, the frame, the units, the timestamp, the meaning of “nothing found”. A stage that returns an empty list for “no object” and the same empty list for “I failed” hides faults.

Test stages separately and together. A stage that passes its own test on clean data may fail when fed the real output of the stage before it, whose errors it was never tested with. Unit tests catch interface bugs; end-to-end tests catch compounding errors.

Log every interface. If you record the data between stages, a failure in the field can be replayed through the pipeline and located. If you record only the final output, you can only guess.

Use feedback. A downstream stage often knows something that would help an upstream one: the tracker knows where an object is likely to be next, so the detector can search a smaller region; the planner knows what matters, so the camera can expose for the road and not the sky. Feedback makes pipelines faster and more robust, and also makes them harder to analyse because errors can circle.

End-to-end learned systems. A neural network can replace several stages with one function from pixels to detections or even to actions. The stages are then inside the network and cannot be inspected separately, but the questions remain: you still need to test the data, the intermediate result if the network exposes one, and the output against the task. Many deployed systems combine learned detection with classical geometry and tracking for exactly this reason.

Budget time as carefully as accuracy. Decide the latency allowed for each stage early. A detector that is more accurate but 100 ms slower can make the robot less safe.

Common misconceptions

Engineering takeaways

1. The pipeline stages are acquire, preprocess, represent, detect and estimate, track and understand, and decide; each passes a defined data type to the next.
2. Chain reliability is the product of stage reliabilities under independence, so good parts can still make a poor system.
3. Latency is the sum of stage times plus waiting; throughput is set by the slowest stage. Latency determines how stale the output is.
4. The distance needed to react and stop is v·L + v²/(2a): range, latency and speed must be considered together.
5. Define interfaces, test stages alone and together, log between stages, and use feedback with care.

References and further reading

Textbook references; the timing numbers in the example are illustrative, not measurements of any real system.

• Siegwart, R., Nourbakhsh, I. R. and Scaramuzza, D. (2011). Introduction to Autonomous Mobile Robots, 2nd ed. MIT Press: the perception-to-action loop.
• Thrun, S., Burgard, W. and Fox, D. (2005). Probabilistic Robotics. MIT Press: modelling and propagating sensor uncertainty.
• Kinematic stopping distance d = v²/(2a) and reaction distance v·t are standard results of elementary mechanics.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: the geometry that lets the stages talk to each other: coordinate frames and rigid transforms.

Related content

Nothing below is required to finish this lesson.

• Before
  P1-T2: Sensors, Measurements, Representations and Semantics

  Distinguish a raw measurement, a representation and semantic understanding and say what each loses and keeps.

• Lab
  P1-L3: Build a First Perception Pipeline

  Assemble raw range data into a useful output (is the way clear, and how far?) and evaluate it.