Skip to the lesson
FixTheRobot Explore Robotics / Perception / P3 Geometry and Camera Perception / P3-T6

Perception · Theory · P3

P3-T6 — Projection and Reprojection Errors

Project what you think you know, compare with what you see, and read the difference.

  • Lesson 6 of 6
  • Module 3: Geometry and Camera Perception
  • Theory
  • About 9 minutes to read

If you know where a point is, the camera model tells you which pixel it should land on. If the camera sees it somewhere else, the gap is the reprojection error. It is the standard score for a calibration, the cost that bundle adjustment minimises, and a diagnostic tool: its size says how wrong the model is, and its pattern says why. This lesson defines it, relates pose and parameter errors to pixel errors with numbers, shows how a pixel error becomes a position error, and covers the pitfalls of outliers and of a good score that hides a bad model.

Learning objectives

After this lesson, you should be able to:

  • Define the reprojection error and compute its RMS for a set of points.
  • Relate errors in translation, rotation, focal length and principal point to the pixel error they cause.
  • Convert a pixel error into a position error at a given depth.
  • Use residual patterns and robust statistics to diagnose a calibration or reject mismatches.

The idea: predict, observe, compare

Suppose you know where the corners of a calibration board are in 3D, and you have a camera model. Project the corners through the model: you get predicted pixels. Detect the corners in the image: you get observed pixels. The difference for each corner is a vector, its residual. If the model is right, the residuals are just detection noise, a fraction of a pixel. If the model is wrong, they are larger and have structure.

Observed and reprojected pointsA set of eight points. For each there is a filled dot where the point was observed in the image and a ring where the camera model predicts it. A short line joins them, the residual. One outlier on the right has a much longer residual than the others. Lines are labelled residual and the outlier is labelled mismatch. mismatch observedpredicted by the modelresidual Reprojection error: how far the model puts each point from where it was seen
Eight observed corners with their predicted positions. The short residuals are noise. The long one is a mismatch (a wrong correspondence, not a wrong model).

Reprojection error answers “does the model explain what the camera saw?”. It does not answer “is the model right?”.

Where reprojection error is used

  • Calibration: the quantity minimised to find the intrinsics and distortion, and the headline quality figure (Lesson P3-T2).
  • Pose estimation (PnP): find the camera pose from known 3D points and their pixels by minimising reprojection error.
  • Bundle adjustment and SLAM: jointly refine camera poses and 3D points by minimising the total reprojection error (the Localization & SLAM track).
  • Outlier rejection: a match whose reprojection error is large under the best model is probably wrong (Lessons P4-T3 and P4-T4).

The mathematics: error, sensitivity and statistics

Definition

For a 3D point Xj observed at pixel xj, with camera parameters θ (intrinsics, distortion, pose) and projection function π:

reprojection error and its RMS
ej=xj−π(θ, Xj)e_{j}=x_{j}-\pi (\theta ,\,X_{j})
RMS=(1N) ∑j ∥ej∥2\mathrm{RMS}=\sqrt{(\frac{1}{N})\,\sum _{j}\,\lVert e_{j}\rVert ^{2}}
xj
the observed pixel position of point j (from a detector, with noise).
π(θ, Xj)
where the model predicts point j should appear, given the parameters θ.
ej
the residual vector (du, dv) for point j, in pixels. Its length ‖ej‖ is the reprojection error of the point.
RMS
the root-mean-square error over N points, the usual summary: it penalises large residuals strongly.

How parameter errors move pixels

Using the pinhole model u = f X/Z + c, a small error in each quantity shifts the pixel by approximately:

first-order pixel shifts
Δu≈f δXZ\Delta u\approx \frac{f\,\delta X}{Z}
(translation error δX across the optical axis)
Δu≈f δθ\Delta u\approx f\,\delta \theta
(rotation error δθ in radians)(\text{rotation error}\,\delta \theta \,\text{in radians})
Δu≈(δff)(u−cx)\Delta u\approx (\frac{\delta f}{f})(u-c_{x})
focal length error
Δu=δcx\Delta u=\delta c_{x}
principal point error
δX
a lateral position error of the camera or the point, in metres.
δθ
a rotation error about the vertical axis (for a horizontal shift), in radians; about 0.01745 per degree.
δf / f
the relative focal length error.
δc_x
the principal point error in pixels.

The difference in depth dependence is the key diagnostic: translation errors shrink with depth (1/Z), while rotation and principal-point errors are the same at every depth, and focal-length errors grow with the distance from the image centre. Observing points at several depths and positions separates them.

From pixel error to position error

A pixel error e at depth Z corresponds to a lateral position error

position error of a pixel error
δX≈Z⋅ef\delta X\approx \frac{Z\cdot e}{f}

so at 10 m with f = 600 pixels, one pixel is 1.7 cm; at 40 m it is 6.7 cm. Along the viewing ray the position is not determined by one image at all (Lesson P3-T4).

Statistics: RMS and robust measures

One outlier dominates the RMS, since errors are squared. The median of the error magnitudes (or the median absolute deviation) is insensitive to a few mismatches, and is usually reported next to the RMS. For rejecting outliers, a rule such as “discard points whose error exceeds 3 times the median” or a robust loss (Huber, Cauchy) is used in the fit itself.

A worked example: size of the error, and what to blame

A camera with f = 600 pixels is calibrated against a board. The residual lengths for eight corners are 0.3, 0.2, 0.4, 0.1, 0.3, 0.2, 5, 0.4 pixels (one of them, 5, is far larger than the rest).

summaries of the example residuals
RMS(all eight)=1.79 px\mathrm{RMS}\text{(all eight)}=1.79\,\mathrm{px}
median=0.30 px\text{median}=0.30\,\mathrm{px}
RMS without the outlier=0.29 px\mathrm{RMS}\,\text{without the outlier}=0.29\,\mathrm{px}

The RMS of 1.79 pixels suggests a poor calibration. The median of 0.30 pixels, and the RMS of 0.29 pixels without the single outlier, show that the model is good and one corner was mismatched (for example the detector found the wrong corner of a repeated pattern). Checking the one residual vector, which is long and points in a direction unrelated to the others, confirms it. Reporting the RMS alone would have led to throwing away a good calibration.

Sizes of effects, in pixels

For the same camera, how large are some common errors?

ErrorAt 2 mAt 6 mPattern in the image
5 cm lateral translation error of the camera15.0 px5.0 pxShifts all points together, by less for far points
1° yaw error10.5 px10.5 pxThe same shift at every depth
2% focal length error (point 150 px from the centre)3.0 px3.0 pxZero at the centre, grows with the distance from it
3 px principal point error3.0 px3.0 pxThe same shift everywhere

A 5 cm sideways error of the camera mount, which is easy to make with a tape measure, is a 15-pixel error at 2 m but only 5 pixels at 6 m. A 1° yaw error is 10 pixels at any distance. If the residuals shrink with depth, suspect translation; if they are constant with depth, suspect rotation or the principal point; if they grow outwards from the centre, suspect focal length or distortion.

Turning the error into centimetres

An RMS error of 0.29 pixel is a position error of Z × 0.29 / 600 = 0.5 cm at 10 m. For a robot grasping at 0.5 m the same pixel error is 0.24 mm, below the resolution of most grippers. The same calibration can be excellent for one task and inadequate for another. The requirement comes from the task, not from the score.

Engineering implications and limitations

Report more than the RMS. Give the RMS, the median, the maximum and the count of rejected points, per image and overall. Plot the residual vectors: structure that the numbers hide shows up at once.

Use held-out data. A calibration evaluated on the images used to fit it is optimistic. Keep a few images out, or collect a separate validation set, and report the error there.

Make a low score believable. Check that the views vary in position, depth and tilt, that the target covers the image, and that the parameters are plausible. A low RMS from a degenerate set of views says little.

Reject mismatches before fitting. Use a robust estimator or an outlier gate. One wrong correspondence in a least-squares fit pulls the solution toward itself and distorts every other residual.

Monitor in operation. A reprojection check against a fixed target, or consistency between two cameras viewing the same scene, is a cheap and useful health signal. A step change in the error says that something moved.

Relate it to the task. Convert the pixel error into metres at the working distance (δX ≈ Z e / f) and compare it with the accuracy the task needs. That decides whether the calibration is good enough.

Common misconceptions

“Zero reprojection error means a perfect model.”

It means the model reproduces these observations. With too many free parameters, or too little data, it can reproduce noise too.

“A high RMS means the calibration is bad.”

A single mismatched point can inflate the RMS enormously. Look at the median and at the individual residuals before concluding.

“A pixel is a fixed distance.”

One pixel is Z/f metres, so the same pixel error means more centimetres at longer range.

“Rotation and translation errors look the same.”

They differ in depth dependence: translation errors fall as 1/Z, rotation errors do not depend on depth.

Engineering takeaways

  1. The reprojection error is the distance between an observed pixel and the model’s prediction; its RMS scores a calibration or a pose estimate.
  2. First-order shifts: f δX/Z (translation), f δθ (rotation), (δf/f)(u − cx) (focal length), δc_x (principal point). Their depth and position dependence separates them.
  3. A pixel error e at depth Z is a lateral error of Z e / f: the same pixel is more centimetres at range.
  4. Outliers dominate the RMS; report the median and inspect residuals, and use robust fitting.
  5. A low score is necessary, not sufficient: check view diversity, plausibility of parameters and held-out error, and compare with the task’s accuracy requirement.

References and further reading

The residuals and the table values were computed for this lesson from the formulas shown.

  • Hartley, R. and Zisserman, A. (2004). Multiple View Geometry in Computer Vision, 2nd ed. Cambridge University Press: reprojection error as the geometric cost; bundle adjustment.
  • Triggs, B., McLauchlan, P., Hartley, R. and Fitzgibbon, A. (2000). Bundle adjustment: a modern synthesis. In Vision Algorithms: Theory and Practice, Springer.
  • Zhang, Z. (2000). A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(11): reprojection error as the criterion and quality measure.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

Question 1

A calibration gives residual lengths (pixels) 0.2, 0.3, 0.1, 0.2, 0.3 and 6.0. What are the RMS and the median, and what does the difference tell you?

Explanation

RMS = √((0.04 + 0.09 + 0.01 + 0.04 + 0.09 + 36)/6) = √(36.27/6) = 2.46 px. The median is (0.2 + 0.3)/2 = 0.25 px. The large gap says that one mismatched point dominates the RMS, while the model fits the other points well.

Question 2

A camera with f = 800 pixels shows a constant 8-pixel shift of all points, independent of their depth. What are the likely causes and how large a yaw error would that be?

Explanation

A constant shift at all depths points to a rotation error or a principal point error (not translation, which would fall as 1/Z). As a rotation, δθ ≈ 8/800 = 0.01 rad = 0.57°.

Question 3

An object is at 20 m and the camera has f = 1000 pixels. How large a position error does a one-pixel reprojection error imply, and is that adequate for passing 0.5 m from an obstacle?

Explanation

δX ≈ Z e / f = 20 × 1 / 1000 = 0.02 m = 2 cm. That is small compared with a 0.5 m clearance, so the pixel error is not the limiting factor; depth along the ray and the pose of the robot will matter more.

What’s next

That completes Module 3. Next: finding things in images that can be recognised again from another view: corners, edges and local features.

Practice this lesson

Investigate the Effect of Camera Parameters

Predict and explain how focal length, principal point and pose errors move projected points.

Next lesson

P4-T1 — Corners, Edges and Local Features

Explain why corners make good features and compute a corner response.

Nothing below is required to finish this lesson.