P3-T3 — Image Coordinates Versus Camera Coordinates

Learning objectives

After this lesson, you should be able to:

• Convert a pixel to normalised image coordinates and to a ray direction in the camera frame.
• Compute the bearing angles of a pixel and explain why equal pixel steps are not equal angles.
• Back-project a pixel with a known depth, and intersect a ray with a ground plane at a known camera height and pitch.
• Estimate how a one-pixel error changes the ground-plane distance and say where the method becomes unreliable.

The idea: a pixel is a ray

Three coordinate systems appear when you work with a camera:

• Pixel coordinates (u, v): column and row, in pixels, origin at the top left. This is where detections and features live.
• Normalised image coordinates (x_n, y_n): the position on an imaginary image plane one unit in front of the pinhole, with the origin at the principal point. No pixels, no focal length. Both numbers are the tangent of an angle.
• Camera-frame coordinates (X, Y, Z): metric 3D positions with the origin at the pinhole.

Normalised coordinates are the bridge: x_n = X/Z and y_n = Y/Z. A pixel fixes only these two ratios, so it fixes the direction of the point from the camera, which is a ray, and leaves the distance along the ray free.

A pixel gives a direction. To get a position add one more constraint: a measured depth, a known size, or a surface that the ray must hit.

The ground plane trick

A wheeled robot drives on a floor, and anything it needs to avoid stands on the floor. If the camera height and pitch are known, every pixel that images the floor corresponds to exactly one floor point, because the ray meets the floor once. That is a depth sensor from a single image, valid as long as the pixel really shows the floor and the floor is flat and level.

The mathematics: pixel, ray, point

Pixel to normalised coordinates and ray

x_n, y_n

normalised image coordinates. For a pixel on the optical axis both are 0. They equal tan of the horizontal and vertical angles from the optical axis.

r

a direction (not unit length) from the pinhole towards the scene. The scene point is a multiple of it: (X, Y, Z) = Z · r.

Z

the depth along the optical axis, the missing information.

The bearing angles of a pixel are θ_h = atan(x_n) and θ_v = atan(y_n). Note the atan: pixel offsets are proportional to the tangent of the angle, so equal pixel steps are not equal angles. At the border of a 90° field of view, a one-pixel step covers about half the angle that it does at the centre.

Back-projection with a depth

If the depth Z is known (from a depth sensor, Lesson P3-T5):

Ground-plane intersection

Robot frame: x forward, y left, z up. The camera is at height h and pitched down by α (no roll, no yaw offset). The camera axes in the robot frame are: forward (optical axis) (cos α, 0, −sin α), right (0, −1, 0), image-down (−sin α, 0, −cos α). The ray of the pixel in the robot frame is

The camera is at height h, so the ray reaches the floor (z = 0) after a parameter t with h + t R_z = 0, which requires R_z < 0:

The forward distance can be written in terms of the angle below the horizon, the depression angle δ = α + atan(y_n):

α

the camera pitch below the horizontal.

δ

the angle of the ray below the horizontal: pitch plus the pixel’s vertical angle.

h

the camera height above the floor, in metres.

x_ground, y_ground

the floor point in the robot frame, relative to the point on the floor below the camera.

The ray only reaches the floor if δ > 0, i.e. the pixel is below the horizon line. Pixels above the horizon see the sky or distant objects, not the floor, and the formula gives a negative or infinite distance for them.

Sensitivity

Differentiating x_ground = h/tan δ with respect to δ gives, for small changes,

The error grows with the square of the distance: the floor near the horizon is compressed into a few rows of pixels, so a single pixel row covers many metres.

A worked example: how far is the box?

A robot camera is mounted at h = 1.2 m, pitched down by 10°. It has f_x = f_y = 600 pixels and principal point (320, 240). A detector reports that the bottom edge of a box, where it touches the floor, is at pixel (u, v) = (420, 400).

1. Normalised coordinates: x_n = (420 − 320) / 600 = 0.1667, y_n = (400 − 240) / 600 = 0.2667.
2. Depression angle: δ = 10° + atan(0.2667) = 10° + 14.93° = 24.93°.
3. Forward distance: 1.2 / tan(24.93°) = 1.2 / 0.4649 = 2.58 m. (The longer formula gives the same: t = 2.751, x = t (cos α − y_n sin α) = 2.58 m.)
4. Sideways position: y = −t x_n = −2.751 × 0.1667 = −0.46 m, i.e. 0.46 m to the right of the robot’s centreline.

So the box is 2.58 m ahead and 0.46 m to the right.

Sensitivity to one pixel

Now see how much a one-pixel error matters at different rows. Using the same camera:

Close to the robot (row 400) one pixel is a centimetre or two; further out (row 260, 5.7 m ahead) one pixel is about 5 cm, and it grows rapidly beyond: the horizon of this camera is at row 134, and at row 140, only 6 rows below it, the floor point is already 128 m away. A detector whose box bottom is uncertain by two or three pixels cannot give a useful distance at long range.

When the assumption fails

The method treats the pixel as a floor point. If the detector’s box bottom is wrong (the top of an object, a shadow, a reflection), or the floor is a ramp, or the camera pitch is off by one degree, the answer is wrong, and it is wrong with confidence. A pitch error of 1° changes δ by 1° everywhere: at δ = 25° that moves the distance from 2.58 m to 2.47 m or 2.70 m, a change of 12 cm or more. Lab P3-L2 showed that a rotation error appears as a constant pixel shift; here it becomes a distance error.

Engineering implications and limitations

State your frame and axes. A pixel-to-ground function has four conventions to fix: the pixel origin, the camera axes, the robot axes and the sign of the pitch. Write them as comments and test with a point you can measure with a tape.

Calibrate height and pitch, not just intrinsics. The ground-plane distance depends directly on the camera height and pitch, which are extrinsics. Measure them after mounting, and recalibrate if the bracket is touched. The pitch can be estimated from the horizon or from a target on the floor.

Choose the right pixel. The ground point is where the object touches the floor: the bottom of the box, the base of a pole, the wheel contact of a car. The centre of a bounding box is not on the floor; use the bottom edge, and check that it is not occluded or cut by the image border.

Gate by distance and by row. Reject or down-weight ground-plane distances beyond the range at which one pixel is more than the accuracy you need, and reject pixels at or above the horizon. Report the uncertainty from the sensitivity formula.

Know the flat-floor assumption. Ramps, kerbs, uneven ground and robot pitch under acceleration violate it. Use IMU pitch to correct the camera pitch each frame, or use depth sensing where the floor is not flat.

Prefer a measured depth when you have one. A depth camera or LiDAR gives distance directly (Module 5 and Lesson P3-T5). The ground-plane trick is cheap and useful as a check on them or when none is available.

Common misconceptions

Engineering takeaways

1. Pixel → normalised coordinates (u − c_x)/f_x, (v − c_y)/f_y → ray (x_n, y_n, 1); a pixel fixes a direction, not a distance.
2. Bearing angles are atan(x_n) and atan(y_n); equal pixel steps are not equal angles.
3. With a depth Z the point is (x_n Z, y_n Z, Z); with a flat floor, camera height h and pitch α the ground distance is h / tan(α + atan y_n).
4. The ground-plane distance error from one pixel grows with distance squared: x²e/(h f).
5. Fix the conventions, calibrate height and pitch, use the contact point, gate by range, and prefer a measured depth when available.

References and further reading

The worked numbers were computed for this lesson from the formulas shown.

• Hartley, R. and Zisserman, A. (2004). Multiple View Geometry in Computer Vision, 2nd ed. Cambridge University Press: back-projection of points to rays.
• Szeliski, R. (2022). Computer Vision: Algorithms and Applications, 2nd ed. Springer: camera geometry and the use of scene constraints.
• Siegwart, R., Nourbakhsh, I. R. and Scaramuzza, D. (2011). Introduction to Autonomous Mobile Robots, 2nd ed. MIT Press: vision-based obstacle distance from a ground-plane assumption.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: why one image cannot give depth, how size, motion and context constrain it, and how much those cues are worth.

Related content

Nothing below is required to finish this lesson.

• Before
  P3-T2: Intrinsic and Extrinsic Calibration

  Distinguish intrinsic and extrinsic parameters and describe how they are estimated.

• Lab
  P3-L1: Project 3D Points into an Image

  Implement pinhole projection and use it to place known 3D objects in an image.