Skip to the lesson
FixTheRobot Explore Robotics / Perception / P3 Geometry and Camera Perception / P3-T1

Perception · Theory · P3

P3-T1 — Camera Models and Pinhole Projection

A camera divides by depth. That one operation explains perspective, and why one image cannot tell you how far away something is.

  • Lesson 1 of 6
  • Module 3: Geometry and Camera Perception
  • Theory
  • About 10 minutes to read

Every robot camera, however complex its lens, is modelled to first order as a pinhole: light travels in straight lines through a single point and lands on an image plane. From this model comes the projection equation that tells you which pixel a 3D point lands on, and it is the equation in nearly every geometric vision algorithm. This lesson derives it from similar triangles, introduces the intrinsic matrix, converts between millimetres and pixels, gives the field of view, and lists what the model does not capture.

Learning objectives

After this lesson, you should be able to:

  • Derive the pinhole projection from similar triangles and write it with the focal length and principal point.
  • Write the intrinsic matrix K and project a 3D point to a pixel.
  • Convert a lens focal length in millimetres to pixels and compute the field of view.
  • State what the pinhole model ignores and what each omission does in practice.

The idea: straight rays through one point

Punch a small hole in one wall of a box and put a sheet of paper on the opposite wall. Light from each point of the scene passes through the hole in a straight line and lands on one point of the paper, producing an upside-down picture. That is a pinhole camera. A real camera uses a lens to collect more light, but for a lens that is in focus, the geometry is close enough to a pinhole for most robotics.

The pinhole camera and similar trianglesA side view. A tree-like object of height Y at distance Z on the left, a pinhole at the origin, and an image plane to the right at distance f. A ray from the top of the object through the pinhole lands below the optical axis on the image plane at height y equal to minus f times Y over Z. Two similar triangles are marked.optical axis (Z) Yobject pinhole image plane y Zf y / f = Y / Z (the image is upside down; the sign is dropped by convention)
Similar triangles: the object height Y at distance Z and the image height y at distance f have the same ratio, so y = f · Y / Z.

Two facts follow directly from the figure.

  • Size is inversely proportional to distance. An object twice as far away appears half as large. This is perspective: parallel lines that recede (the edges of a road) converge in the image.
  • Depth is lost. Every point along the same ray, whether 1 m or 100 m away, lands on the same pixel. One image alone says which direction something is in, not how far away it is (Lesson P3-T4).

Projection divides by depth. Dividing destroys the depth, and that cannot be undone from the image alone.

The two numbers that fix the scale are the focal length f (the distance from the pinhole to the image plane, expressed in pixels) and the principal point (cx, cy), the pixel where the optical axis meets the image. A longer focal length magnifies and narrows the view; a shorter one widens it.

The mathematics: projection and the intrinsic matrix

Use the camera frame of Lesson P1-T4: origin at the pinhole, X to the right, Y down, Z forward along the optical axis. A point (X, Y, Z) with Z > 0 lands on the pixel

pinhole projection
u=fx⋅XZ+cxu=\frac{f_{x}\cdot X}{Z}+c_{x}
v=fy⋅YZ+cyv=\frac{f_{y}\cdot Y}{Z}+c_{y}
(X, Y, Z)
the point in the camera frame, in metres. Z is the depth along the optical axis, not the Euclidean distance.
fx, fy
focal lengths in pixels along the two image axes. For square pixels fx = fy. They say how many pixels one unit of X/Z covers.
(cx, cy)
the principal point in pixels, near the image centre in most cameras but not exactly at it.
(u, v)
the pixel column and row. The origin is the top left corner, v increases downward, matching Y.

The same in matrix form, with homogeneous coordinates (an extra 1 appended), shows how the division by depth arises:

intrinsic matrix K
[u′][u^{\prime}]
[fx 0 cx][X][f_{x}\,0\,c_{x}][X]
[v′w]=[0fycy001][YZ]\begin{bmatrix}v^{\prime} \\ w\end{bmatrix}=\begin{bmatrix}0 & f_{y} & c_{y} \\ 0 & 0 & 1\end{bmatrix}\begin{bmatrix}Y \\ Z\end{bmatrix}
u=u′w, v=v′w, with w=Zu=\frac{u^{\prime}}{w},\,v=\frac{v^{\prime}}{w},\,\text{with}\,w=Z

The 3 × 3 matrix is the intrinsic matrix K. The third row copies Z into the scale w; dividing by it is the perspective division. The point of the matrix form is composition: with the extrinsics of Lesson P1-T4 a world point projects as K [R | t] X.

From millimetres to pixels

A lens datasheet gives the focal length in millimetres and a sensor gives the pixel pitch p (the size of one pixel). The focal length in pixels is

focal length in pixels
fpx=fmmpf_{\text{px}}=\frac{f_{\text{mm}}}{p}
fpx=fmm⋅WpxWsensorf_{\text{px}}=\frac{f_{\text{mm}}\cdot W_{\text{px}}}{W_{\text{sensor}}}

where Wpx is the image width in pixels and Wsensor the sensor width in mm (if the image is a cropped or resized version of the sensor, use the actual used width). Resizing an image by a factor s multiplies fx, fy, cx and cy by s; cropping subtracts the crop offset from cx, cy.

Field of view

The half-width of the image at distance f is W/2 pixels, so

field of view
HFOV=2⋅arctan⁡(W2 fx)\mathrm{HFOV}=2\cdot \arctan (\frac{W}{2\,f_{x}})
VFOV=2⋅arctan⁡(H2 fy)\mathrm{VFOV}=2\cdot \arctan (\frac{H}{2\,f_{y}})

A larger focal length gives a smaller angle: the picture is zoomed in.

What the model leaves out

Real effectWhat it doesWhere it is handled
Lens distortionStraight lines bend, more towards the image border (barrel or pincushion)Distortion coefficients in calibration (Lesson P3-T2)
Focus and depth of fieldOnly a range of distances is sharp; the rest is blurredLens choice, aperture and focus setting
Rolling shutterRows are exposed at different times; moving scenes are skewedGlobal-shutter sensors or a motion model
Image sensor modelNon-square pixels, skew between axes (usually negligible)Separate fx and fy; skew term in K
LightVignetting, noise, saturationPreprocessing (Lesson P2-T6)

A worked example: from a lens to a pixel

A robot camera has a lens of focal length 4 mm on a sensor whose pixel pitch is 3 µm, giving a 1280 × 720 image.

Step 1: the focal length in pixels

focal length in pixels
fpx=4 mm3 μm=4000 μm3 μm=1333 pixelsf_{\text{px}}=\frac{4\,\mathrm{mm}}{3\,\mu m}=\frac{4000\,\mu m}{3\,\mu m}=1333\,\text{pixels}

Step 2: the field of view

field of view of the example camera
HFOV=2 arctan⁡(12802×1333)=51.3∘\mathrm{HFOV}=2\,\arctan (\frac{1280}{2\times 1333})=51.3^{\circ}
VFOV=2 arctan⁡(7202×1333)=30.2∘\mathrm{VFOV}=2\,\arctan (\frac{720}{2\times 1333})=30.2^{\circ}

Step 3: project a point

Take a simpler camera for the arithmetic: fx = fy = 600, (cx, cy) = (320, 240), a 640 × 480 image. A point is 0.5 m to the right, 0.2 m above the optical axis and 4 m ahead: (X, Y, Z) = (0.5, −0.2, 4) (Y is negative because up is negative Y).

projection of the point
u=600×0.54+320=75+320=395u=\frac{600\times 0.5}{4}+320=75+320=395
v=600×(−0.2)4+240=−30+240=210v=\frac{600\times (-0.2)}{4}+240=-30+240=210

The point lands at pixel (395, 210): 75 pixels right of the principal point and 30 pixels above it. Move the point to 8 m and the offsets halve, to (357.5, 225.0): the point converges towards the principal point as it recedes. At infinity it would be exactly at (cx, cy).

Which points are visible?

A point is visible if Z > 0 and the pixel is inside the image: 0 ≤ u < 640 and 0 ≤ v < 480 here. The formula returns a number for a point behind the camera too, as a mirrored image; it must be rejected by the test on Z. A point with X/Z larger than 320/600 = 0.53 (about 28° from the axis) falls off the right edge. These are the two tests in Lab P3-L1.

Engineering implications and limitations

Know the units of f. A datasheet’s 4 mm and a calibration’s 1333 are the same quantity in different units. Mixing them silently is a classic bug; keep a comment with the unit wherever f appears.

Track what happened to the image. Every crop, resize or rotation changes K. Calibrating at 1920 × 1080 and running at 960 × 540 requires halving f and c; cropping a region of interest moves c. A pipeline that resizes the image without updating K produces correct-looking pictures and wrong geometry.

Choose the focal length from the task. The pixels-on-target calculation (Lesson P2-T1) and the field-of-view requirement together pick the lens. A wide angle (small f) sees more and resolves less; a telephoto does the reverse. If you need both, you need two cameras or a sensor with more pixels.

The depth in the formula is Z, not the range. Z is measured along the optical axis. A point off to the side at the same Euclidean distance has a smaller Z. Depth cameras report Z (the distance along the optical axis) or the range depending on the driver; read the documentation.

The model is a first approximation. For a narrow-field camera it is nearly exact after distortion correction; for a fisheye or 360° camera a pinhole model does not exist and other models (equidistant, spherical) are used.

Common misconceptions

“The focal length is the number printed on the lens, and that is all I need.”

The millimetre value must be combined with the pixel pitch (or the sensor width) to give pixels, and it changes if the image is resized or cropped.

“A camera measures distance.”

It measures direction. Distance comes from depth sensors, from known object size, or from motion and stereo.

“The principal point is the centre of the image.”

It is near the centre, but can be several pixels off, and it moves when the image is cropped. Calibration finds it.

“Projection is a linear map.”

It is linear in homogeneous coordinates; in ordinary coordinates the division by Z makes it non-linear.

Engineering takeaways

  1. The pinhole model gives u = fx X / Z + cx and v = fy Y / Z + cy: size falls as 1/Z, and depth is lost.
  2. K collects fx, fy, cx, cy; the perspective division by Z is why the map is non-linear in ordinary coordinates.
  3. fpx = fmm / pixel pitch; HFOV = 2 atan(W / (2 fx)); resizing scales K, cropping shifts c.
  4. Check visibility with Z > 0 and the image bounds; the formula mirrors points behind the camera.
  5. The pinhole model omits distortion, focus, rolling shutter and noise; calibrate and handle them where they matter.

References and further reading

Standard textbooks; the lens and pixel numbers in the example are illustrative.

  • Hartley, R. and Zisserman, A. (2004). Multiple View Geometry in Computer Vision, 2nd ed. Cambridge University Press: chapter 6, camera models, and the intrinsic matrix.
  • Szeliski, R. (2022). Computer Vision: Algorithms and Applications, 2nd ed. Springer: chapter 2, image formation and the pinhole camera.
  • Forsyth, D. A. and Ponce, J. (2012). Computer Vision: A Modern Approach, 2nd ed. Pearson: perspective projection and cameras.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

Question 1

A camera has fx = 500 pixels and cx = 320. Where does a point with X = 1 m and Z = 5 m land, and where when Z = 10 m?

Explanation

u = 500 × 1/5 + 320 = 420 for Z = 5 m, and u = 500 × 1/10 + 320 = 370 for Z = 10 m. The offset from the principal point halves when the depth doubles; at infinity the point would be at 320.

Question 2

An image calibrated at 1280 × 720 with fx = 900 and cx = 640 is downsampled to 640 × 360 before processing. What are the new fx and cx, and what happens if you forget to change them?

Explanation

Both halve: fx = 450 and cx = 320. If you keep 900 and 640, projected points and back-projected rays are wrong by a factor of two in their offsets from the centre, and distances computed from pixels are twice too large.

Question 3

Why can the projection formula return a plausible pixel for a point behind the camera, and how do you prevent it from being used?

Explanation

With Z negative, X/Z and Y/Z simply flip sign, giving a pixel on the opposite side of the principal point, as if the point were mirrored in front of the camera. Test Z > 0 (and the image bounds) before treating the result as a valid observation.

What’s next

Next: calibration. How the intrinsic and extrinsic parameters are estimated and what the distortion terms add.

Practice this lesson

Project 3D Points into an Image

Implement pinhole projection and use it to place known 3D objects in an image.

Next lesson

P3-T2 — Intrinsic and Extrinsic Calibration

Distinguish intrinsic and extrinsic parameters and describe how they are estimated.

Nothing below is required to finish this lesson.