Skip to the lesson
FixTheRobot Explore Robotics / Perception / P3 Geometry and Camera Perception / P3-T2

Perception · Theory · P3

P3-T2 — Intrinsic and Extrinsic Calibration

Intrinsics describe the camera. Extrinsics describe where it is. Calibration estimates both, and everything geometric depends on getting them right.

  • Lesson 2 of 6
  • Module 3: Geometry and Camera Perception
  • Theory
  • About 11 minutes to read

A robot that wants to turn pixels into positions needs numbers that nobody measured when the camera was built or bolted on: the exact focal length and image centre, the lens distortion and the pose of the camera on the robot. Estimating these is calibration. This lesson separates the two kinds of parameter, writes the distortion model, outlines how calibration is performed and checked, and lists what goes wrong in the field.

Learning objectives

After this lesson, you should be able to:

  • Distinguish intrinsic from extrinsic parameters and list the parameters of each.
  • Apply and invert the radial distortion model for a point, with a worked example.
  • Describe how a planar-target calibration works and how its quality is judged.
  • Name the events that invalidate a calibration and the symptoms.

The idea: two kinds of unknown

Two sets of numbers sit between a world point and its pixel.

The projection chain with intrinsics and extrinsicsFour boxes in a row joined by arrows. World point X_w. Camera frame point X_c after the extrinsics R and t. Normalised image coordinates after division by depth. Pixel (u, v) after intrinsics K and distortion. Under the first arrow the label extrinsics, six numbers, changes when the camera moves. Under the last arrow the label intrinsics and distortion, fixed for a given camera and lens.World pointX_w Camera frameX_c = R X_w + t Normalised(X/Z, Y/Z) Pixeldistortion + K → (u, v) extrinsics R, tsix numbers; change when the camera moves intrinsics K, distortion kfixed for a camera and lens (until it changes) Intrinsics describe the camera. Extrinsics describe where it is.
The projection chain. The extrinsics (left) depend on where the camera is; the intrinsics and distortion (right) belong to the camera and lens.
  • Extrinsic parameters are the pose of the camera in a reference frame: a rotation R and a translation t, six numbers in 3D. They change whenever the camera moves relative to the frame, and so they are needed per robot (camera on the robot) and, for a moving camera, per frame (from localization).
  • Intrinsic parameters belong to the camera and its lens: the focal lengths fx, fy, the principal point (cx, cy), possibly a skew, and the distortion coefficients. They stay constant as long as the optics and the sensor do not change.

Separate what changes from what does not. Calibrate the intrinsics once per camera and lens setting, and keep extrinsics as a separate, replaceable transform.

Why distortion exists

A real lens does not map straight lines to straight lines. Wide lenses bend lines near the border outwards (barrel distortion); some lenses bend them inwards (pincushion). A pinhole model with no distortion term puts points at the wrong pixel, by up to tens of pixels near the edge of a wide-angle lens, and the error grows with the distance from the image centre.

The mathematics: the full model and distortion

Extrinsics and intrinsics together

For a point Xw in the world frame, with the camera pose given as a rotation R and translation t that take world coordinates to camera coordinates:

the full projection
Xc=R Xw+tX_{c}=R\,X_{w}+t
xn=XcZc, yn=YcZcx_{n}=\frac{X_{c}}{Z_{c}},\,y_{n}=\frac{Y_{c}}{Z_{c}}
(u, v)=K⋅distort⁡(xn, yn)(u,\,v)=K\cdot \operatorname{distort}(x_{n},\,y_{n})
R, t
the extrinsics: a 3 × 3 rotation and a 3-vector that map world points into the camera frame. (Note the direction: this is TCW; the camera pose in the world is its inverse, as in Lab P3-L1.)
xn, yn
normalised image coordinates: the point projected onto a plane at unit distance, with no pixels and no lens yet.
distort(·)
the lens distortion, applied in normalised coordinates before K.
K
the intrinsic matrix with fx, fy, cx, cy as in Lesson P3-T1.

The radial distortion model

The common model (Brown–Conrady) replaces the normalised coordinates by a distorted version that depends on the squared distance r2 = xn2 + yn2 from the optical axis:

radial and tangential distortion
xd=xn(1+k1 r2+k2 r4+k3 r6)+2 p1 xn yn+p2(r2+2 xn2)x_{d}=x_{n}(1+k1\,r^{2}+k2\,r^{4}+k3\,r^{6})+2\,p1\,x_{n}\,y_{n}+p2(r^{2}+2\,x_{n}^{2})
yd=yn(1+k1 r2+k2 r4+k3 r6)+p1(r2+2 yn2)+2 p2 xn yny_{d}=y_{n}(1+k1\,r^{2}+k2\,r^{4}+k3\,r^{6})+p1(r^{2}+2\,y_{n}^{2})+2\,p2\,x_{n}\,y_{n}
k1, k2, k3
radial coefficients. A negative k1 gives barrel distortion (points pulled towards the centre), a positive k1 pincushion.
p1, p2
tangential coefficients, caused by a lens that is not exactly parallel to the sensor. Usually small.
r²
the squared distance of the undistorted normalised point from the optical axis. The effect grows as r², r⁴ and r⁶: it is tiny near the centre and large at the border.

Going the other way, from distorted pixels to undistorted ones, has no closed form. The usual approach is a fixed-point iteration: start with the distorted point as a first guess of the undistorted point, apply the forward model to get the error, correct, and repeat a few times. Libraries implement this and cache a lookup table for the whole image.

How calibration is done

The standard method images a planar target with known geometry (a checkerboard or a grid of dots) from many poses. For each image the corners are detected to sub-pixel accuracy, giving pairs of known 3D target points and measured pixels. An initial solution is found in closed form (Zhang’s method uses the homographies of the plane to the image) and then refined by minimising the reprojection error over all intrinsics, distortion coefficients and target poses at once:

calibration as least squares
min⁡ over(K, k, {Ri, ti}) of ∑i ∑j ∣∣xij−π(K, k, Ri, ti, Xj) ∣∣2\min \,\text{over}(K,\,k,\,\{R_{i},\,t_{i}\})\,\text{of}\,\sum _{i}\,\sum _{j}\,\lvert \rvert x_{ij}-\pi (K,\,k,\,R_{i},\,t_{i},\,X_{j})\,\lvert \rvert ^{2}

Here xij is the measured pixel of target point j in image i and π is the projection of the previous section. The target poses are nuisance parameters (extrinsics of the target in each image) that are estimated together with the intrinsics. With 15 to 30 well-spread views the intrinsics are determined to a fraction of a percent.

Other calibrations you will meet

  • Camera to robot (hand-eye or extrinsic) calibration: find the pose of the camera on the robot body or the arm, by observing a target while moving the robot and solving for the fixed transform.
  • Stereo calibration: find the relative pose between the two cameras of a stereo pair (the baseline and rotation), after each camera’s intrinsics.
  • Camera to LiDAR calibration: find the transform between a camera and a LiDAR, using a target visible to both (Lesson P8-T3).

A worked example: distorting and undoing one point

A lens has k1 = -0.25, k2 = 0.08 (and no tangential distortion). The camera has fx = 600, cx = 320. A point has normalised coordinates (0.6, 0.4).

Forward: where does it land?

distortion of the point
r2=0.62+0.42=0.520r^{2}=0.6^{2}+0.4^{2}=0.520
scale=1+(−0.25)(0.520)+(0.08)(0.520)2=0.8916\text{scale}=1+(-0.25)(0.520)+(0.08)(0.520)^{2}=0.8916
(xd, yd)=(0.5350, 0.3567)(x_{d},\,y_{d})=(0.5350,\,0.3567)

The point is pulled towards the centre by 10.8%. In pixels the column is u = 600 × 0.5350 + 320 = 641.0 instead of 600 × 0.6 + 320 = 680.0 without distortion, a difference of 39.0 pixels. A pipeline that ignores the distortion puts this point nearly 39 pixels from where it is, enough to miss the edge of a gripper target.

Backward: undoing the distortion by iteration

Given the distorted point (0.5350, 0.3567), start with the guess (x, y) = (xd, yd). Each iteration divides the distorted point by the scale computed at the current guess:

iteration 1
scale 0.9103→(0.5877, 0.3918)\text{scale}\,0.9103\to (0.5877,\,0.3918)
iteration 2
scale 0.8952→(0.5976, 0.3984)\text{scale}\,0.8952\to (0.5976,\,0.3984)
iteration 3
scale 0.8923→(0.5995, 0.3997)\text{scale}\,0.8923\to (0.5995,\,0.3997)
iteration 4
scale 0.8918→(0.5999, 0.3999)\text{scale}\,0.8918\to (0.5999,\,0.3999)
iteration 5
scale 0.8917→(0.6000, 0.4000)\text{scale}\,0.8917\to (0.6000,\,0.4000)

After a few iterations the guess settles at (0.6000, 0.4000), matching the original (0.6, 0.4) to four decimals. The iteration converges quickly for moderate distortion, and in practice 5 to 10 steps are used. This is the undistort step of every vision library.

What good calibration looks like

After calibration, the RMS reprojection error of the target corners should be small, well under a pixel for a good setup. A low RMS is necessary but not sufficient: it can be low if the views are all similar (not enough variety to determine the parameters) or if the model has enough freedom to absorb a bias. Check the parameters for plausibility (is the principal point near the centre? is the focal length near the datasheet value?), view the residual pattern for structure, and test on a held-out set of images that were not used in the fit.

Engineering implications and limitations

Calibrate with the setting you will use. Zoom, focus, resolution, region of interest and even temperature change the intrinsics. A lens that is refocused has a different focal length in pixels (focus breathing). Calibrate after you finalise the lens settings, and lock them (a drop of lens lock or a locked focus ring).

Cover the image. Distortion is largest at the border. A calibration whose target corners occupy only the centre does not determine the border behaviour, and the fitted model may be wrong exactly where it matters. Place the target in all regions, at several distances and tilts.

Keep the rig steady. Extrinsic calibrations drift when brackets loosen, after knocks, or with thermal expansion. Store the calibration with a date, the camera serial number and the lens settings, and provide a quick check (observe a known target and verify the reprojection error) at start-up or on a schedule.

Do not overfit the model. Using k3 or tangential terms for a narrow-field lens with little distortion gives parameters that oscillate and may behave wildly outside the calibrated region. Start with k1, k2 and add terms only if the residual shows structure and held-out error improves.

Distinguish the failures. Parameter errors have fingerprints (Lab P3-L2): a constant offset suggests the principal point or a rotation; an error that grows with distance from the centre suggests focal length or distortion; a scale error that depends on depth suggests the translation. Use these to decide what to recalibrate.

Automate the check. Many production systems run a self-check (a known pattern in the scene, a comparison between overlapping sensors) and refuse to run, or lower speed, when it fails.

Common misconceptions

“Calibration is a one-time job.”

It is valid for the camera, lens setting and mounting at the time. Knocks, temperature, refocusing and a replaced camera all invalidate part of it.

“A low reprojection error proves the calibration is right.”

It proves the model fits the data you used. Too few or too similar views, or a model with too much freedom, can give a low error and wrong parameters.

“Distortion only matters for fisheye lenses.”

Many ordinary wide-angle lenses produce several pixels of error near the border, enough to matter for accurate positions.

“Intrinsics and extrinsics are interchangeable parameters.”

They describe different things and change at different times. Mixing them in one lump makes it impossible to recalibrate only what has changed.

Engineering takeaways

  1. Intrinsics (fx, fy, cx, cy, distortion) belong to the camera and lens; extrinsics (R, t) say where the camera is.
  2. Radial distortion scales normalised coordinates by 1 + k1 r² + k2 r⁴; undistorting needs a fixed-point iteration.
  3. Planar-target calibration minimises reprojection error over intrinsics, distortion and target poses; coverage and variety of views decide its quality.
  4. Hand-eye, stereo and camera-LiDAR calibrations estimate other fixed transforms in the same way.
  5. Calibrate at the final settings, cover the image, keep the rig steady, store the calibration with metadata and check it routinely.

References and further reading

The distortion coefficients and points in the example are illustrative; the table values were computed for this lesson.

  • Zhang, Z. (2000). A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence 22(11): calibration from a planar target seen at several orientations.
  • Brown, D. C. (1971). Close-range camera calibration. Photogrammetric Engineering 37(8): the radial and tangential distortion model.
  • Hartley, R. and Zisserman, A. (2004). Multiple View Geometry in Computer Vision, 2nd ed. Cambridge University Press: camera calibration and parameter estimation.
  • OpenCV documentation, Camera Calibration and 3D Reconstruction: distortion model and the calibration functions as implemented.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

Question 1

Which of these change when the camera is bumped against a door frame, and which do not: focal length, principal point, rotation R, translation t?

Explanation

R and t change (the camera moved relative to the robot or world). The focal length and principal point stay the same unless the lens or sensor shifted inside the camera, which a hard knock can also do, so a check against a known target is needed.

Question 2

Why is a calibration made with the target only in the middle of the image risky for a wide-angle lens?

Explanation

Distortion is a function of the distance from the optical axis and grows as r², r⁴ and r⁶, so it is almost invisible at the centre. Data only from the centre cannot determine the distortion coefficients, and the fitted values may be far wrong at the border.

Question 3

A calibration reports an RMS reprojection error of 0.2 pixel but the principal point is 40 pixels from the image centre and the focal length is 30% from the datasheet value. What would you do?

Explanation

Distrust it. A low RMS can occur with too little variety of views or a model that absorbs error. Implausible parameters point to degenerate data (target views too similar, too few, or at too small a distance). Re-capture with the target covering the image and at varied tilts, and test on held-out images.

What’s next

Next: moving between pixels, normalised coordinates and rays in the camera frame, and using them to find where a ray meets the ground.

Practice this lesson

Investigate the Effect of Camera Parameters

Predict and explain how focal length, principal point and pose errors move projected points.

Next lesson

P3-T3 — Image Coordinates Versus Camera Coordinates

Convert between pixels, normalised image coordinates and camera-frame rays.

Nothing below is required to finish this lesson.