9.3 — Coordinate Frames and Camera Calibration

Learning objectives

After this lesson, you should be able to:

• Transform a point between camera and robot frames.
• Project and back-project with a pinhole camera.
• Compute the sideways error of a small orientation error.
• Explain why a moved camera breaks a learned policy.

The one-sentence idea

Every number needs a frame; every frame needs a calibrated transform.

Small angular errors become large position errors.

Where is it, in whose coordinates?

“The cup is at (0.3, 0.1, 0.0)” means nothing until you say measured from where, with which axes. A camera sees the world in its own frame; the robot arm moves in the base frame; a gripper has its own frame. Everything a robot does with vision needs the transformations between them (you met frames in the kinematics prerequisites). If the transformation between the camera and the robot base is wrong by a few centimetres, a grasp that is perfect in the image misses in the world. This lesson shows how such errors arise, how large they get and how calibration fixes them.

Moving a point between frames

A rigid transformation is a rotation plus a translation. In the common 4×4 homogeneous form it can be composed by matrix multiplication. Suppose the camera looks down at the table from above, offset from the robot base. A point seen by the camera becomes a point in the base frame:

import numpy as np

def transform(rotation_deg_about_z, translation):
    a = np.radians(rotation_deg_about_z)
    T = np.eye(4)
    T[:2, :2] = [[np.cos(a), -np.sin(a)], [np.sin(a), np.cos(a)]]
    T[:3, 3] = translation
    return T

base_from_camera = transform(90, [0.5, 0.0, 0.8])             # camera pose in the base frame: turned 90 degrees, 0.5 m forward, 0.8 m up
point_in_camera = np.array([0.1, 0.2, -0.8, 1.0])             # a cup seen by the camera (homogeneous point)

point_in_base = base_from_camera @ point_in_camera
print("cup in the camera frame:", point_in_camera[:3].tolist())
print("cup in the robot base frame:", np.round(point_in_base[:3], 3).tolist())
print("going back gives the original:", bool(np.allclose(np.linalg.inv(base_from_camera) @ point_in_base, point_in_camera)))

cup in the camera frame: [0.1, 0.2, -0.8]
cup in the robot base frame: [0.3, 0.1, 0.0]
going back gives the original: True

Toy model. A camera pose invented for the example. The pose of a real camera comes from calibration.

Name transformations as “base_from_camera” (it maps camera coordinates to base coordinates): the name tells you which way a point travels and prevents the classic mistake of using the inverse.

From pixels to rays: the camera model

A pinhole camera projects a 3D point (X, Y, Z) in the camera frame to pixel coordinates u = f·X/Z + c_x, v = f·Y/Z + c_y. The numbers f (focal length in pixels) and c_x, c_y (the image centre) are the intrinsic parameters. A single pixel does not give a 3D point, only a ray: you need depth to complete it.

import numpy as np

f, cx, cy = 600.0, 320.0, 240.0                                # intrinsic parameters (pixels)

def project(P):
    X, Y, Z = P
    return np.array([f * X / Z + cx, f * Y / Z + cy])

def back_project(u, v, Z):
    return np.array([(u - cx) * Z / f, (v - cy) * Z / f, Z])

P = np.array([0.10, -0.05, 0.80])
pixel = project(P)
print("pixel:", np.round(pixel, 1).tolist())
print("back-projected with the right depth:", np.round(back_project(*pixel, 0.80), 3).tolist())
print("back-projected with 5 cm too much depth:", np.round(back_project(*pixel, 0.85), 3).tolist())

pixel: [395.0, 202.5]
back-projected with the right depth: [0.1, -0.05, 0.8]
back-projected with 5 cm too much depth: [0.106, -0.053, 0.85]

A depth error of 5 cm moves the recovered point sideways too, by about 0.6 cm here. Calibration is what supplies f, c_x, c_y and lens distortion. A classic method (Zhang) has the camera observe a planar pattern in at least two different orientations (the motion need not be known), then solves in closed form and refines by maximum likelihood, including radial lens distortion.^[1]

How big does a calibration error get?

Small angular errors become large position errors at a distance. If the camera's orientation is off by 1 degree, a point at distance d is mislocated sideways by about d × tan(1°):

import numpy as np

for degrees in (0.5, 1.0, 2.0):
    for d in (0.5, 1.0, 2.0):
        error_cm = d * np.tan(np.radians(degrees)) * 100
        print(f"orientation error {degrees} deg, object {d} m away -> {error_cm:4.1f} cm sideways error")

orientation error 0.5 deg, object 0.5 m away ->  0.4 cm sideways error
orientation error 0.5 deg, object 1.0 m away ->  0.9 cm sideways error
orientation error 0.5 deg, object 2.0 m away ->  1.7 cm sideways error
orientation error 1.0 deg, object 0.5 m away ->  0.9 cm sideways error
orientation error 1.0 deg, object 1.0 m away ->  1.7 cm sideways error
orientation error 1.0 deg, object 2.0 m away ->  3.5 cm sideways error
orientation error 2.0 deg, object 0.5 m away ->  1.7 cm sideways error
orientation error 2.0 deg, object 1.0 m away ->  3.5 cm sideways error
orientation error 2.0 deg, object 2.0 m away ->  7.0 cm sideways error

A 1-degree error puts a point 1 m away 1.7 cm off, enough to miss a small object. This is why calibration quality, camera mounting stiffness (a camera that is bumped loses its calibration) and checking the calibration regularly matter more for learned policies than they might seem: the policy learned the geometry of its training setup.

Why frames matter for VLAs in particular

A VLA trained on a dataset implicitly learned that dataset's camera placement and action frame. Move the camera, or change the axes of the action (lesson 4.1), and the same pixels now mean different motions, with no error message. Pooled datasets, like the 22-embodiment Open X-Embodiment data,^[2] make these differences explicit work for the dataset builders. For you: record the camera pose with every dataset, check it at deployment, and treat a changed camera as a changed robot.

Common misconceptions

Engineering takeaways

1. Name transforms a_from_b.
2. Pinhole: u = fX/Z + cx.
3. 1° at 1 m is 1.7 cm.
4. Treat a moved camera as a changed robot.

Say which frame, every time.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

Next: joint limits and collision checking.

Back to the VLA roadmap