Perception · Theory · P3
P3-T4 — Perspective and Depth Ambiguity
Many different scenes make the same image. Monocular depth always comes from an assumption, and the assumption can be wrong.
Projection divides by depth, so the depth disappears from the image. A small object near the camera and a large object far away can produce identical pixels. Humans resolve this constantly with knowledge: we know how tall a person is, we see the floor recede, we move our heads. Robots can use the same cues, and each one is an assumption that can fail. This lesson sets out the ambiguity precisely, quantifies the most common size-based estimate and its error, and shows how motion supplies depth without assumptions.
Learning objectives
After this lesson, you should be able to:
- Explain why one image cannot determine depth and give an example of two scenes that produce the same image.
- Estimate distance from known object size and compute how errors in size and in pixel measurement propagate.
- Describe other monocular cues (ground plane, motion parallax, vanishing points) and what each assumes.
- State how a perception system should represent depth that comes from an assumption.
The idea: the ray has no length
In Lesson P3-T3 a pixel became a ray. Everything along that ray, near or far, projects to the same pixel. A single image therefore constrains a point to a line, never to a place.
The same holds for whole scenes: a miniature model close up and a full-size building far away, a painted backdrop and a real corridor. Film sets exploit exactly this.
Perspective is ambiguous by construction. Every monocular depth estimate is the projection plus an assumption about the world.
The cues, and what each one assumes
| Cue | How it gives depth | What it assumes | How it fails |
|---|---|---|---|
| Known size | Z = f · S / s: depth from the real size S and image size s | The object’s real size is known | Wrong class or size (child against adult, model against truck) |
| Ground plane | The floor point below an object has a unique depth (Lesson P3-T3) | Flat floor, known camera height and pitch | Slopes, kerbs, unknown pitch, occluded base |
| Texture and size gradient | Surfaces of uniform texture look finer with distance | Uniform texture on the surface | Patterns that are not uniform, painted textures |
| Vanishing points | Parallel lines converge; their convergence gives orientation | Man-made structure with parallel lines | Natural scenes, curved roads |
| Motion parallax | Near points move more than far ones as the camera moves | The camera motion is known; the scene is static | Moving objects; unknown or tiny motion |
| Learned priors | A network learned typical scene geometry | The scene resembles the training data | Out-of-distribution scenes: confident and wrong |
Stereo and depth cameras (Lesson P3-T5) avoid the ambiguity by measuring depth with a second view or an active signal.
The mathematics: size, error and parallax
Depth from known size
By similar triangles (Lesson P3-T1), an object of real height S at depth Z has image height s = f S/Z, so
- Z
- depth along the optical axis, in metres.
- f
- focal length in pixels.
- S
- the real size of the object (its height or width), in metres. This is the assumed quantity.
- s
- the measured size of the object in the image, in pixels.
Errors propagate in two ways. The relative error of Z equals the relative error of the assumed size plus the relative error of the pixel measurement:
(For independent errors combine them as the square root of the sum of squares.) The second term grows as the object gets smaller in the image: the same one-pixel error is a larger fraction of 20 pixels than of 200.
Depth from motion
If the camera moves sideways by a known distance B between two frames and a static point shifts by d pixels (its parallax or disparity), the geometry is that of a stereo pair with baseline B:
Depth now follows from measurement and a known motion, with no assumption about the object. The cost is that the scene must be static, the motion known (from odometry or an IMU) and large enough to produce a measurable shift: for the same disparity accuracy, depth error grows as Z² (Lab P3-L3, Stage 1). This is the principle behind structure from motion and visual SLAM (Localization & SLAM track).
What a robot should output
A depth that comes from an assumption should be reported with an uncertainty that includes the assumption, for example as a range of depths rather than a point (“between 6 and 8.5 m, assuming a person is between 1.2 and 1.9 m tall”), and with a flag that says which cue produced it. A planner that cannot see the difference between a measured and an assumed distance will treat them alike.
A worked example: how far away is the pedestrian?
A camera has f = 800 pixels. A pedestrian appears 160 pixels tall. Assuming an adult height of 1.7 m:
If the assumption is wrong
Suppose the person is a child of 1.2 m. The true depth is 800 × 1.2 / 160 = 6.0 m, so the estimate of 8.5 m is 42% too far (the ratio 1.7/1.2 = 1.42). The robot believes the child is 2.5 m further away than they are. At walking speed of 1.4 m/s that is about 1.8 seconds of lost reaction time. This is the cost of the assumption, and no amount of image processing detects it, because the image is identical.
Pixel error at long range
The same camera sees a person who appears only 20 pixels tall:
A two-pixel error in the box height, easily caused by a loose detector box, changes the distance by about 7 m (10%). At 160 pixels the same two pixels change it by only 0.1 m (1.3%). The estimate is useful nearby and nearly useless far away, for exactly the reason that the object becomes small in the image.
Depth from motion for comparison
The robot moves sideways by B = 0.2 m (from odometry) and a static point shifts by d = 12 pixels in an image with f = 600:
No assumption about the object appears here. If the disparity has an error of 0.5 pixel the distance changes by about 0.42 m (4.2%), and the error grows quickly for small disparities, which is why far objects need either a longer baseline or more motion. The price is that the point must be static between the frames and that the motion must be known; if the point moves too the parallax contains its motion as well.
Engineering implications and limitations
Label every depth with its source. “Measured by stereo”, “from the ground plane”, “from assumed height” are different quantities with different error models. Carry the cue and its uncertainty with the number, and let the consumer decide.
Use priors that are wide enough. A person-height prior of 1.7 m ± 0.1 m is false for children and for people far from the average. Use a distribution that reflects the real population, and pass the resulting range of depths.
Combine cues. Ground-plane distance and size-based distance fail differently: one fails on slopes, the other on unusual sizes. Agreement raises confidence; disagreement is a flag that an assumption is violated.
Do not trust a learned monocular depth network as a measurement. Networks that predict depth from one image are impressive and can be accurate in familiar scenes, but they learn priors, and in unfamiliar scenes they produce confident, plausible, wrong depth. Treat their output as a hypothesis unless it is validated against a metric sensor for your environment.
Use motion when you have it. A moving robot has free baselines. Tracking points across frames with odometry gives depth without size assumptions, at the price of static-scene and motion-accuracy requirements.
Do not hide the ambiguity from safety logic. A safety function should use a measured distance, or the nearest value in the uncertainty range, not the central estimate of an assumption.
Common misconceptions
“If two scenes look identical they are the same scene.”
Projection discards depth, so a family of different scenes produces the same image. Which one you are in depends on assumptions or other measurements.
“Size-based distance is accurate if the detector is.”
It is limited by the knowledge of the real size. A detector that is perfect on the pixels still gives a 40% range error for a child taken as an adult.
“A deep network can recover the true depth from one image.”
It can estimate plausible depth from priors learned on its training data. It does not remove the ambiguity, and it can be wrong without any sign.
“More pixels fix the ambiguity.”
Resolution improves the precision of angle measurements, not their depth content. A higher-resolution image of the same ray still has no distance.
Engineering takeaways
- A pixel is a ray; objects of different size at different distances can make identical images.
- Z = f S / s gives depth from known size, with δZ/Z ≈ δS/S + δs/s: wrong size assumptions and pixel errors at small image size both hurt.
- Ground-plane, texture, vanishing-point and learned cues each rest on an assumption that can fail.
- Motion parallax Z = f B / d gives depth without an assumption about the object, but needs a known baseline and a static scene.
- Report depth with its source and an uncertainty that includes the assumption, and let safety logic use the worst case.
References and further reading
The worked numbers were computed from the formulas shown; heights of 1.7 m and 1.2 m are illustrative.
- Hartley, R. and Zisserman, A. (2004). Multiple View Geometry in Computer Vision, 2nd ed. Cambridge University Press: the loss of depth in projection and the geometry of two views.
- Szeliski, R. (2022). Computer Vision: Algorithms and Applications, 2nd ed. Springer: stereo, structure from motion and monocular depth cues.
- Marr, D. (1982). Vision. W. H. Freeman: the classic discussion of why vision is an ill-posed inverse problem and what constraints resolve it.
Knowledge check
Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.
Question 1
A camera with f = 1000 pixels sees a 0.5 m wide box that spans 50 pixels. How far is it? If the real box is 0.6 m wide, what is the true distance, and by what percentage was the estimate wrong?
Explanation
The estimate is Z = 1000 × 0.5 / 50 = 10 m. The true distance is 1000 × 0.6 / 50 = 12 m. The estimate was 10/12 − 1 = −17% (too near), exactly the ratio of the assumed to the true size (0.5/0.6).
Question 2
Why does a one-pixel error in the height of a detected car matter more at 80 m than at 10 m for size-based distance?
Explanation
The error is relative: δZ/Z ≈ δs/s. At 80 m the car is about eight times smaller in the image than at 10 m, so one pixel is eight times larger as a fraction of its height, and the distance is uncertain by eight times as large a fraction.
Question 3
A robot moves 0.1 m sideways between two frames (f = 500 pixels). A static tree trunk shifts by 5 pixels. How far is it, and why would a walking pedestrian give a wrong answer with the same method?
Explanation
Z = 500 × 0.1 / 5 = 10 m. A moving pedestrian shifts in the image because of both the robot’s motion and their own, so the shift is not pure parallax from the baseline and the formula returns a wrong distance. The static-scene assumption is violated.
What’s next
Next: sensors that measure depth directly: stereo, structured light and time of flight, and how to use a depth image.
Related content
Nothing below is required to finish this lesson.
- BeforeP3-T3: Image Coordinates Versus Camera Coordinates
Convert between pixels, normalised image coordinates and camera-frame rays.
- LabP3-L3: Recover Geometry from Depth Observations
Back-project a depth image and measure distances, heights and widths.
Get notified when the next lesson and new labs are released.
One email per new lesson or lab.