P2-T1 — Pixels, Image Dimensions and Color Representations

Learning objectives

After this lesson, you should be able to:

• Describe an image as an array indexed by row, column and channel, and state its dimensions, channels and bit depth.
• Compute the memory of an image and the size of a pixel’s footprint on a scene at a given distance.
• Explain integer overflow and gamma encoding, and why they break naive pixel arithmetic.
• Decide how many pixels an object needs to be detected, from the camera’s resolution and field of view.

The idea: a grid of measurements

A camera sensor is an array of tiny light detectors. Each one counts the light that arrives during the exposure and reports a number. Arranged in rows and columns, those numbers are the image. Nothing in the array says where an object is; the structure is only the neighbourhood: pixels next to each other came from nearby directions in the scene.

Four properties describe a digital image, and each one has consequences for a robot:

• Width and height (in pixels) set the spatial resolution: how much detail can be recorded.
• Channels are the numbers per pixel. One for a grey image, three for colour (red, green, blue), four for colour plus transparency, or one for a depth value.
• Bit depth is the number of bits per channel value. Eight bits give 256 levels (0 to 255); 10, 12 and 16 bits are common in industrial and scientific cameras.
• Coordinates: the origin is at the top left, the column index u increases to the right and the row index v increases downwards. In code the pixel is image[v][u]: row first.

A pixel is not a little square of the scene. It is one sample of the light arriving from one direction, and its position in the grid is the only geometry the image contains.

The row-first, downward-pointing convention is a source of bugs when images meet robot coordinates: in the robot frame x points forward and y left, so an image row increasing downward means “nearer to the robot” on a camera looking down at the floor. Lab P1-L3 had exactly this conversion.

The mathematics: size, resolution and quantisation

Memory

The memory of an uncompressed image is the number of values times the bytes per value:

W, H

width and height in pixels.

C

channels per pixel (1, 3 or 4).

B

bit depth per channel, so B/8 bytes per value.

A 1920 × 1080 colour image with 8 bits per channel needs 1920 × 1080 × 3 = 6 220 800 bytes, about 6.2 MB. At 30 frames per second that is 187 MB/s of raw data from one camera, which is why robots compress, crop or downsample early, and why the memory layout (rows first, interleaved channels) matters for speed.

What a pixel covers

If a camera with horizontal field of view θ has W pixels across, each pixel spans an angle of about θ/W (the instantaneous field of view). At distance d the width of the scene seen by the image and the width on the scene that one pixel covers are

θ

horizontal field of view of the camera, in degrees or radians.

d

distance from the camera to the plane of interest, in metres, measured along the optical axis.

W

pixels across the image.

footprint

the width in the scene that a single pixel represents. It grows in proportion to d.

An object of width w therefore covers about w/footprint pixels, and a detector needs enough pixels to see its shape. As a working rule, a recognisable object often needs on the order of 10 to 20 pixels across, and a measurement of its position to a tenth of its size needs more; the figure depends on the method and has to be checked.

Quantisation

With B bits there are 2^B levels. The step between adjacent levels in a range of brightness R is R/(2^B − 1), and the rounding error is at most half a step. With 8 bits there are 255 steps, so the dynamic range of a scene with a bright window and a dark corner (a ratio of 1000:1 in light) cannot be recorded in one exposure: either the window clips at 255 or the corner is crushed into the lowest levels.

Integer overflow

Pixel values are usually stored as unsigned 8-bit integers. Arithmetic on them wraps around: 200 + 100 is not 300 but 300 mod 256 = 44, and 50 − 100 is 206 (256 − 50). Averaging two pixels by (a + b)/2 in 8-bit arithmetic is wrong when a + b exceeds 255. The remedies are to convert to a wider type (or to floating point) before computing, and to clip at the end.

Gamma

Standard 8-bit images (sRGB) store values that are a non-linear function of the light intensity, roughly a power of about 1/2.2, so that the 256 levels are spread in a way that matches human vision. The consequence is that pixel values are not proportional to the amount of light: an 8-bit value of 128 is about 22% of full intensity, not 50%. Averaging pixel values, blurring, or adding two images in this encoded space is not the same as averaging or adding light. For measurements of brightness, converting to linear intensity first is the correct way; for many robotics tasks (finding a coloured block) the difference is small, and for others (photometric measurements, high dynamic range) it matters.

A worked example: can the camera see the pallet label?

A warehouse robot has a camera with W = 1280 pixels across and a horizontal field of view θ = 90°. A pallet label is 0.15 m wide and the robot wants to read its position and rough shape.

At 5 m the label covers about 19 pixels, which is enough to find its position; at 10 m only 10 pixels, which is marginal for shape and probably not enough for reading. The practical range of detection is therefore set by simple geometry before any algorithm is chosen. If a longer range is needed the options are a narrower field of view (more pixels per metre, less coverage), a higher-resolution sensor, or a second camera.

Why a mean of pixels can be wrong

Suppose a program averages two neighbouring pixels of values 180 and 200 in a uint8 array with (a + b) / 2. In 8-bit arithmetic 180 + 200 = 380 wraps to 124, and the average becomes 62 instead of 190. The result is a dark pixel in the middle of a bright region, which looks like noise or a bug in the filter, and it is neither: it is arithmetic. The same code with a 16-bit or floating-point intermediate gives 190. Numerical libraries differ in whether they wrap, clip (saturate) or promote, and the safe habit is to convert explicitly.

An image from the first lab, as numbers

The simulated camera in Lab P1-L3 is 40 × 28 pixels with three channels, so each image holds 40 × 28 × 3 = 3 360 numbers. The robot sits at pixel (20, 27) and one pixel is 5 cm on the floor, so the image covers 2 m across and 1.4 m ahead. Those two numbers (the pixel footprint and the origin) are the only geometry in the image; changing either changes every position the pipeline reports.

Engineering implications and limitations

Know the layout. Check whether your library stores images as height × width × channels or channels × height × width, as RGB or BGR (OpenCV’s default order is blue, green, red), and as 8-bit integers or floats in 0 to 1. A swapped channel order turns red objects blue and silently breaks colour thresholds.

Choose resolution for the task. More pixels cost bandwidth, memory and time, and may not help: if the object covers 30 pixels at the range you need, 4 times the resolution will not make detection easier but will make it 4 times slower. Compute the pixels-on-target from the geometry first.

Mind the exposure. Clipped pixels (255) and crushed pixels (0) carry no information. Count them. A histogram with a spike at 255 or at 0 is a warning that the stage after cannot work, whatever it is.

Do arithmetic in a wide type. Convert to float or a 16-bit integer before adding, averaging or differencing; clip and convert back at the end.

Keep the geometry attached. Cropping or resizing changes the mapping between pixels and the world. Whenever an image is cropped or scaled, the camera constants (principal point, focal length in pixels) change too (Lesson P3-T1).

Common misconceptions

Engineering takeaways

1. An image is an array of numbers indexed by row and column (row first, origin top left, rows increase downward) with channels and a bit depth.
2. Memory is W × H × C × B/8 bytes; raw video quickly reaches hundreds of megabytes per second.
3. A pixel’s footprint on the scene is 2 d tan(θ/2)/W: it grows with distance and sets the range at which an object has enough pixels to be found.
4. 8-bit arithmetic wraps around and pixel values are gamma-encoded: compute in a wider type and know the encoding.
5. Check layout (BGR or RGB, uint8 or float), exposure and resolution against the task before choosing algorithms.

References and further reading

Standard references; the camera numbers in the example are illustrative.

• Szeliski, R. (2022). Computer Vision: Algorithms and Applications, 2nd ed. Springer: image formation, sampling, quantisation and colour.
• Gonzalez, R. C. and Woods, R. E. (2018). Digital Image Processing, 4th ed. Pearson: sampling, quantisation and image arithmetic.
• IEC 61966-2-1 (1999). sRGB: a standard default colour space: the sRGB transfer function (gamma encoding).

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: colour. How RGB, grey and HSV describe the same pixel, and why HSV makes colour detection more robust.

Related content

Nothing below is required to finish this lesson.

• Before
  P1-T6: How to Evaluate a Perception System

  Choose metrics and test data that match what the robot needs, and recognise misleading evaluations.

• Lab
  P2-L1: Detect Colored Objects in an Image

  Find coloured objects in an image with a colour-space threshold and locate their centres.