M4.1 — RGB Images and Image Representations

Learning objectives

After this lesson, you should be able to:

• Describe an image as an array and read its shape.
• Explain RGB channels and index a single channel.
• Explain the uint8 range and the wrap-around trap.
• Convert a colour image to grayscale.
• Estimate the data size of a camera stream.
• Turn an image array into a channels-first float tensor with a batch dimension.

The one-sentence idea

An image is an array of numbers with a height, a width and three colour channels, and getting it into a model means fixing its type, range and layout.

Check the shape, the type, the range and the channel order.

An image is a grid of numbers

To a computer, a picture is an array. A grayscale image is a 2-D array in which each entry, a pixel, holds a brightness. The array’s shape is (height, width). By convention each value is stored as an 8-bit integer from 0 (black) to 255 (white). Positions are written (row, column), with the origin at the top-left corner and the row index increasing downwards, which is the reverse of the usual graph axes and a frequent source of confusion in robotics code.

import numpy as np

gray = np.array([[  0,  64, 128],
                 [ 64, 128, 192],
                 [128, 192, 255]], dtype=np.uint8)
print(gray.shape, gray.dtype)
print(gray[0, 2], gray[2, 0])      # top-right and bottom-left pixels

(3, 3) uint8
128 128

This one is a smooth diagonal gradient from black at the top-left to white at the bottom-right. Real images are the same thing at a larger size: a 640 × 480 photo is a 480-by-640 array of numbers.

RGB: three channels

A colour image stores three numbers per pixel: how much red, green and blue light there is. Each of these planes is a channel, and the array shape becomes (height, width, 3) in the common “channels last” layout. Pure red is [255, 0, 0], white is [255, 255, 255] and black is [0, 0, 0].

img = np.zeros((2, 2, 3), dtype=np.uint8)
img[0, 0] = [255, 0, 0]        # top-left: red
img[0, 1] = [0, 255, 0]        # top-right: green
img[1, 0] = [0, 0, 255]        # bottom-left: blue
img[1, 1] = [255, 255, 255]    # bottom-right: white
print(img.shape)
print(img[:, :, 0])            # the red channel on its own
print(img[1, 0])               # all three values of one pixel

(2, 2, 3)
[[255   0]
 [  0 255]]
[  0   0 255]

The red channel is 255 where the pixel is red or white and 0 elsewhere, exactly as you would expect. Selecting a channel is ordinary slicing from M2.2.

A warning about channel order: most libraries use RGB, but OpenCV’s image-loading functions use BGR (blue first). If colours look swapped, with skin turning blue, the channels are in the wrong order. Reversing the last axis fixes it:

bgr = img[..., ::-1]           # reverse the last axis
print(bgr[0, 0])               # the red pixel, now stored as blue, green, red

[  0   0 255]

Value ranges and a common trap

Images are stored as uint8 (0 to 255) but models want floating-point numbers, usually scaled to the range 0 to 1 and often then normalised per channel by subtracting a mean and dividing by a standard deviation. The first step is a division by 255. The reason to convert early is a trap in the integer type: uint8 arithmetic wraps around instead of growing.

a = np.array([200], dtype=np.uint8)
b = np.array([100], dtype=np.uint8)
print(a + b)                                  # 300 does not fit in 8 bits
print(a.astype(np.float32) + b)               # convert first
print((a / 255).round(3))                     # scaled to the 0..1 range

[44]
[300.]
[0.784]

200 + 100 should be 300, but an 8-bit integer can only hold up to 255, so the result wrapped to 300 − 256 = 44. A brightened image with such an error would show dark patches where it should be bright. Convert to float before doing arithmetic on pixels.

Converting colour to grayscale

Turning an RGB image into a single channel is a weighted sum of the three channels, which is a dot product with a weight vector (M1.1). Human vision is most sensitive to green and least to blue, so the standard weights are about 0.299 for red, 0.587 for green and 0.114 for blue:

weights = np.array([0.299, 0.587, 0.114])
gray_img = img @ weights                 # (2, 2, 3) @ (3,) -> (2, 2)
print(gray_img.round(1))

[[ 76.2 149.7]
 [ 29.1 255. ]]

Pure green becomes much brighter (149.7) than pure blue (29.1), matching how bright they look to us. The result has shape (2, 2), one number per pixel, the channel axis having been summed away.

Cropping, resizing and how big images are

Cropping is slicing: img[10:90, 20:120] keeps rows 10 up to 90 and columns 20 up to 120. Resizing changes the number of pixels, and needs interpolation (deciding what value a new pixel takes) from a library such as OpenCV, Pillow or PyTorch. Models usually expect a fixed input size, such as 224 × 224, so camera images are cropped and resized before they are used.

Size matters in robotics, because a camera produces a lot of data:

h, w = 480, 640
frame = h * w * 3                          # bytes in one RGB uint8 frame
print(frame)
print(frame * 30)                          # bytes per second at 30 frames per second
print(224 * 224 * 3)                       # numbers in a 224 x 224 model input

921600
27648000
150528

A single 640 × 480 colour frame is 921,600 bytes, and a 30 frames-per-second camera produces about 27.6 million bytes a second. A 224 × 224 model input holds 150,528 numbers. Reducing resolution is one of the main ways robot systems control compute cost, at the price of losing small details.

Not every robot camera gives colour. A depth image has a single channel in which each pixel stores a distance, typically in millimetres or metres, and shape (height, width). Many robots use an RGB image and a depth image of the same view together.

From image to tensor

Putting it together, getting an image into a model takes four steps, using the tensor operations of M2.3:

1. Start with an array of shape (H, W, 3) and type uint8.
2. Convert to float32 and divide by 255 to scale to 0–1.
3. Reorder to channels first, (3, H, W), which PyTorch models expect.
4. Add a batch dimension, giving (1, 3, H, W).

import torch

frame = np.random.default_rng(0).integers(0, 256, size=(4, 6, 3), dtype=np.uint8)   # stand-in for a camera frame
x = torch.from_numpy(frame).float() / 255            # float32, scaled to 0..1
x = x.permute(2, 0, 1).unsqueeze(0)                  # channels first, then a batch axis
print(x.shape, x.dtype)
print(bool(x.min() >= 0 and x.max() <= 1))

torch.Size([1, 3, 4, 6]) torch.float32
True

The tensor has shape (1, 3, 4, 6): one image, three channels, 4 rows, 6 columns. This is the form the vision encoders in M4.3 consume. Getting the order, the range and the channel convention right is a large part of avoiding silent bugs in a robot-learning pipeline: a model fed 0–255 values or BGR colours may run without any error and simply perform badly.

Common misconceptions

Engineering takeaways

1. An image is an array. Grayscale is (H, W) and RGB is (H, W, 3).
2. Row 0 is the top. Positions are (row, column).
3. Standard 8-bit values run 0–255, and models usually want float32 in 0–1.
4. uint8 arithmetic wraps around, so convert to float first.
5. Check whether the channel order is RGB or BGR.
6. Grayscale is a weighted sum of the channels.
7. Camera streams are large, so resolution drives compute cost.
8. A model input is usually (batch, 3, H, W), as a float tensor.

Before debugging a vision model, check what the pixels actually are.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

You can now read an image as data. Next is how vision systems pick out useful structure from it, starting from edges and convolutions.

Back to the VLA roadmap prerequisites