Skip to the lesson
FixTheRobot Home / VLA / Phase 7 / 7.1

VLA · Phase 7 · VLA Implementation and Experimentation

7.1 — Understanding a Pretrained VLA’s Inference Pipeline

What happens between receiving an observation and returning an action.

  • Roadmap phase 7
  • VLA Implementation and Experimentation
  • About 13 minutes to read

Most deployment bugs are in the steps around the model. This lesson builds each of them in small code with a stub model in the middle.

Claims about papers, models and textbooks carry numbered references, listed with what each one is cited for in the References. Code marked “Toy model” is our own illustration, not the cited system.

Learning objectives

After this lesson, you should be able to:

  • List the stages of VLA inference.
  • Write input checks that catch common bad frames.
  • Preprocess an image the way a model expects.
  • Decode and clip an output into a command.
  1. Validate the inputs
  2. Preprocess the image and tokenise the instruction
  3. Run the model
  4. Decode and de-normalise the output
  5. Clip to limits and send to the robot
The inference pipeline.

The one-sentence idea

The model is one stage in a pipeline of small, checkable steps.

Validate, preprocess like training, decode like training, clip.

From camera frame to motor command

Between “the camera took a picture” and “the motor received a number” sit a handful of small steps. Most bugs in VLA deployment live in these steps, not in the model. Following the pipeline of lesson 1.4 end to end, a pretrained VLA’s inference path looks like this:

  1. Check the inputs (right shape, right type, no NaNs).
  2. Preprocess the image (resize, scale pixel values, reorder the axes).
  3. Tokenise the instruction.
  4. Run the model: encoder, projector and backbone produce action tokens or a chunk.[1]
  5. Decode the output to continuous values and de-normalise it to the robot’s scale.[1]
  6. Post-process: clip to safe limits and hand over to the robot interface.

This lesson builds the steps around the model with a stub in the middle, so you can see each one run. Lesson 7.3 then looks at loading a real released model.

Step 1: reject bad inputs early

A model happily accepts a black frame, a frame in the wrong colour order or a NaN, and returns an answer that looks fine. Check the inputs loudly:

import numpy as np

def validate(image, instruction, state):
    problems = []
    if image.ndim != 3 or image.shape[2] != 3: problems.append(f"image must be height x width x 3, got {image.shape}")
    if image.dtype != np.uint8:               problems.append(f"image must be uint8, got {image.dtype}")
    if not np.isfinite(state).all():          problems.append("state contains NaN or infinity")
    if not instruction.strip():               problems.append("instruction is empty")
    if image.size and image.mean() < 2:       problems.append("image is almost completely black (camera not working?)")
    return problems

good = (np.full((480, 640, 3), 120, np.uint8), "pick up the red block", np.zeros(7))
bad  = (np.zeros((480, 640, 3), np.float32), "", np.array([0, 0, np.nan, 0, 0, 0, 0.0]))
print("good input problems:", validate(*good))
for p in validate(*bad): print("bad input problem:", p)

Output:

good input problems: []
bad input problem: image must be uint8, got float32
bad input problem: state contains NaN or infinity
bad input problem: instruction is empty
bad input problem: image is almost completely black (camera not working?)

Cheap, boring and valuable: a robot that refuses a bad frame is safer than one that acts on it.

Step 2: preprocess the image

Models expect a fixed image size and pixel scale, the same as in training. Resizing, scaling to 0–1, subtracting the mean and dividing by the standard deviation, and moving colour channels first are routine:

import numpy as np

def resize_nearest(img, out_h, out_w):
    h, w, _ = img.shape
    rows = (np.arange(out_h) * h / out_h).astype(int)
    cols = (np.arange(out_w) * w / out_w).astype(int)
    return img[rows][:, cols]

mean = np.array([0.485, 0.456, 0.406]); std = np.array([0.229, 0.224, 0.225])     # a commonly used set of image statistics

def preprocess(img):
    x = resize_nearest(img, 224, 224).astype(np.float32) / 255.0                  # size and scale
    x = (x - mean) / std                                                           # normalise per colour channel
    return x.transpose(2, 0, 1).astype(np.float32)                                                    # channels first

frame = np.random.default_rng(0).integers(0, 256, (480, 640, 3), dtype=np.uint8)
out = preprocess(frame)
print("input:", frame.shape, frame.dtype, "-> output:", out.shape, out.dtype)
print("value range after normalising:", round(float(out.min()), 2), "to", round(float(out.max()), 2))

Output:

input: (480, 640, 3) uint8 -> output: (3, 224, 224) float32
value range after normalising: -2.12 to 2.64

Toy model. Nearest-neighbour resizing and one common set of statistics. The right size, interpolation and statistics are those the model was trained with; get them from the model’s documentation.

Using different preprocessing at run time than in training is a classic silent failure, like the wrong action statistics in lesson 4.3.

The whole pipeline, with a stub model

Now wire the pieces together. The “model” returns a fixed normalised action, so the output is predictable and you can verify the rest of the chain:

import numpy as np

action_mean = np.array([0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.5])
action_std  = np.array([0.02, 0.02, 0.02, 0.1, 0.1, 0.1, 0.5])
limit       = np.array([0.05, 0.05, 0.05, 0.3, 0.3, 0.3, 1.0])

def stub_model(image_tensor, instruction):
    return np.array([1.0, -1.0, 0.0, 0.5, 0.0, 0.0, 3.0])           # a normalised action; the last entry is exaggerated

def infer(image, instruction, state):
    if not np.isfinite(state).all() or image.ndim != 3:
        raise ValueError("bad input")
    x = image.astype(np.float32) / 255.0
    normalised = stub_model(x, instruction)                           # the model
    physical = normalised * action_std + action_mean                  # de-normalise
    return np.clip(physical, -limit, limit)                           # never exceed the limits

frame = np.full((224, 224, 3), 100, np.uint8)
print("command:", np.round(infer(frame, "close the gripper", np.zeros(7)), 3).tolist())

Output:

command: [0.02, -0.02, 0.0, 0.05, 0.0, 0.0, 1.0]

Look at the last number: the stub asked for a gripper value of 3 standard deviations, 2.0 in physical units, and the clip held it at 1.0. Lesson 9.4 covers limits in depth.

Where the time goes

Each stage costs time; measuring them is the first step to meeting a control rate (lesson 8.6). A helper that records the time of each stage is a few lines (the numbers depend on your machine, so none are quoted here):

import time

def timed(stage_fn, *args):
    t0 = time.perf_counter()
    result = stage_fn(*args)
    return result, (time.perf_counter() - t0) * 1000

_, ms = timed(sum, range(100000))
print("timed a stage; it returns the result and the elapsed milliseconds:", isinstance(ms, float) and ms >= 0)

Output:

timed a stage; it returns the result and the elapsed milliseconds: True

Wrap each stage this way, log the numbers, and look at the slowest. The image encoder and the backbone usually dominate, but measure instead of assuming.

Common misconceptions

Misconception 1 “Inference means calling the model.”

Correction The steps around the model are where most errors occur.

Misconception 2 “Any resize is fine.”

Correction Use exactly the preprocessing the model was trained with.

Misconception 3 “A confident output is a correct output.”

Correction A model returns a plausible answer even for a black frame.

Engineering takeaways

  1. Six stages.
  2. Validate inputs.
  3. Match training preprocessing.
  4. Decode, then clip.
  5. Time each stage.

The model is only one step.

Knowledge check

Try each question first, then reveal the explanation.

Question 1

Why check for an almost black image?

Explanation

The model would still return a plausible action for a broken camera.

Question 2

What must preprocessing match?

Explanation

The size, scaling and statistics used in training.

Question 3

What follows decoding?

Explanation

Clipping to safe limits before the command is sent.

References

Numbered notes in the text, like [1], point here. Each entry says what it is cited for and how that was checked. Papers and books can be revised or misread, so follow the links before relying on a detail. Code marked “Toy model” is our own illustration of an idea, not the code of the cited system.

  1. Kim, M. J., et al. (2024). OpenVLA: An open-source vision-language-action model. Conference on Robot Learning (CoRL) 2024. arXiv:2406.09246

    Cited for:

    • The model has three parts: a visual encoder that turns the image into patch embeddings, a projector that maps them into the language model’s input space, and the language-model backbone. Checked against the paper, via summaries of its text.
    • At run time the predicted action tokens are decoded to continuous values and de-normalised to the robot’s physical scale. Checked against the paper, via summaries of its text.

What’s next

Next: preparing datasets for training.

Next lesson

7.2 — Dataset preparation and preprocessing

Formatting, cleaning and normalising demonstrations for training.

Back to the VLA roadmap