VLA · Phase 8 · VLA Evaluation and Generalization
8.6 — Inference Latency and Action Frequency
How model speed limits how often a robot can be commanded.
A slow model does not have to mean a slow robot, but it always means old observations. This lesson does the arithmetic of latency, frequency, chunks and asynchronous execution, and shows why to measure the tail.
Claims about papers, models and textbooks carry numbered references, listed with what each one is cited for in the References. Code marked “Toy model” is our own illustration, not the cited system.
Learning objectives
After this lesson, you should be able to:
- Distinguish latency from action frequency.
- Compute action rates for synchronous and asynchronous chunked execution.
- Compute how far the world moves during one call.
- Measure percentiles of latency.
- Observe at time t
- Model call takes L milliseconds
- Actions arrive at t + L, from an old observation
- A chunk supplies several actions per call
- Asynchronous execution overlaps calls with motion
The one-sentence idea
Chunks and async raise the action rate; they do not remove staleness.
Measure p99, not just the mean.
Two different speeds
Two numbers are easy to confuse. Latency is how long one model call takes, from observation in to action out. Action frequency is how many actions per second the robot receives. RT-1 ran at 3 Hz, choosing a new action about every third of a second.[1] The relationship depends on how the model is used: if each call yields one action and the robot waits, frequency is at most 1 / latency. If each call yields a chunk (lesson 4.4), the frequency can be much higher than the call rate, because one call supplies several actions. SmolVLA goes further with an asynchronous inference stack that decouples perception and action prediction from action execution, allowing higher control rates with chunked action generation.[2]
From latency to achievable frequency
The arithmetic is short. A model call takes L milliseconds and returns a chunk of k actions, of which m are executed. In the synchronous case the robot waits for each call, so each batch of m actions costs L plus the time to execute them. In the asynchronous case the next call overlaps with execution:
latency_ms = 250 # one model call (an assumed number)
control_hz = 50 # the robot can execute 50 actions per second
step_ms = 1000 / control_hz
def rate(executed_per_call, asynchronous):
execute_ms = executed_per_call * step_ms
cycle_ms = max(latency_ms, execute_ms) if asynchronous else latency_ms + execute_ms
return executed_per_call / cycle_ms * 1000
print("one action per call, synchronous:", round(rate(1, False), 1), "actions per second")
for m in (5, 10, 25):
print(f"{m:>2} actions per call: synchronous {rate(m, False):5.1f}/s | asynchronous {rate(m, True):5.1f}/s (robot limit {control_hz}/s)")Output:
one action per call, synchronous: 3.7 actions per second 5 actions per call: synchronous 14.3/s | asynchronous 20.0/s (robot limit 50/s) 10 actions per call: synchronous 22.2/s | asynchronous 40.0/s (robot limit 50/s) 25 actions per call: synchronous 33.3/s | asynchronous 50.0/s (robot limit 50/s)
Toy model. The 250 ms and the 50 Hz limit are assumptions to make the arithmetic readable, not measurements of any model.
With one action per call the robot gets under 4 actions per second. Chunks of 10 raise that to about 22 per second synchronously and 40 asynchronously. Even asynchronously the robot's limit of 50 is not reached until executing a chunk takes at least as long as the model call: with 25 actions (500 ms of motion) the 250 ms call is hidden behind the motion and the rate reaches 50, while with 10 actions (200 ms of motion) the robot still waits a little
Latency also makes observations stale
An action computed at time t is applied at t + latency, using an observation from t. Whatever moved in between is unknown to the policy (lesson 1.1). The staleness in distance is simply speed × latency:
speed_m_s = 0.3 # speed of something in the scene
for latency_ms in (20, 100, 250, 600):
print(f"latency {latency_ms:>3} ms -> the object has moved {speed_m_s * latency_ms / 1000 * 100:5.1f} cm")Output:
latency 20 ms -> the object has moved 0.6 cm latency 100 ms -> the object has moved 3.0 cm latency 250 ms -> the object has moved 7.5 cm latency 600 ms -> the object has moved 18.0 cm
For a static scene, staleness hardly matters, which is why chunked, slow models still work on many tabletop tasks. For moving objects, a person handing something over, or a mobile robot, it is critical.
Measure the tail, not the average
Models rarely take exactly the same time on every call: the first call is slow, memory is reclaimed, other processes interfere. The average may meet the deadline while one call in a hundred misses it badly. Report percentiles:
import numpy as np
rng = np.random.default_rng(0)
latency = rng.normal(80, 8, 5000) # milliseconds: typical calls
slow = rng.random(5000) < 0.02 # 2% of calls hit a stall
latency[slow] += rng.uniform(100, 400, slow.sum())
deadline = 100
print("mean latency:", round(float(latency.mean()), 1), "ms")
print("median (p50):", round(float(np.percentile(latency, 50)), 1), "ms")
print("p99: ", round(float(np.percentile(latency, 99)), 1), "ms")
print("calls missing the", deadline, "ms deadline:", round(float(np.mean(latency > deadline)) * 100, 1), "%")Output:
mean latency: 83.9 ms median (p50): 80.0 ms p99: 283.4 ms calls missing the 100 ms deadline: 2.1 %
The mean looks comfortable but the p99 does not, and the share of missed deadlines matters for what the loop does next (lessons 7.3 and 9.7). Always measure on the target hardware, with the real model, over thousands of calls.
Common misconceptions
Misconception 1 “Action frequency equals 1 / latency.”
Correction Only with one action per call and no overlap.
Misconception 2 “Asynchronous execution removes the cost of a slow model.”
Correction It removes waiting, not the stale observation.
Misconception 3 “A good average latency is enough.”
Correction Rare slow calls cause missed deadlines.
Engineering takeaways
- Rate depends on actions per call and overlap.
- Staleness = speed × latency.
- Measure percentiles.
- Test on target hardware.
Fast enough on average is not fast enough.
Knowledge check
Try each question first, then reveal the explanation.
Question 1
250 ms per call, one action per call, synchronous. Maximum action rate?
Explanation
4 per second.
Question 2
Something moves at 0.5 m/s. Latency is 200 ms. How far does it move?
Explanation
10 cm.
Question 3
Why report p99 latency?
Explanation
Rare slow calls can miss deadlines even when the average is fine.
References
Numbered notes in the text, like [1], point here. Each entry says what it is cited for and how that was checked. Papers and books can be revised or misread, so follow the links before relying on a detail. Code marked “Toy model” is our own illustration of an idea, not the code of the cited system.
(2022). RT-1: Robotics Transformer for real-world control at scale. arXiv:2212.06817
Cited for:
- A model of about 35 million parameters that runs at 3 Hz. Checked against the authors’ own blog post.
(2025). SmolVLA: A vision-language-action model for affordable and efficient robotics. arXiv. arXiv:2506.01844
Cited for:
- Introduces an asynchronous inference stack that decouples perception and action prediction from action execution, allowing higher control rates with chunked action generation. Checked against the paper’s abstract.
What’s next
That completes evaluation. Next: deploying a policy as a robot component.