10.2 — Gemini Robotics

Learning objectives

After this lesson, you should be able to:

• Distinguish Gemini Robotics from Gemini Robotics-ER.
• Convert a pointed pixel plus depth into a base-frame position.
• Describe a planner, executor and check loop.
• Estimate the operator time for 100 demonstrations.

The one-sentence idea

One model reasons about space; another acts; a loop checks.

Perception outputs need geometry to become motion.

Two models from one family

Google DeepMind's Gemini Robotics report introduces two models. Gemini Robotics is a vision-language-action model built on Gemini 2.0 that outputs robot control directly. The authors report fluid manipulation of varied objects, robustness to changes in object type and position, operation in unseen settings and open-vocabulary instruction following.^[1] Gemini Robotics-ER (“embodied reasoning”) extends Gemini's multimodal reasoning into the physical world, with better spatial and temporal understanding. It supports object detection, pointing, trajectory and grasp prediction, multi-view correspondence and 3D bounding-box prediction, and the VLA builds on top of it.^[1]

With further fine-tuning, the authors report the model can take on long-horizon, highly dexterous tasks, learn new short-horizon tasks from as few as 100 demonstrations, and adapt to new robot embodiments, and they also discuss safety considerations.^[1] For the architecture, the tables and figures, a re-check of the numbers and a critical review, see the Gemini Robotics paper guide.^[1]

What “embodied reasoning” outputs look like

The ER model answers questions about a scene in forms a robot can use: where is something (a point or a box), how could it be grasped, what path might it follow. These are perception results in image coordinates, and they still need to be turned into robot motion. Here is the bridge for a pointing result, using the camera model of lesson 9.3. The model points at a mug in the image; a depth reading and the camera pose turn the pixel into a position in the robot's base frame:

import numpy as np

f, cx, cy = 600.0, 320.0, 240.0                               # camera intrinsics (pixels)
base_from_camera = np.eye(4)
base_from_camera[:3, :3] = [[0, -1, 0], [-1, 0, 0], [0, 0, -1]]   # a camera looking straight down
base_from_camera[:3, 3] = [0.5, 0.0, 0.9]                     # mounted 0.9 m above the table, 0.5 m in front of the base

point_in_image = {"label": "mug", "u": 380, "v": 200}         # an output in the style of a pointing result (our own format)
depth = 0.9                                                   # metres from the camera to the table, from a depth sensor

x = (point_in_image["u"] - cx) * depth / f
y = (point_in_image["v"] - cy) * depth / f
camera_point = np.array([x, y, depth, 1.0])
base_point = base_from_camera @ camera_point
print("mug at pixel", (point_in_image["u"], point_in_image["v"]), "-> camera frame", np.round(camera_point[:3], 3).tolist())
print("mug in the robot base frame:", np.round(base_point[:3], 3).tolist())

mug at pixel (380, 200) -> camera frame [0.09, -0.06, 0.9]
mug in the robot base frame: [0.56, -0.09, 0.0]

Toy model. The output format, camera pose and numbers are invented for the example; they are not Gemini Robotics-ER’s interface.

The result (about 0.56 m in front of the base, 9 cm to the side, on the table at z = 0) is something a grasp planner or a VLA prompt can use. Every link in this chain is a place for errors, which is why lessons 9.3 and 9.4 matter.

Using the two together

One way to use a reasoning model and an action model together is hierarchical: the reasoning model breaks an instruction into steps, the action model carries each out, and the reasoning model checks the outcome (compare the language-feedback loop of Inner Monologue in lesson 8.5^[2]). A small control skeleton of that pattern, with stand-ins for both models:

def planner(instruction):
    plans = {"tidy the desk": ["pick up the pen", "put the pen in the cup", "pick up the paper", "put the paper in the tray"]}
    return plans.get(instruction, [])

attempts = {"pick up the paper": 0}
def executor(step):                       # stand-in for the action model: one step fails the first time
    if step == "pick up the paper":
        attempts[step] += 1
        return attempts[step] > 1
    return True

log = []
for step in planner("tidy the desk"):
    for attempt in range(1, 4):
        ok = executor(step)
        log.append(f"{step!r} attempt {attempt}: {'done' if ok else 'failed, retry'}")
        if ok: break
print("\n".join(log))

'pick up the pen' attempt 1: done
'put the pen in the cup' attempt 1: done
'pick up the paper' attempt 1: failed, retry
'pick up the paper' attempt 2: done
'put the paper in the tray' attempt 1: done

The check after each step is the closed loop at the planning level. Lesson 11.5 looks at this structure in detail.

What “100 demonstrations” means in hours

The report says new short-horizon tasks can be learned from as few as 100 demonstrations.^[1] It helps to see what that costs in operator time, because collecting data is the real expense (lesson 5.1):

demos = 100
for seconds_each in (20, 45, 90):
    setup_seconds = 30                                        # resetting the scene between demonstrations
    total_min = demos * (seconds_each + setup_seconds) / 60
    print(f"{seconds_each:>3} s per demonstration + {setup_seconds} s reset -> {total_min:5.1f} minutes ({total_min/60:.1f} hours) of operator time")

 20 s per demonstration + 30 s reset ->  83.3 minutes (1.4 hours) of operator time
 45 s per demonstration + 30 s reset -> 125.0 minutes (2.1 hours) of operator time
 90 s per demonstration + 30 s reset -> 200.0 minutes (3.3 hours) of operator time

Roughly one and a half to three and a half hours of focused work. Cheap compared with training from scratch, but not free, and it assumes the base model already knows the objects and the robot's embodiment well enough. “As few as” describes the best case in the report; your task, robot and success criterion decide your case (lessons 8.2 and 8.3).

Common misconceptions

Engineering takeaways

1. Built on Gemini 2.0.
2. ER: pointing, detection, grasp and trajectory prediction.
3. Convert image results using calibration.
4. Check each step.

Reasoning proposes; geometry and control dispose.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

Next: Figure’s Helix.

Back to the VLA roadmap