Navigation · Theory · M9
M9-T6 — Learned and Language-Guided Navigation
A network that maps pictures to steering is appealing and unfinished: here is what it offers, what it has not shown, and how to keep a robot safe while it learns.
Everything so far in this track is written by hand: maps, costs, planners, controllers, recoveries. Learning-based methods replace some of it with a model trained on data, and language-guided methods let a person say “take this to the nurses’ station on the left” instead of giving coordinates. The ideas are promising and the evidence is thinner than for the classical stack. This lesson draws the line between what is shown and what is hoped.
Learning objectives
After this lesson, you should be able to:
- Describe three families: learned end-to-end policies, learned components inside a classical stack, and language-guided navigation.
- Say what learned methods offer (semantics, adaptation, less manual tuning) and what they lack (guarantees, predictable failure, easy debugging).
- Estimate how latency and success-rate uncertainty limit a learned system, with a worked example.
- Describe an architecture that uses a learned model with an independent safety layer, and say what must be shown before trusting it.
The intuition
A classical navigation stack is like a set of written procedures: if the corridor is blocked, clear the map, then spin, then wait. You can read it, test each step and say why it did what it did. A learned policy is like a trainee who has watched thousands of hours of corridors: it often does the right thing without being able to say why, can do something clever the procedures never cover, and can also do something odd in a situation unlike its training.
Language adds a front end: the person speaks, a large model interprets, and something else has to move the robot. The difficult part is not the sentence; it is the link between words and places on the robot’s map, and the guarantee that whatever the model says, the robot will not drive into a person.
The mathematics
Three families
| Family | What is learned | Examples (from the literature) | Typical evidence |
|---|---|---|---|
| End-to-end policy | A map from observations (image, depth, goal) to velocity or waypoints, trained by imitation or reinforcement learning | Target-driven visual navigation (Zhu et al.), cognitive mapping and planning (Gupta et al.), BADGR (Kahn et al.) | Success rate and SPL on benchmark scenes, often in simulation; some real-robot trials |
| Learned components in a classical stack | A cost or traversability estimate, a local policy, a place recognizer, a semantic map | Learned traversability, goal-oriented semantic exploration (Chaplot et al.) | Improvements in a particular module, with the rest unchanged |
| Foundation-model and language-guided | General visual and language models used for goal specification or as a navigation prior | LM-Nav (Shah et al.), ViNT (Shah et al.), vision-and-language navigation (Anderson et al.) | Demonstrations and benchmarks; the range of conditions is still being explored |
What learning offers
- Semantics. “The door on the left of the nurses’ station” needs meaning; a costmap has none.
- Adaptation to data. A traversability estimate can learn what the sensors do not measure directly.
- Less hand-tuning for some modules, replaced by data collection and training.
What is not shown
- Guarantees. There is no bound on behaviour outside the training distribution, and no worst-case argument comparable to a planner that is complete on a grid.
- Predictable failure. A classical failure has a stage and a log line. A learned policy’s failure may be unexplainable and may change with a new training run.
- Real-world evidence. Many results are in simulation or in a few environments. How well simulation predicts the real robot has to be measured (lesson M9-T4).
- Cost of verification. Every retraining changes the system; the evaluation of lesson M9-T4 has to be rerun, with enough runs for the rare events.
Latency and rates
A large model may run at a few hertz. In the time it takes to decide, the robot moves:
Architecture for safe use
- Keep the structure. Let the learned model produce a goal, a waypoint or a cost, and let a classical planner and controller turn that into motion.
- Constrain the output. Limit speed, acceleration and the region the model may command.
- Monitor independently. A collision monitor based on current range data, not on the model, slows or stops the robot regardless of what the model says (module 7).
- Log and evaluate. Record the model version with every run, and evaluate with the same scenarios and intervals as any other configuration.
A worked example
How much does a slow model cost? Scout runs a visual policy that produces a command every 0.25 s (4 Hz), at 1.0 m/s, with braking of 1.2 m/s² and 0.3 s of reaction time:
The robot commits to 0.25 m of motion for each decision, and needs about a metre to stop. A lidar-based monitor running at 20 Hz needs 1.0 × (0.05 + 0.30) + 0.42 = 0.77 m: it can stop the robot in less distance than the policy, which is why the monitor must run independently, at its own rate.
How good is “78 of 100”? A learned policy reaches the goal in 78 of 100 test episodes. The classical stack reaches it in 94 of 100 on the same scenarios. The Wilson 95% intervals are 0.69 to 0.85 and 0.88 to 0.97: they do not overlap, so the classical stack is better on this test, with a gap of at least a few percentage points. It would be wrong to conclude the opposite from a demo video of the policy’s best run. And it would be wrong to conclude that learning cannot win: if the test included glass walls and open-vocabulary goals that the classical stack cannot express, the comparison changes.
A language instruction. The nurse says “take this to the pharmacy on the second floor”. A language model must map this to a node of Scout’s semantic map (lesson M2-T5), return a goal, and say when it is not sure. If the system silently picks the wrong “pharmacy”, the planner will faithfully take the tray there. Two checks are cheap and effective: ask the person to confirm a goal that has low confidence, and refuse a goal in a region the robot may not enter.
Engineering implications and limitations
Use learning where it is strongest
Perception, semantic understanding and goal specification are good fits, because errors there can be caught downstream. Low-level safety-critical control is a poor fit for the first deployment.
Treat the model as a component with a version
Record the model identifier, the training data description and the date with every run. A silently updated model is a silently changed robot.
Test the failure modes you can name
Run scenes unlike the training scenes: different lighting, glass, mirrors, crowds, clutter, a reflective floor after cleaning. Report the success rate with its interval, the failures by type, and the situations where the safety monitor intervened.
State what is unproven
Say plainly in any design document: which behaviours were tested, with how many runs, in what environments, and which claims (for example generalisation to a new building, or safety around people) have not been tested. The capstone asks for this kind of statement.
Keep the classical fallback
A robot that can fall back to a classical stack when the model is uncertain, or when a monitor trips, degrades gracefully. A robot that depends only on the model has the model’s failure modes and nothing else.
Common misconceptions
“End-to-end learning makes the navigation stack obsolete.”
It replaces some modules in some settings. The evidence for replacing the whole stack in safety-critical environments is limited, and the stack provides debuggability and guarantees.
“A language model understands the building.”
It maps words to guesses. Without a grounded map and confirmation, a plausible but wrong goal is a real risk.
“Success in simulation means success on the robot.”
Simulation predictivity is an empirical question that has to be measured for each setup.
“If the policy is safe on average it is safe.”
Safety is about the rare case. Averages over typical runs say little about the tail; an independent monitor and design arguments address it.
Engineering takeaways
- Learned navigation comes as end-to-end policies, learned components in a classical stack, and language- or foundation-model-guided systems.
- Learning offers semantics, adaptation and less hand-tuning; it lacks guarantees, predictable failure and, often, real-world evidence.
- Slow decisions cost distance: speed times inference time, plus the stopping distance.
- Use learned models for perception and goal specification behind a classical planner and an independent monitor.
- Evaluate with intervals, version the model, and say plainly what has not been tested.
References and further reading
Primary papers for the families named above; the field is moving quickly, so check for newer work.
- Y. Zhu, R. Mottaghi, E. Kolve, J. J. Lim, A. Gupta, L. Fei-Fei and A. Farhadi, “Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning”, IEEE ICRA, 2017.
- S. Gupta, J. Davidson, S. Levine, R. Sukthankar and J. Malik, “Cognitive Mapping and Planning for Visual Navigation”, IEEE CVPR, 2017.
- P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, I. Reid, S. Gould and A. van den Hengel, “Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments”, IEEE CVPR, 2018.
- D. Shah, B. Osiński, B. Ichter and S. Levine, “LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action”, Conference on Robot Learning (CoRL), 2022.
- D. Shah, A. Sridhar, N. Dashora, K. Stachowicz, K. Black, N. Hirose and S. Levine, “ViNT: A Foundation Model for Visual Navigation”, Conference on Robot Learning (CoRL), 2023.
Knowledge check
Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.
Question 1
A policy runs at 5 Hz and the robot drives at 0.8 m/s. How far does it move between decisions, and why does this matter for safety?
Explanation
0.8 × 0.2 = 0.16 m per decision. The robot is committed to that motion before the next decision, and the model cannot react faster than 5 Hz, so an independent monitor running at a higher rate is needed to stop the robot.
Question 2
A learned system succeeds in 85 of 100 runs and the classical stack in 90 of 100 on the same scenarios. A colleague says the classical stack is better. What do you reply?
Explanation
Not proven: the 95% intervals (about 0.77 to 0.91 and 0.83 to 0.94) overlap heavily. More runs on the same scenarios, with a paired comparison, are needed.
Question 3
A language-guided system maps “the pharmacy” to the wrong room with high confidence. Which two safeguards in the architecture would limit the damage?
Explanation
Confirmation of goals before execution (or a refusal for low-confidence and restricted regions), and an independent safety monitor and keep-out areas in the classical stack that stop the robot entering places it must not, whatever the model says.
What’s next
Next: the capstone, where design, trade-offs and a safety case bring the track together.
Related content
Nothing below is required to finish this lesson.
- BeforeM9-T5: Other Platforms: Legged, Aerial, Marine and Off-Road
Say what changes for legged, aerial, marine and off-road robots, and where grid navigation stops being enough.
Get notified when the next lesson and new labs are released.
One email per new lesson or lab.