M6.6 — Introduction to Reinforcement Learning

Learning objectives

After this lesson, you should be able to:

• Define reward, return and the discount factor, and compute a discounted return.
• Explain exploration versus exploitation with an ε-greedy bandit.
• Describe a Q-value and the Q-learning update.
• Explain the idea of a policy gradient.
• Explain why sparse rewards and real-world cost make robot RL hard.
• Describe how imitation and reinforcement learning are combined.

The one-sentence idea

Reinforcement learning improves a policy by trying actions, receiving rewards, and shifting toward actions that led to more total reward.

No answers, only scores: find what earns the most.

Learning from reward instead of examples

In imitation learning (M6.2) the robot is shown what to do. In reinforcement learning (RL) it is only told how well it did. The robot acts, the world responds, and a number called the reward arrives, positive for good outcomes and negative or zero for bad ones. There is no correct action to copy. The policy must discover, by trial and error, which actions lead to more reward over time.

The setup uses the loop of M6.5 with one addition. At each step the environment returns, along with the next observation, a reward r. The policy’s aim is to maximise the return: the total reward collected over an episode, with later rewards usually discounted by a factor γ between 0 and 1 so that sooner is better than later:

import numpy as np

def discounted_return(rewards, gamma):
    return sum(gamma ** t * r for t, r in enumerate(rewards))

rewards = [0, 0, 0, 1]                      # nothing until the goal is reached on the fourth step
print(discounted_return(rewards, 0.9))
print(discounted_return([0, 0, 0, 0, 0, 1], 0.9))    # reaching the same goal two steps later

0.7290000000000001
0.5904900000000001

Reaching the goal in four steps is worth 0.9³ = 0.729, and two steps later only 0.9⁵ ≈ 0.59, so a policy maximising the return prefers the shorter route. (The long decimals are ordinary floating-point rounding.) A reward is a design decision made by a person. It stands in for what “success” means, and as M6.4 noted for evaluation, it is only a proxy.

Exploration versus exploitation

Because the robot does not know which actions are good, it faces a dilemma at every step: exploit the best action found so far, or explore a different one that might be better. The simplest setting is a bandit: several options (“arms”) each paying out with an unknown probability. An ε-greedy strategy picks the best-looking arm most of the time and a random one with probability ε:

rng = np.random.default_rng(0)
true_p = np.array([0.2, 0.5, 0.8])           # the real payout rate of each arm (unknown to the learner)
counts, wins = np.zeros(3), np.zeros(3)

for pull in range(500):
    if rng.random() < 0.1 or counts.min() == 0:      # explore, and try every arm at least once
        arm = int(rng.integers(3))
    else:
        arm = int(np.argmax(wins / counts))           # exploit the best estimate
    counts[arm] += 1
    wins[arm] += rng.random() < true_p[arm]

print((wins / counts).round(2), counts)
print(int(np.argmax(counts)))

[0.14 0.42 0.79] [ 14.  36. 450.]
2

After 500 pulls the learner’s estimates (0.14, 0.42, 0.79) are close to the true rates (0.2, 0.5, 0.8), and it has put 450 of its pulls on the best arm, arm 2. Exploring cost it a few pulls on worse arms, and without it the learner could have settled on the wrong arm forever. Real robot tasks have a huge space of possible behaviours, which makes exploration the hard part.

Value and Q-learning

A value function estimates the return a policy can expect from a situation. The Q-value Q(s, a) is the expected return from taking action a in state s and acting well afterward. If you knew Q, the best policy is simply to pick the action with the highest Q-value. Q-learning learns Q from experience, nudging each estimate toward “the reward I got, plus the discounted value of the best next action”:

Below, a robot on a 4 × 4 grid starts in a corner and must reach the opposite corner, which pays a reward of 1. It knows nothing at first, and learns by playing 300 episodes, mostly following its current best guess and sometimes trying a random move:

rng = np.random.default_rng(0)
N, goal = 4, (3, 3)
moves = [(-1, 0), (1, 0), (0, -1), (0, 1)]                  # up, down, left, right
Q = np.zeros((N, N, 4))

def greedy(pos):                                            # best action, breaking ties at random
    q = Q[pos]
    return int(rng.choice(np.flatnonzero(q == q.max())))

def step(pos, a):
    r, c = pos
    dr, dc = moves[a]
    nxt = (min(max(r + dr, 0), N - 1), min(max(c + dc, 0), N - 1))     # stay inside the grid
    return nxt, (1.0 if nxt == goal else 0.0), nxt == goal

for episode in range(300):
    pos = (0, 0)
    for _ in range(50):
        a = int(rng.integers(4)) if rng.random() < 0.2 else greedy(pos)
        nxt, reward, done = step(pos, a)
        target = reward + (0.0 if done else 0.9 * Q[nxt].max())
        Q[pos][a] += 0.5 * (target - Q[pos][a])             # the Q-learning update
        pos = nxt
        if done:
            break

pos, path = (0, 0), [(0, 0)]                                # follow the learned greedy policy
for _ in range(20):
    pos, _, done = step(pos, greedy(pos))
    path.append(pos)
    if done:
        break
print(len(path) - 1, path[-1])
print(round(float(Q[0, 0].max()), 3))

6 (3, 3)
0.59

(When several actions tie, as they all do in an empty table, the code breaks the tie at random. Always taking the same one can leave the robot never stumbling onto the reward.) The learned greedy policy reaches the goal in 6 moves, the shortest possible on this grid (3 down and 3 right). The value of the start state, 0.59, is the discounted reward of a 6-step trip, 0.9⁵ = 0.5905, matching the prediction of the formula. The robot learned a policy without ever being shown one. Q-learning of this table form works only for small problems. For images and continuous actions the table is replaced by a neural network, which is where the methods get harder.

Policy gradients: adjusting the policy directly

A second family improves the policy directly. The idea of a policy gradient is simple: if an action was followed by more reward than usual, make it more likely, and if by less, make it less likely. Applied to the bandit with a softmax policy (M3.1), the update on the policy’s logits θ is θ += lr · (reward − baseline) · (one-hot(action) − π), where the baseline is the running average reward and reduces noise:

def softmax(z):
    e = np.exp(z - z.max())
    return e / e.sum()

rng = np.random.default_rng(1)
theta, baseline = np.zeros(3), 0.0                       # the policy starts uniform over three arms
for t in range(2000):
    pi = softmax(theta)
    arm = rng.choice(3, p=pi)                            # sample from the stochastic policy
    reward = float(rng.random() < true_p[arm])
    theta += 0.1 * (reward - baseline) * (np.eye(3)[arm] - pi)
    baseline += 0.05 * (reward - baseline)
print(softmax(theta).round(2))

[0.01 0.02 0.97]

The policy started with probability 1/3 on each arm and ended with essentially all of it on the best arm. Notice that it needed thousands of pulls to find a pattern a person could spot in a few dozen, which hints at the cost of learning from reward alone.

Why reinforcement learning on robots is hard

• Sparse rewards and exploration. If the reward only comes on success, random behaviour almost never finds it. The experiment below shows how quickly chance success vanishes as a task needs more steps:

def random_success_rate(length, horizon=40, episodes=2000, seed=0):
    rng = np.random.default_rng(seed)
    wins = 0
    for _ in range(episodes):
        pos = 0
        for _ in range(horizon):
            pos = max(0, pos + (1 if rng.random() < 0.5 else -1))     # a random walk along a corridor
            if pos == length:
                wins += 1
                break
    return wins / episodes

print([random_success_rate(n) for n in (3, 6, 12)])

[0.979, 0.61, 0.1055]

A random walk reaches the end of a corridor 3 cells long almost every time within 40 steps (98%), a 6-cell corridor 61% of the time, and a 12-cell one only 11% of the time, and the fraction keeps falling as the corridor lengthens. A real manipulation task is far more demanding than a corridor, so learning from a sparse reward from scratch can take an impractical number of attempts.

• Sample inefficiency and cost. Real robots are slow, wear out, and have to be reset between attempts. Methods that need millions of trials are normally run in simulation.
• Reward design. A badly specified reward gets exploited: a robot rewarded for closeness to an object may hover next to it forever. Writing a reward that captures what you mean is difficult.
• Safety. Exploration means trying things that fail, which on a real robot can damage hardware or people.

Because of these limits, practical systems rarely use RL alone. A common pattern is to start from behavior cloning on demonstrations (M6.2), so that the policy already succeeds sometimes, and then improve it with reinforcement learning, in simulation or on the real robot. VLA training largely relies on imitation, and reinforcement learning enters as an improvement stage in the advanced phase on research directions. Understanding it now means you will recognise the terms there: reward, return, value, exploration and policy gradient.

Common misconceptions

Engineering takeaways

1. RL learns from a reward signal instead of correct actions.
2. The return is the total discounted reward, and the discount makes sooner rewards worth more.
3. Exploration versus exploitation is the central dilemma.
4. Q(s, a) is the expected return of an action, and the policy can pick the highest.
5. Q-learning updates Q toward reward plus the discounted best next value.
6. Policy gradients make actions that earned above-average reward more likely.
7. Sparse rewards make exploration very hard as tasks get longer.
8. Practical robot systems often start from imitation and refine with RL.

Imitation tells the robot what to do. Reward only tells it how well it did, so it has to find the rest out.

Knowledge check

Try each question first, then reveal the explanation.

What’s next

That completes the prerequisites. The roadmap proper starts with Phase 1, an introduction to what a Vision-Language-Action model is.

Back to the VLA roadmap prerequisites