M7-T5: ROS 2 execution and scheduling

Learning objectives

After this lesson, you should be able to:

• Describe the ROS 2 executor model and the role of callback groups.
• Predict how a long callback affects other callbacks on the same thread.
• Choose QoS history, depth and reliability for sensor and command streams.
• Apply practical measures: separate executors, reentrant groups and bounded work in callbacks.

Conceptual explanation

Implementation-specific ROS 2 executor behaviour has changed between distributions (new executors and events-based execution have been added). The principles below hold across them; check the documentation of the distribution you use for the available executors and their guarantees.

In ROS 2 a node’s work is a set of callbacks: subscription handlers, timers, service handlers, action handlers. An executor waits for ready callbacks and runs them on one or more threads.

• Single-threaded executor: one callback at a time; a callback that takes 80 ms delays every other callback, in all nodes spun on that executor, by up to 80 ms.
• Multi-threaded executor: several callbacks in parallel, subject to callback groups: a mutually exclusive group runs one callback at a time; a reentrant group allows parallel execution of its callbacks (your code must then be thread-safe).
• QoS sets the queue behaviour: history (keep last $N$ or keep all), depth, reliability (best effort or reliable) and durability. A deep queue on a slow subscriber turns an overload into growing latency, as in the workload lesson.

Engineering intuition

An executor is a single cashier serving a queue of customers (callbacks). One customer with a huge basket (a 100 ms callback) delays everyone behind them. Opening more tills (threads) helps only if customers are independent: two customers who need the same item from the same shelf (a mutex, a mutually exclusive group) still wait for each other.

The fixes are structural: keep callbacks short (hand heavy work to a worker thread or a separate node), put time-critical callbacks (IMU, control) on their own executor and thread, make sensor subscriptions best-effort with depth 1–5, and measure the callback durations.

Mathematical formulation

For callbacks sharing one thread, the worst-case latency of a callback of a short job $i$ is bounded by the longest job that may be running when it becomes ready plus the work queued ahead:

For periodic timers the same fixed-priority analysis of the RTOS lesson applies, but ROS 2 executors do not preempt: they are non-preemptive, so every callback contributes blocking equal to its full execution time. A callback of 30 ms on the same thread as a 10 ms timer makes the timer jitter by up to 30 ms. With a subscription queue of depth $q$ and a callback taking $S$ for a message stream at period $T_{m}<S$, messages are dropped at the rate $(1/T_{m}-1/S)$ once the queue is full, and the data age at processing is about $q$ $S$.

$C_{j}$

execution time of callback $j$.

$q$

subscription queue depth.

$S$

callback service time per message.

Architecture and data flow

DDS middleware receives samples into a per-subscription history buffer (QoS) → the executor’s wait set marks the subscription ready → the executor takes the message and runs the callback → the callback publishes downstream messages. Timers are executed in the same way. The order among ready callbacks depends on the executor implementation (for example round-robin over entities in the classic executor); do not rely on an order that is not documented.

Real-world UAV applications

• Making a VIO node and its consumers meet frame deadlines.
• Separating IMU preintegration from heavy image processing.
• Tuning the offboard setpoint publisher so it never stalls.
• Diagnosing latency spikes seen in end-to-end measurements.

Implementation example

This runs as shown. A model of a single-threaded, non-preemptive executor with a 100 Hz timer and an 80 ms heavy callback sharing the thread, against the same system with the heavy callback moved to a separate thread. It reports the timer’s jitter. ($A$ model of the mechanism, not rclcpp code.)

import numpy as np
def timer_jitter(shared, T=0.010, C_timer=0.0005, C_heavy=0.080, T_heavy=0.250, H=30.0):
    # non-preemptive single thread: each callback runs to completion. Returns the start latency of every timer firing.
    t = 0.0; lat = []; next_timer = 0.0; next_heavy = 0.05; free = 0.0
    while next_timer < H:
        if shared and next_heavy <= next_timer and free <= next_heavy + 1e-12 or (shared and next_heavy <= free and next_heavy <= next_timer):
            start = max(free, next_heavy); free = start + C_heavy; next_heavy += T_heavy; continue
        start = max(free, next_timer); lat.append(start - next_timer); free = start + C_timer; next_timer += T
    return np.array(lat)
a = timer_jitter(True); b = timer_jitter(False)
print(f"timer start latency, shared thread: p99 {np.percentile(a, 99) * 1e3:5.1f} ms max {a.max() * 1e3:5.1f} ms ; separate thread: max {b.max() * 1e3:5.2f} ms")
assert a.max() > 0.06 and b.max() < 1e-9

The corresponding structure in rclpy: a reentrant callback group and a multi-threaded executor, with best-effort QoS for the sensor stream (check the API of your ROS 2 distribution).

import rclpy
from rclpy.node import Node
from rclpy.callback_groups import ReentrantCallbackGroup, MutuallyExclusiveCallbackGroup
from rclpy.executors import MultiThreadedExecutor
from rclpy.qos import QoSProfile, ReliabilityPolicy, HistoryPolicy
from sensor_msgs.msg import Image, Imu

class Perception(Node):
    def __init__(self):
        super().__init__('perception')
        qos = QoSProfile(reliability=ReliabilityPolicy.BEST_EFFORT, history=HistoryPolicy.KEEP_LAST, depth=2)    # fresh and incomplete beats stale and complete
        self.heavy_group = MutuallyExclusiveCallbackGroup()      # image processing: one at a time
        self.fast_group = ReentrantCallbackGroup()               # IMU and timers: never wait behind images
        self.create_subscription(Image, 'camera/image', self.on_image, qos, callback_group=self.heavy_group)
        self.create_subscription(Imu, 'imu/data', self.on_imu, qos, callback_group=self.fast_group)
    def on_image(self, msg): pass       # heavy work here, or hand it to a worker thread
    def on_imu(self, msg): pass         # short and bounded

rclpy.init(); node = Perception(); ex = MultiThreadedExecutor(num_threads=3); ex.add_node(node); ex.spin()

Common engineering mistakes

Failure modes

Debugging approach

1. Log the start time and duration of each callback.
2. Use ros2 topic delay or header-stamp comparisons for message age.
3. Trace with tools such as ros2_tracing to see executor activity.
4. Change one thing at a time: thread count, group type, QoS depth.

Practical exercises

1. Measure the duration of every callback in a node of yours and list the three longest.
2. Move the longest callback to its own executor and compare the timer jitter.
3. Set the subscription depth to 1, 5 and 50 for a camera stream and measure data age.
4. Explain why a service call inside a mutually exclusive callback can deadlock.

Interview questions

Write an answer in your own words, then reveal the model answer. Nothing is sent anywhere.

References and further reading

Primary papers, official documentation and standards. Each claim above that is specific to a tool or version points here.

• ROS 2: Executors
• ROS 2: Quality of Service settings
• ros2_tracing: tracing tools for ROS 2
• ROS 2 Humble: Real-time programming tutorial

What’s next