Engineering Guides · Module 15

Diagnosing simulation-to-hardware differences

Compare means and spreads of the same test in simulation and on hardware. A simulation that is too clean is the usual finding.

Intermediate
Module 15: Simulation-to-Real Integration
Written for ROS 2 Jazzy where ROS 2 is used
Sources checked 2026-11-06

Assumptions: Numbers in the examples are illustrative and the code is executed by the site’s tests. Facts taken from a cited source are marked; other statements are general engineering knowledge and must be confirmed on the datasheet of the part you use.

1. Engineering problem

“It worked in simulation” is a symptom, not a diagnosis. The cause is a parameter or effect the simulation lacks. Running the same test in both and comparing the numbers shows which.

2. When is this needed?

A controller or algorithm works in simulation and fails on hardware.
When validating a simulator.
After a hardware change.

3. Selection and design decisions

Run the same small test (step response, stationary recording, fixed trajectory) in both worlds and compare the mean (bias or systematic difference) and the spread (noise, jitter) of the result. A spread much smaller in simulation means missing noise or disturbance; a different mean means a wrong model parameter (gain, radius, delay).

Overview
Observation	Likely missing effect	Check
Real spread much larger	Noise, vibration, quantisation	Stationary recording
Real mean shifted	Gain, radius, bias, mass	Calibration tests
Real slower to respond	Delay, acceleration limit, filtering	Latency measurements
Real response has steps	Quantisation, static friction	Encoder resolution, friction test
Failures only on hardware at certain times	Timing jitter, CPU load	Profiling

4. Advantages and disadvantages

Many repeats give stable statistics and cost time.
Adding every effect costs modelling; add the one that explains the gap.
Always keep one comparison test as a simulator regression.

5. Requirements and constraints

Same test, same inputs, same analysis in both worlds.
Enough repeats to estimate the spread (at least several).
Differences recorded with their explanation.

6. Architecture

same test ─▶ simulation runs ─┐
                                  ├─▶ compare mean and spread ─▶ missing effect ─▶ add to the model
         hardware runs ───────────┘

7. Implementation

Choose a small repeatable test.
Run it several times in simulation and on hardware.
Compare means and spreads (below).
Identify the missing effect and add it to the simulation.
Repeat until the statistics agree within tolerance.

import statistics as st
def compare(sim, real, name, tol_mean=0.1, tol_std=0.5):
    ms, mr, ss, sr = st.mean(sim), st.mean(real), st.stdev(sim), st.stdev(real)
    flags = []
    if abs(ms - mr) > tol_mean * abs(mr): flags.append(f"mean differs ({ms:.3f} vs {mr:.3f})")
    if abs(ss - sr) > tol_std * sr: flags.append(f"spread differs ({ss:.3f} vs {sr:.3f}): the simulation is {'too clean' if ss < sr else 'too noisy'}")
    return f"{name}: " + ("; ".join(flags) or "consistent")
rise_sim, rise_real = [0.30, 0.31, 0.29, 0.30, 0.30], [0.42, 0.55, 0.38, 0.61, 0.47]
yaw_sim, yaw_real = [0.001, -0.001, 0.0, 0.002, -0.002], [0.010, -0.014, 0.006, 0.021, -0.017]
print(compare(rise_sim, rise_real, "wheel step rise time (s)"))
print(compare(yaw_sim, yaw_real, "stationary gyro z (rad/s)", tol_mean=1.0))

Printed (checked by the site’s tests):

wheel step rise time (s): mean differs (0.300 vs 0.486); spread differs (0.007 vs 0.094): the simulation is too clean
stationary gyro z (rad/s): spread differs (0.002 vs 0.016): the simulation is too clean

Reading it: the simulated rise time has almost no spread (0.007 s) against 0.094 s on hardware, and a mean of 0.30 s against 0.486 s: the simulation is too clean and too fast, so delay and variability are missing. The stationary gyro in simulation is 8 times quieter than the real one. The values are illustrative.

8. Real examples

Wheel step test: five steps on the robot and five in simulation, showing a missing 100 ms delay.

9. Failure scenarios

Comparing a single run.
Adding noise without checking its level against the real data.
Tuning on the hardware difference instead of fixing the model.

10. Troubleshooting

Check the test is identical in both worlds.
Compare more runs.
Look at the shape, not only the mean.

11. Validation

The comparison prints the lines shown (executed by the site tests).
After the fix the simulated statistics match the hardware within the stated tolerance.

13. Further references

The comparison is elementary statistics; for rigorous comparison use hypothesis tests suited to your data.