Benchmarking the complete system

1. Engineering problem

System performance varies from run to run with scheduling, thermal state and the environment. A single measurement cannot show whether a change helped. Repeated runs with intervals can.

2. When is this needed?

• Comparing configurations or software versions.
• Before and after an optimisation.
• Writing a performance requirement.

3. Selection and design decisions

Run each configuration at least ten times under the same conditions (same data, warm start, same thermal state), compute the mean, standard deviation and a confidence interval (for ten runs, the 95 % interval half-width is 2.262 × s ÷ √10, from the t distribution with 9 degrees of freedom), and report the effect size (difference in pooled standard deviations). Record the software versions, parameters, hardware, power mode and ambient conditions.

4. Advantages and disadvantages

• More runs narrow the interval and cost time.
• Fixed workloads are repeatable; live scenarios are realistic.
• Throttled hardware (thermal) changes results during a long benchmark: control it.

5. Requirements and constraints

• Same conditions across configurations.
• Randomisation of run order to avoid drift effects.
• Raw results saved with the configuration.

6. Architecture

workload + configuration ─▶ N runs ─▶ metrics ─▶ mean, interval, effect size ─▶ decision

7. Implementation

1. Fix the workload, hardware state and metrics.
2. Run each configuration ten or more times in random order.
3. Compute the statistics (below).
4. Decide using the interval and the effect size, not one run.
5. Record everything with the versions.

import statistics as st
def summary(xs):
    m, s = st.mean(xs), st.stdev(xs); n = len(xs)
    half = 2.262 * s / n ** 0.5 if n == 10 else None                  # t value for 9 degrees of freedom, 95 %
    return m, s, half
a = [41.2, 40.8, 42.5, 41.0, 41.9, 40.5, 42.1, 41.3, 41.7, 40.9]       # ms, configuration A, ten runs
b = [38.9, 39.4, 38.2, 39.9, 38.5, 39.1, 38.8, 39.6, 38.4, 39.0]       # ms, configuration B
for name, xs in (("A", a), ("B", b)):
    m, s, h = summary(xs)
    print(f"config {name}: mean {m:.2f} ms, std {s:.2f}, 95 % interval {m - h:.2f} to {m + h:.2f} ms")
ma, mb = st.mean(a), st.mean(b)
pooled = ((st.variance(a) + st.variance(b)) / 2) ** 0.5
print(f"difference {ma - mb:.2f} ms ({(ma - mb) / ma * 100:.1f} %), effect size {(ma - mb) / pooled:.1f} pooled standard deviations")
print("the intervals do not overlap:", summary(a)[0] - summary(a)[2] > summary(b)[0] + summary(b)[2])

Printed (checked by the site’s tests):

config A: mean 41.39 ms, std 0.64, 95 % interval 40.93 to 41.85 ms
config B: mean 38.98 ms, std 0.54, 95 % interval 38.59 to 39.37 ms
difference 2.41 ms (5.8 %), effect size 4.1 pooled standard deviations
the intervals do not overlap: True

Reading it: ten runs of configuration A give a mean of 41.39 ms (95 % interval 40.93 to 41.85) and ten runs of B give 38.98 ms (38.59 to 39.37); the 2.41 ms (5.8 %) difference is 4.1 pooled standard deviations, and the intervals do not overlap, so B is faster under these conditions. The data are illustrative.

8. Real examples

• Nightly benchmark: a recorded bag replayed ten times per commit; the dashboard shows the interval, and a regression fails the build.

9. Failure scenarios

• Comparing two single runs.
• Running A while the machine is cold and B when it is throttled.
• Not recording the configuration, so the result cannot be reproduced.

10. Troubleshooting

1. If the spread is large, find the source of variation (background load, thermal, scheduling).
2. If the intervals overlap, run more or accept no difference.
3. Check run order effects.

11. Validation

• The calculation prints the lines shown (executed by the site tests).
• Results are reproducible on a second day with the same configuration.

12. Engineering checklist

• Workload fixed
• Ten or more runs
• Random order
• Interval and effect size
• Versions recorded
• Thermal state controlled

13. Further references

• The confidence-interval formula is standard statistics (Student’s t distribution); see statistics texts.

Related on FixTheRobot