UAV Systems Engineering · M7-L3 · Hands-on lab (runs on your machine)
Profile perception pipelines on an onboard computer
Find the stage that owns your latency tail, with numbers.
What you will do
Take a pipeline that you actually fly (camera capture, pre-processing, detector or VIO front end, back end, publisher) and profile it with a repeatable method: stage timers that write to a CSV, a tegrastats recording running at the same time, a long enough run to reach thermal steady state, and an analysis that reports percentiles per stage and the correlation of latency with temperature, clocks and utilisation. The reference script analyses synthetic recordings so that you can test the analysis; the format of tegrastats lines differs between JetPack versions and modules, so adapt the parser to your output. No vendor figures are used or implied.
Before you start
- Lessons M7-T2 (Jetson platforms), M7-T3 (workload management), M7-T6 (latency).
- A Jetson-class module with your pipeline, or synthetic data.
- Python 3 with NumPy.
Stage timers
Add a timer around each stage, using a monotonic clock, and write one CSV row per frame with the frame id and the duration of each stage. Keep the instrumentation cheap (a clock read and an append to an in-memory list, flushed at the end).
import time, csv
class StageTimer:
def __init__(self, stages): self.stages = stages; self.rows = []; self.cur = {}
def start(self, frame_id): self.cur = {'frame': frame_id}; self._t = time.monotonic_ns()
def mark(self, stage):
now = time.monotonic_ns(); self.cur[stage] = (now - self._t) / 1e6; self._t = now # milliseconds since the previous mark
def end(self): self.rows.append(self.cur)
def dump(self, path):
with open(path, 'w', newline='') as f:
w = csv.DictWriter(f, fieldnames=['frame'] + self.stages); w.writeheader(); w.writerows(self.rows)
# in the pipeline loop: timer.start(i); grab(); timer.mark('capture'); preprocess(); timer.mark('pre'); infer(); timer.mark('infer'); post(); timer.mark('post'); timer.end()In a second terminal record the system, with a timestamp per line from the shell (check the exact tegrastats options of your JetPack version):
sudo tegrastats --interval 500 | while read -r l; do echo "$(date +%s.%N) $l"; done > tegrastats.logAnalysis script (synthetic data)
import numpy as np, re
rng = np.random.default_rng(22)
# ---- synthetic recording: 10 minutes at 30 fps; the GPU temperature rises and the clock throttles after minute 6
n = 18000; t = np.arange(n) / 30.0
temp = 45 + 38 * (1 - np.exp(-t / 150.0)); throttled = temp > 80; freq = np.where(throttled, 620, 900)
stages = {'capture': 3.0 + 0.4 * rng.standard_normal(n), 'pre': 2.0 + 0.3 * rng.standard_normal(n),
'infer': 14.0 * (900 / freq) + 1.2 * rng.standard_normal(n), 'post': 4.0 + 0.8 * rng.gamma(2.0, 1.0, n) / 2}
stages['post'][rng.random(n) < 0.004] += 40.0 # rare stall in the post-processing stage
# ---- end of synthetic block (load your CSV and tegrastats log here instead)
total = sum(stages.values())
print(f"{'stage':>8} {'p50':>7} {'p95':>7} {'p99':>7} {'max':>7} (ms)")
for k, v in stages.items(): print(f"{k:>8} {np.percentile(v, 50):7.2f} {np.percentile(v, 95):7.2f} {np.percentile(v, 99):7.2f} {v.max():7.2f}")
print(f"{'total':>8} {np.percentile(total, 50):7.2f} {np.percentile(total, 95):7.2f} {np.percentile(total, 99):7.2f} {total.max():7.2f}")
# which stage owns the tail? compare each stage's contribution on the frames above the total's p99
tail = total > np.percentile(total, 99); share = {k: float(v[tail].mean() - v.mean()) for k, v in stages.items()}
print("excess in the tail frames (ms):", {k: round(s, 2) for k, s in share.items()})
# thermal effect: compare the first and last minutes
a, b = t < 60, t > 540
print(f"first minute p50 total {np.percentile(total[a], 50):.1f} ms at {temp[a].mean():.0f} C; last minute {np.percentile(total[b], 50):.1f} ms at {temp[b].mean():.0f} C, clock {freq[b].mean():.0f} MHz")
print(f"correlation of 'infer' latency with temperature: {np.corrcoef(stages['infer'], temp)[0, 1]:.2f}")
assert max(share, key=share.get) in ('post', 'infer') and np.percentile(total[b], 50) > 1.2 * np.percentile(total[a], 50)
assert np.corrcoef(stages['infer'], temp)[0, 1] > 0.5Experiments
- Run the full pipeline for at least 20 minutes in its enclosure, with the airflow it will have in flight. Report the first-minute and last-minute p50 and p99 latency per stage.
- Repeat with the clocks pinned (
jetson_clocks) and compare the variance and the temperature. - Run a second, competing workload (for example logging and video encoding) and report the change in the p99 of each stage.
- Identify the stage that owns the tail and propose one change (bounded queue, core pinning, a smaller model, a lower input resolution). Re-measure and report the effect.
How to know it worked
- The analysis script runs and its assertions pass on the synthetic data.
- You have a per-stage percentile table for a recording of your own pipeline of at least 20 minutes.
- You can state which stage owns the tail and show the evidence, and the effect of one change.
If it does not work
| Symptom | Likely cause | What to check |
|---|---|---|
| tegrastats lines do not parse | The format differs between JetPack versions | Print a line, adapt the regular expressions; keep the wall-clock stamp |
| Timers add overhead | Instrumentation inside a hot loop | Use monotonic_ns and an in-memory list; compare with timing off |
| No throttling observed | The run was too short or the enclosure too cool | Run 20+ minutes at the hot case |
| Stages do not sum to the total | Queue time between stages is not included | Add timestamps at enqueue and dequeue |
References
Record your progress
Saved in this browser. This lab runs on your own machine, so the site cannot check it for you.