P7-T1 — Detection Across Consecutive Frames

Learning objectives

After this lesson, you should be able to:

• Distinguish a detection (an observation in one frame) from a track (an estimate of an object over time).
• Describe the tracking-by-detection pipeline and its main failure modes.
• Use the frame rate, object speed and spacing to decide whether nearest-neighbour association can work.
• Compute the error of a velocity estimated from two noisy positions and explain why filtering is needed.

The idea: observations are not objects

A detection is an observation: in frame k something appears at this position with this size and class. A track is a belief about an object over time: object 7 has been moving to the right at about 1.4 m/s for the last two seconds. Detections come out of the sensor; tracks are built from them and carry a state, an identity and a history.

Why detections alone are not enough

A planner that avoids obstacles can treat each frame separately for a static world. For moving objects it needs three things the detector does not give.

1. Identity. “The pedestrian I was yielding to” must remain the same object from frame to frame, or the planner forgets that it has already decided to yield.
2. Velocity. A position says where; a velocity says where next. A planner that sees only positions plans for a world that is frozen.
3. Persistence. A pedestrian who is missed for two frames, or goes behind a pole, has not disappeared. A system that forgets objects when the detector blinks cannot be trusted near people.

The tracking-by-detection pipeline

The most common way to build a tracker uses the detector as given and adds a layer above it. In each new frame the tracker (1) predicts where each existing track should be now, (2) associates the new detections with the predicted tracks, (3) updates each matched track with its detection, (4) creates tracks for detections nobody claimed, and (5) deletes tracks that have gone unmatched for too long. Lessons P7-T2 to P7-T5 take these steps one by one. Everything in this module is a version of this loop.

Where it goes wrong

The loop makes errors of five kinds. Two objects close together can swap identities. A track can be broken by a missed detection or an occlusion, so the object returns with a new identity. A false alarm can become a ghost track. Two objects can be reported as one when their detections merge. And an object can be lost for good when the prediction drifts away from where it really went. A tracker is evaluated by how often these happen (Lesson P7-T6).

The mathematics: how far, how noisy, how often

Motion between frames. An object moving at speed v travels, between two frames Δt apart,

v

the object’s speed, in the units of the position (metres per second, or pixels per second).

Δt, f

the time between frames and the frame rate; f = 1/Δt.

d

the displacement between consecutive frames. A tracker’s gate has to cover it.

Nearest-neighbour association asks that the right detection be the closest one to the track’s last position. If the neighbouring object is a distance s away, the nearest detection to an object’s old position is the object’s own new detection only if d < s/2, because beyond that the neighbour’s old position is closer than the object’s new one in the worst case. This gives the feasibility limit

s

the distance between two objects (the smallest spacing in the scene).

f

the frame rate needed so that an object moves less than half the spacing between frames.

Prediction with a motion model (Lesson P7-T3) lifts this limit, because the tracker looks near where the object should now be rather than where it was.

Velocity from noisy positions. The crudest velocity estimate is a difference of two measured positions: v̂ = (z_k − z_k−1) / Δt. If each position has independent noise of standard deviation σ, the difference has standard deviation √2 σ, and so

σ

standard deviation of the position measurement, in position units.

σ_v

standard deviation of the velocity estimate. Because it is divided by Δt, a high frame rate makes it worse, not better.

Missed detections. If the detector finds a visible object in each frame independently with probability p, the chance of m misses in a row is (1 − p)^m. Real misses are not independent (an object that is hard to see in one frame is hard in the next), so this is a lower bound on how long gaps last.

A worked example: pedestrians and frame rate

A robot watches pedestrians walking at v = 1.5 m/s. In a queue they stand s = 1.2 m apart.

Can nearest-neighbour association work? The condition is d < s/2 = 0.6 m, that is Δt < 0.6 / 1.5 = 0.4 s, a frame rate above 2.5 frames per second. The table shows the displacement per frame at several rates.

At 2 Hz the pedestrian moves farther than half the spacing; the nearest old position to a new detection can belong to the person behind. A high frame rate makes association easy, but it has a cost.

The velocity problem. The detector localises each person to σ = 0.05 m. At 10 Hz, Δt = 0.1 s:

The true speed is 1.5 m/s, so a two-point velocity has an error of nearly half the speed: a pedestrian standing still would often appear to be walking at 0.7 m/s in some direction. At 30 Hz the noise would be 0.05 × √2 / 0.033 = 2.1 m/s, larger than the speed itself. Differencing positions is a poor velocity estimator whatever the frame rate; a filter that combines many frames (Lessons P7-T3 and P7-T4) does far better.

Misses. With p = 0.9 a single miss has probability 0.1 per frame. At 30 Hz that is three misses per second per object, so a tracker that deletes a track on its first miss would break an identity several times a second. Three misses in a row have probability 0.1³ = 0.001 per frame, which is 0.03 per second at 30 Hz, about once in 33 s, if the misses were independent. Waiting for three misses before deleting removes nearly all such breaks, at the price of keeping an object that really left for three more frames.

Engineering implications and limitations

Choose the frame rate for association and for velocity. Association wants small displacements, so a higher rate; velocity from raw differences wants a larger Δt. A filter resolves the conflict, so run the detector at the rate the application needs, not at the rate a velocity estimate would need.

Timestamp every detection. The prediction depends on the real time between frames. A dropped frame or a jitter in capture time is a motion the tracker will misinterpret (Lesson P8-T3).

Keep the detector’s uncertainty. Give the tracker the detection’s covariance or at least its score and size. Small, low-score detections are noisier, and a tracker that treats all detections alike over-trusts the bad ones.

Log the raw detections. To debug a tracker you must replay it. Store the detections with timestamps; the tracker’s output alone does not show whether the detector or the association was at fault.

Do not hide detector flicker with tracking alone. A tracker can smooth over a miss, but a detector that blinks at 30% of the frames for a class of objects needs fixing at the source (Module 6).

Mind the labelling. The labs in this module simulate the detector’s positions with stated noise, misses and false alarms. They show how tracking behaves; they say nothing about the accuracy of any real detector.

Common misconceptions

Engineering takeaways

1. A detection is a single-frame observation; a track is an estimate of an object over time with identity, velocity and persistence.
2. Tracking by detection loops over predict, associate, update, create and delete.
3. Nearest-neighbour association needs d = v/f < s/2; at 2 Hz, 1.5 m/s pedestrians 1.2 m apart can swap, at 10 Hz they cannot.
4. A two-point velocity has noise √2σ/Δt: 0.71 m/s for σ = 0.05 m at 10 Hz. Use a filter.
5. Independent misses of probability 0.1 give runs of three with probability 0.001, but real misses are correlated.

References and further reading

Textbooks and a published baseline. The walking speed, spacing, localisation error and miss probability in the example are illustrative.

• Bar-Shalom, Y., Li, X.-R. and Kirubarajan, T. (2001). Estimation with Applications to Tracking and Navigation. Wiley: tracking terminology and the predict-associate-update loop.
• Bewley, A., Ge, Z., Ott, L., Ramos, F. and Upcroft, B. (2016). Simple Online and Realtime Tracking. ICIP: a tracking-by-detection baseline.
• Thrun, S., Burgard, W. and Fox, D. (2005). Probabilistic Robotics. MIT Press: filtering with noisy observations.
• Szeliski, R. (2022). Computer Vision: Algorithms and Applications, 2nd ed. Springer: motion and tracking.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: the central step of the loop, association: which detection belongs to which track, with gating, optimal assignment, and identity.

Related content

Nothing below is required to finish this lesson.

• Before
  P6-T6: False Positives, False Negatives and Domain Shift

  Explain error types and why performance drops when the scene changes.

• Lab
  P7-L1: Track Moving Objects Across Simulated Frames

  Link detections over time with nearest-neighbour association and see where it fails.