P7-T6 — Occlusion, Identity Switches and Missed Detections

Learning objectives

After this lesson, you should be able to:

• Explain how crossing paths, occlusion and missed detections each cause identity errors.
• Compute the growth of position uncertainty and of the gate while a track coasts.
• Name the cues beyond position that reduce identity switches, and their limits.
• Choose metrics and tests that expose these failures instead of averaging them away.

The idea: ambiguity is the enemy, not noise

Noise makes a tracker wobble; ambiguity makes it wrong. If every detection has exactly one plausible track, small errors do no harm. Trouble starts when two readings of the scene fit the data almost equally well, and the tracker has to pick one. All three situations below are ambiguity.

Crossing paths

When two objects pass close, their detections fall inside each other’s gates. The tracker can assign them either way. If it chooses the way that fits the positions better but is the wrong one, the two identities have swapped. Using velocity helps: objects that continue in straight lines can be told apart by the direction they were going, and constant-velocity prediction already does this. It fails when the objects meet and then turn, stop, or move together.

Occlusion

When an object is hidden, the detector has nothing to say. The track coasts: it predicts, its uncertainty grows, and the gate widens. If the object emerges within the gate, the track picks it up and keeps its identity. If another object is in the widened gate, it may capture the track. If the object turned or stopped while hidden, the prediction is wrong and the identity is lost.

Missed detections

The detector sometimes misses a visible object, especially a small, far, or partly hidden one. A tracker that deletes on the first miss breaks identities at every blink. One that coasts for a few frames absorbs most of them. The risk of coasting is that the real object has left and the track follows nothing.

Merged and split detections

Two objects very close in the image can be reported as one box, and one large object as two. A merge makes one track starve while the other gets a detection that belongs to both; a split creates a second track on the same object. These errors originate in the detector (Module 6) and the tracker can only handle them by checking size and shape consistency.

What helps

1. A better motion model, so that the prediction through the gap is closer to the truth.
2. Appearance. A descriptor of what the object looks like (colour histogram, learned embedding) distinguishes objects that are near each other. It works if they look different and the view does not change much; it fails for identical objects such as boxes of the same type.
3. Class and size consistency. A person track should not be given a car detection, and an object’s box area changes slowly.
4. Global reasoning. Optimal assignment (Lesson P7-T2), several hypotheses kept alive until the ambiguity resolves, or reasoning about which regions can hide objects.
5. Other sensors. A lidar sees through the shadow a camera cannot (Module 8).

The mathematics: uncertainty during a gap

While a track coasts it receives no measurement. Its position variance grows because the velocity is not perfectly known and the object can accelerate. Let the velocity have standard deviation σ_v at the last update and the process noise have acceleration standard deviation σ_a. After k coasting steps of length Δt, the position standard deviation is roughly (a simplified bound that treats the three error sources as independent)

σ_p0

the position standard deviation at the last update.

k Δt σ_v

the position error caused by the velocity error alone: it grows linearly with the gap.

k² Δt² σ_a / 2

the position error caused by random acceleration: it grows with the square of the gap, so long gaps are dominated by it.

The statistical gate (Lesson P7-T2) is a multiple of σ_p, so the gate radius grows by the same law. A widening gate is what lets a coasting track find its object again, and it is also what lets the wrong object in.

Crossing ambiguity. Two objects cross with relative speed u at closest distance d_min. They are within one gate radius g of each other for the duration

g

the gate radius.

d_min

the closest distance between the paths.

u

the relative speed.

T_amb

the length of the ambiguous stretch. Only if d_min is smaller than g are the objects ever confusable; a larger g or a slower approach make the stretch longer.

Metrics. The error counts are the identity switches (IDSW), the fragmentations (a trajectory covered by several tracks), the misses (FN) and false alarms (FP); these combine into MOTA, and identity-focused measures are IDF1 and HOTA (Lesson P7-T2). Each failure should also be counted separately on the scenes that provoke it, because an average over a long easy sequence hides them.

Worked examples: a gap and a crossing

Coasting through an occlusion. A track was last updated with position σ_p0 = 1.5, velocity σ_v = 0.3 units per frame, and σ_a = 0.2 units per frame². The object disappears behind an occluder for k = 6 frames (Δt = 1).

The standard deviation is 4.3, almost three times the 1.5 at the last update. A 99% gate (3.03σ for a chi-square limit of 9.21 with equal x and y deviations) has a radius of about 13 units. The same track, one frame after the update, would have σ_p = sqrt(1.5² + 0.3² + 0.1²) = 1.53 and a gate radius of 4.6. The wide gate lets the track find the emerging object, and any other object within 13 units.

A crossing. Two objects approach at a relative speed u = 4 units per frame; their paths pass at d_min = 3. The gate radius is g = 8.

For about 3.7 frames the two objects are inside each other’s gate. A tracker that matches by distance alone can swap them if one of the detections is missed in that stretch, or if the noise changes the order of the distances. If the paths pass at d_min = 9, larger than the gate, they never share a gate and the crossing is harmless. A smaller gate (with a good motion model) removes the ambiguity here: with g = 4 and d_min = 3 the stretch is 2 sqrt(16 − 9) / 4 = 1.3 frames.

Missed detections. At a detection probability of 0.9 per frame, the probability of at least one miss in a ten-frame run is 1 − 0.9¹⁰ = 0.65. A tracker that deletes on the first miss would break the track of the average object in a ten-frame sequence about two times out of three. Coasting makes this ordinary event harmless.

Engineering implications and limitations

Test with hard scenes on purpose. Build or collect sequences with crossings, occlusions of different lengths and detector drop-outs, and report the metrics for each. The labs do this with simulated occluders and misses.

Report uncertainty on identity. When two tracks are ambiguous, say so (a flag, or a probability that identities have swapped) and let the planner treat both as possible. Silently choosing is the worse option.

Use the scene. Where occluders are known (a pillar on the map, a shelf), a track predicted behind one should be kept; one predicted in open view and not seen should be dropped sooner.

Re-identification is a separate module. Matching an emerging object to an old track by appearance is its own problem with its own errors. If you add it, test it separately and look at its false-match rate: a confident wrong re-identification is worse than a new identity.

Do not hide detector problems with tracker logic. If a class is missed 40% of the time, coasting will keep tracks alive and will hide that the detector needs work. Keep monitoring the detector’s own recall (Module 6).

Know when to give up. After a long gap, a lost track is better ended and reported as lost than extended on a guess. The consumer should be able to see the age since the last detection.

Common misconceptions

Engineering takeaways

1. The three situations are crossing paths, occlusion and missed detections; all are ambiguity, not noise.
2. While a track coasts its position uncertainty grows (in the example from 1.5 to 4.3 over six frames) and the gate widens with it.
3. Two paths are confusable only if their closest distance is below the gate radius; the ambiguous stretch is 2√(g² − d²)/u frames.
4. At 90% detection probability, 65% of ten-frame runs contain at least one miss; tracking must tolerate this.
5. Test and report on hard scenes separately, report ambiguity to the consumer, and keep checking the detector itself.

References and further reading

Textbooks, papers and a benchmark for the concepts named. The uncertainties, speeds and distances in the examples are illustrative.

• Bar-Shalom, Y., Li, X.-R. and Kirubarajan, T. (2001). Estimation with Applications to Tracking and Navigation. Wiley: gating, track maintenance and multiple hypothesis tracking.
• Blackman, S. (2004). Multiple Hypothesis Tracking for Multiple Target Tracking. IEEE Aerospace and Electronic Systems Magazine.
• Wojke, N., Bewley, A. and Paulus, D. (2017). Simple Online and Realtime Tracking with a Deep Association Metric. ICIP: appearance cues to reduce identity switches.
• Bernardin, K. and Stiefelhagen, R. (2008). Evaluating Multiple Object Tracking Performance: The CLEAR MOT Metrics. EURASIP Journal on Image and Video Processing.
• Dendorfer, P. et al. (2021). MOTChallenge: A Benchmark for Single-Camera Multiple Target Tracking. International Journal of Computer Vision.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next, Module 8: combining sensors, so that the strengths of one cover the weaknesses of another.

Related content

Nothing below is required to finish this lesson.

• Before
  P7-T5: Track Initiation, Persistence and Deletion

  Design the track lifecycle with confirmation and deletion rules.

• Lab
  P7-L4: Recover from Temporary Occlusion

  Keep an identity through a gap with prediction and a lifecycle policy.