Camera–LiDAR object detection and association

1. Engineering problem

A camera detector gives a class and a 2D box; a LiDAR cluster gives a 3D position and extent. Associating them gives labelled 3D objects, which a planner can treat differently (a person versus a pole).

2. When is this needed?

• After calibration and synchronisation are validated.
• When the planner needs object classes.
• When labels are wrongly assigned.

3. Selection and design decisions

The usual method: project each cluster’s 3D bounding box into the image (see the projection guide), compute the 2D box, and compare it with each detection’s box by intersection over union (IoU = overlap area ÷ union area). Accept matches above a threshold (0.5 is a common starting value) and keep the rest as unlabelled obstacles.

4. Advantages and disadvantages

• High IoU threshold reduces wrong labels and loses labels when calibration is imperfect.
• Use depth and size checks to reject implausible matches.
• Matching only at the same stamp is more accurate than at the nearest frame.

5. Requirements and constraints

• Calibration error in pixels is small against the box size.
• Cluster and image are time-aligned within the requirement.
• Unlabelled clusters are never dropped silently.

6. Architecture

clusters ─▶ project 3D box ─▶ 2D boxes ┐
                                         ├─▶ IoU matrix ─▶ assignment ─▶ labelled objects + unlabelled obstacles
detections (class, 2D box) ──────────┘

7. Implementation

1. Project each cluster’s 3D box into the image.
2. Compute the IoU against every detection (below).
3. Assign matches above the threshold.
4. Keep unmatched clusters as unlabelled obstacles.
5. Publish with the cloud stamp, class and confidence.

def iou(a, b):
    x0, y0 = max(a[0], b[0]), max(a[1], b[1]); x1, y1 = min(a[2], b[2]), min(a[3], b[3])
    inter = max(0, x1 - x0) * max(0, y1 - y0)
    area = lambda r: (r[2] - r[0]) * (r[3] - r[1])
    return inter / (area(a) + area(b) - inter)
det = {"person": (100, 100, 200, 300), "car": (300, 150, 500, 280)}
proj = {"c1": (110, 120, 190, 290), "c2": (320, 160, 470, 270), "c3": (230, 150, 280, 250), "c4": (90, 90, 210, 310)}
for c, box in proj.items():
    scores = {n: round(iou(box, d), 2) for n, d in det.items()}
    best = max(scores, key=scores.get)
    print(c, scores, "->", best if scores[best] >= 0.5 else "unlabelled")

Printed (checked by the site’s tests):

c1 {'person': 0.68, 'car': 0.0} -> person
c2 {'person': 0.0, 'car': 0.63} -> car
c3 {'person': 0.0, 'car': 0.0} -> unlabelled
c4 {'person': 0.76, 'car': 0.0} -> person

Reading it: c1 and c4 both overlap the person box (IoU 0.68 and 0.76), so one of them needs the depth or size check; c2 matches the car (0.63); c3 overlaps nothing and stays an unlabelled obstacle. Boxes are illustrative.

8. Real examples

• Pedestrian-aware navigation: clusters labelled person get a larger inflation radius than static obstacles.

9. Failure scenarios

• A calibration error that makes every IoU low, so nothing is labelled.
• A label given to background clusters behind the object.
• Dropping unlabelled clusters and driving into them.

10. Troubleshooting

1. Overlay the projected clusters and the detector boxes on one image.
2. Count unmatched on both sides per frame.
3. Check the time offset and extrinsics when matches shift consistently.

11. Validation

• The example prints the lines shown (executed by the site tests).
• Association accuracy on labelled frames meets the requirement (see the validation guide).

12. Engineering checklist

• Calibration error in pixels known
• Time alignment checked
• IoU threshold chosen from data
• Unlabelled clusters kept
• Ambiguity handled
• Accuracy measured

13. Further references

• ROS 2 sensor_msgs/PointCloud2 (Jazzy): PointCloud2.msg, read 2026-11-06.
• See also the association guide and the projection guide.

Related on FixTheRobot