Associating detections with LiDAR clusters

1. Engineering problem

A camera knows what an object is; a LiDAR knows where it is. Associating the two gives a labelled 3D object. The simplest method projects each LiDAR cluster into the image and checks which detection it falls inside.

2. When is this needed?

• Building a camera–LiDAR detection pipeline.
• When labels land on the wrong object.
• When objects overlap in the image.

3. Selection and design decisions

Common ways to associate: centroid-in-box, overlap of the cluster’s projected hull with the box (intersection over union), and a global assignment (Hungarian algorithm) when many objects are present. The result should include “no match” and “ambiguous” as outcomes.

4. Advantages and disadvantages

• Looser boxes match more and mislabel more; tighter miss near edges.
• Ground and background points in a cluster pull the centroid; remove ground first.
• Association quality depends on calibration and time offset.

5. Requirements and constraints

• Clusters and image are close in time.
• The extrinsic error in pixels is smaller than the typical box margin.
• Ambiguous and unmatched cases are explicit outputs.

6. Architecture

detections (boxes) ─┐
                      ├─▶ association (projected centroid or hull vs box) ─▶ labelled 3D objects
clusters ─▶ project ─┘

7. Implementation

1. Remove ground points and cluster the remaining cloud.
2. Project each cluster centroid (or hull) into the image with the calibration.
3. Test containment or overlap against each box (below).
4. Resolve ambiguous cases with depth and size checks.
5. Publish matched objects, and keep unmatched clusters as unlabelled obstacles.

def contains(box, u, v):
    x0, y0, x1, y1 = box
    return x0 <= u <= x1 and y0 <= v <= y1
boxes = {"person": (100, 100, 200, 300), "car": (300, 150, 500, 280)}
clusters = {"c1": (150, 210), "c2": (400, 200), "c3": (260, 200), "c4": (160, 150)}   # cluster centroids projected into the image (u, v)
for name, (u, v) in clusters.items():
    hits = [b for b, box in boxes.items() if contains(box, u, v)]
    print(name, "->", hits[0] if len(hits) == 1 else ("ambiguous: " + ", ".join(hits) if hits else "no detection"))

Printed (checked by the site’s tests):

c1 -> person
c2 -> car
c3 -> no detection
c4 -> person

Reading it: c1 lies inside the person box and c2 inside the car box; c3 lies in neither, so it is an obstacle with no label; c4 is also inside the person box (the box contains two clusters, which a global assignment or a depth check must settle). Boxes and points are illustrative.

8. Real examples

• Pedestrian detection: a camera detector provides the “person” label and the matched cluster provides the distance used for braking.

9. Failure scenarios

• A label assigned to the background cluster behind a person.
• A calibration error larger than the box margin shifts every association.
• A moving object matched to a stale cluster because of a time offset.

10. Troubleshooting

1. Overlay the projected clusters and boxes on one image.
2. Count unmatched and ambiguous cases per frame.
3. Check the time offset and extrinsics if associations shift consistently.

11. Validation

• The example prints the lines shown (executed by the site tests).
• Association accuracy is measured on labelled frames and meets the requirement.

12. Engineering checklist

• Ground removed
• Calibration error in pixels known
• Time offset checked
• Unmatched and ambiguous handled
• Depth or size check
• Accuracy measured on labelled data

13. Further references

• ROS 2 sensor_msgs/CameraInfo and Image (Jazzy): ros2/common_interfaces, read 2026-11-05.
• The Hungarian algorithm is standard (for example scipy.optimize.linear_sum_assignment); this guide’s example uses a simple containment test.

Related on FixTheRobot