P8-T4 — Early Fusion, Late Fusion and Object-Level Fusion

Learning objectives

After this lesson, you should be able to:

• Describe early, late and object-level fusion and where in the pipeline each combines the data.
• Estimate the data volume at each fusion level.
• Compare the architectures in terms of information, alignment demand, modularity and failure behaviour.
• Choose an architecture for a given task and set of constraints.

The idea: when to merge

Think of two witnesses to an accident. You can ask each to describe what they saw and then compare the accounts (late fusion). You can have both look at the same photograph and agree on what it shows (early fusion). Or you can compare the conclusions each has already drawn (object level). The first loses detail, since each witness has already summarised. The second needs them to be looking at the same photograph in the same way.

Early fusion: merge the raw data

Raw data from both sensors are put into one representation before any detection: lidar points painted with the colours of the pixels they project onto, or a depth channel added to the image, or image features lifted into a common bird’s-eye grid together with lidar features. One model then sees everything. Strengths: it can use correlations between the sensors that earlier summaries would have lost (a faint edge in the image plus a few lidar points make an object neither sensor would report on its own). Weaknesses: it needs precise calibration and synchronisation, because the merge happens point by point; it needs high bandwidth; if one sensor fails, the model has to be trained to cope; and it is hard to inspect.

Late fusion: merge the detections

Each sensor has its own detector, and a fusion step merges the outputs, associating a camera box with a lidar cluster and combining their positions, sizes and classes. Strengths: each pipeline can be developed, tested and replaced on its own; the interface is small; a failed sensor simply stops contributing. Weaknesses: an object that neither detector finds alone is lost; the association step can fail; and the information about how sure each detector was has to be passed on explicitly.

Object-level fusion: merge the tracks

Each sensor’s detections are tracked separately (Module 7) and the fused scene is built from the track lists, often by a filter that treats each track as a measurement. Strengths: the smallest data, the easiest to run across several computers or vehicles, and the tracks already carry velocity and identity. Weaknesses: the tracks were formed without the other sensor’s help, so errors made in them are inherited; and correlations between the tracks’ errors (they share the same ego-motion) must be handled with care.

Learned fusion

Many current systems merge image and lidar features inside a neural network, often in a bird’s-eye-view grid, which blurs the line between early and late fusion. The questions remain the same: what is the alignment requirement, what happens if one sensor drops out, and how will you know when the combination is wrong. A learned fusion still needs calibrated, synchronised inputs, and it adds the risk that it has learned to depend on one sensor and fails silently when that sensor degrades.

The mathematics: data volume and what the choice costs

The data rate at a fusion point is the size of one message times the rate:

size

bytes per message: for an image, width × height × channels × bytes per value; for a point cloud, the number of points × bytes per point; for an object list, the number of objects × bytes per object.

f

the rate at which messages are produced, in hertz.

The sizes fall by orders of magnitude from raw data to objects, so late fusion is cheap to transport and early fusion is not.

Information and alignment. Early fusion at the level of a pixel or a point needs the alignment errors of Lesson P8-T3 to be small compared with the feature it fuses: a rotation error of 0.5° at 20 m displaces a point by 0.17 m, many pixels at that range, so the colour painted on a lidar point belongs to the neighbouring pixel. Object-level fusion tolerates larger errors, because it associates objects with gates (Lesson P7-T2) that absorb a few tenths of a metre, but it too fails once the error exceeds the gate.

Fusing two estimates. At the object level the standard combination of two position estimates with covariances P₁ and P₂ is

The formula is correct only if the two estimates are independent. Two tracks that were built using the same ego-motion estimate share an error, and the combination then reports a smaller covariance than it has earned (Lesson P8-T5).

Worked examples: data volume and a decision

Data volume. A camera gives 1920 × 1080 pixels × 3 bytes = 6.22 MB per frame. A lidar frame of 100,000 points at 16 bytes each (position, intensity, time) is 1.6 MB. At 10 Hz:

The raw level is nearly 2000 times the detection or object rate (78.2 MB/s against 40 kB/s). A network that carries 100 MB/s would be mostly full with raw fusion and almost empty with the others. This is why raw fusion is usually done on one computer next to the sensors, and why late or object-level fusion suits systems spread over several devices.

Failure behaviour. The camera is blinded by the sun for two seconds. Late fusion: the camera detector reports nothing, the lidar detector continues, and the fusion layer passes on the lidar objects with a flag that the camera is missing. Early fusion: the network receives a washed-out image with perfectly valid lidar points; whether it still works depends on whether it saw such conditions in training, and there is no flag. A learned early-fusion model that is not tested for a missing or corrupted sensor can silently lose accuracy.

Choosing. The table sets the conditions of three tasks against the architectures.

There is no best architecture; there is a best match to the data volume, the alignment you can achieve, the failure behaviour you need and the effort you can afford to train, test and debug.

Engineering implications and limitations

Start late and move earlier only for a measured gain. A late, object-level fusion is easy to debug. If its errors are caused by information lost before the merge, you can show it with data, and then move the merge earlier.

Define the interface in physical terms. Between sensor pipelines and the fusion pass objects in a common frame, with a timestamp, a covariance, a class with probabilities, and a status (updated this cycle or predicted). Do not pass opaque features across a boundary that a different team owns.

Associate before you average. Late fusion has a data association problem of its own: which camera detection goes with which lidar cluster. Use gating and an optimal assignment (Lesson P7-T2) and test the case when one sensor sees an object the other does not.

Do not double-count. If a lidar track and a camera track both used the same prior (a map, an ego-motion estimate), their errors are correlated and fusing them as independent makes the result over-confident. Fuse the measurements, not the already-fused estimates, or account for the correlation.

Test each sensor’s absence. Remove each sensor in turn on logged data and check that the system degrades in a defined way and says so. This is especially important for learned fusion.

Mind the latency. Late fusion waits for the slowest pipeline. If the camera pipeline takes 80 ms and the lidar 30 ms, the fused output is 80 ms old; the lidar objects must be predicted forward to the common time (Lesson P8-T3).

Keep the labs’ scope in mind. Lab P8-L1 is object-level: it lifts 2D detections with depth; it is not a learned early fusion, and its simulated sensors are not real ones.

Common misconceptions

Engineering takeaways

1. Fusion can occur on raw data (early), on per-sensor detections (late) or on tracks (object level); earlier means richer but more demanding.
2. Data volume falls by orders of magnitude: 78 MB/s raw against 40 kB/s for detections in the example.
3. Early fusion needs precise calibration and synchronisation and is harder to inspect; late fusion is modular and degrades gracefully.
4. Fuse estimates in information form P = (P₁⁻¹ + P₂⁻¹)⁻¹, but only if the errors are independent; do not double-count shared information.
5. Start with the simplest architecture that meets the requirement, test each sensor’s absence, and move earlier only for a measured gain.

References and further reading

Papers and handbook chapters on fusion architectures. The data sizes and rates are illustrative round numbers.

• Liu, Z. et al. (2023). BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird’s-Eye View Representation. ICRA.
• Qi, C. R., Liu, W., Wu, C., Su, H. and Guibas, L. J. (2018). Frustum PointNets for 3D Object Detection from RGB-D Data. CVPR: lifting 2D detections into 3D regions.
• Durrant-Whyte, H. and Henderson, T. C. (2016). Multisensor Data Fusion. In Springer Handbook of Robotics, 2nd ed.
• Bar-Shalom, Y., Li, X.-R. and Kirubarajan, T. (2001). Estimation with Applications to Tracking and Navigation. Wiley: track-to-track fusion and correlated errors.

Knowledge check

Three conceptual questions. Write an answer, then reveal the explanation. Your answers stay in your browser.

What’s next

Next: the arithmetic of combining estimates, uncertainty and conflicting observations.

Related content

Nothing below is required to finish this lesson.

• Before
  P8-T3: Coordinate Alignment and Timestamps

  Align measurements with extrinsics and handle time offsets.

• Lab
  P8-L1: Combine Camera Detections with Depth Information

  Lift 2D detections to 3D positions using depth and a camera model.