Mutual Information-calibrated Conformal Feature Fusion for Uncertainty-Aware Multimodal 3D Object Detection at the Edge

Alex C. Stutts, Danilo Erricolo, Sathya Ravi, Theja Tulabandhula, Amit Ranjan Trivedi

TL;DR

Researchers have developed a lightweight framework for 3D object detection that doesn't just predict where an object is, but also provides statistically sound "uncertainty boxes" around its predictions. By fusing LiDAR and camera data through a Variational Autoencoder (VAE) and calibrating results using Normalized Mutual Information (NMI), the system achieves state-of-the-art inference speeds on edge hardware while significantly improving detection reliability.

What Problem Does This Paper Address?

In the world of autonomous robotics, knowing what is in front of you is only half the battle; you also need to know how certain you are about that detection. Most current 3D object detection models focus purely on accuracy (mean predictions) and ignore uncertainty.

When uncertainty is addressed, researchers often rely on "Bayesian" methods or Monte Carlo sampling, which require running the model multiple times for a single frame. This is computationally expensive and impractical for "edge" devices—the low-power computers found on actual robots or drones that need to make split-second decisions. This paper tackles the challenge of providing robust, real-time uncertainty quantification without the heavy computational overhead.

Methods

The authors introduce a novel architecture that combines three sophisticated concepts:

  1. Multimodal Feature Fusion via VAE: Instead of simply "stacking" camera images and LiDAR point clouds, the model uses a Variational Autoencoder (VAE) approach. It treats features from each sensor as multivariate Gaussian distributions. These are fused using a multivariate Gaussian product, which creates a joint latent space that is more resilient even if one sensor's data is suboptimal (e.g., a blurry camera image).
  2. Conformal Inference (CI): To generate uncertainty bounds, the authors use CI. Unlike standard statistics that assume data follows a specific "bell curve," CI provides mathematically guaranteed prediction intervals based on how much a new data point "conforms" to the training set. This allows the model to output "Upper" and "Lower" bounding boxes that represent the range of uncertainty.
  3. NMI-Based Calibration: The "aha!" moment of the paper is using Normalized Mutual Information (NMI). NMI measures how much information one sensor modality (like LiDAR) shares with another (like RGB). The authors use NMI as a dynamic "tuner" in their loss function. If the sensors share high mutual information, the model is encouraged to tighten its uncertainty bounds; if they don't, the bounds stay cautious.

The backbone of the system uses efficient, well-known models: PointNet for LiDAR, and YOLOv5s combined with MobileNetV2 for camera processing, ensuring the pipeline remains fast.

Datasets

The framework was evaluated using the KITTI 3D object detection benchmark, a gold-standard dataset in the autonomous driving community. It contains real-world driving sequences with synchronized LiDAR and camera data, categorized into "Easy," "Moderate," and "Hard" scenarios based on object occlusion and distance.

Key Insights and Ideas

  • The Inverse Correlation: The study found a clear inverse correlation between NMI and predictive uncertainty. As the model trains and gets better at extracting shared information between sensors, the inherent uncertainty drops.
  • Lightweight is Possible: The authors proved that you don't need iterative sampling (like Monte Carlo) to get reliable uncertainty. By integrating the uncertainty logic directly into the loss function (a "multitask loss"), the model learns to output uncertainty in a single forward pass.
  • Visualizing "How Wrong": Instead of a single number representing error, the model produces physical "uncertainty boxes" (visualized in purple in the paper). This is far more intuitive for downstream tasks like path planning—a robot can literally see the "buffer zone" it should avoid.

Key Results

  • Speed: The model achieved an inference runtime of approximately 9.87ms to 14.82ms on modern hardware. This is roughly 38% faster than PointPillars, one of the most popular LiDAR-only models, despite the added complexity of camera fusion and uncertainty logic.
  • Accuracy Boost: When the model used NMI-calibrated uncertainty to refine its predictions, the Average Precision (AP) increased by at least 39%.
  • Consistency: Unlike many models that see a sharp drop-off in performance on "Hard" cases, this framework maintained relatively consistent accuracy across all benchmark categories, likely due to the robust nature of the VAE-based fusion.

Limitations and Future Work

The authors note that while the model is highly efficient, the "Hard" cases in the KITTI dataset have fewer annotations, which can make it difficult for the model to learn precise uncertainty in the most challenging environments.

Future work could involve testing this framework on even more diverse datasets (like NuScenes or Waymo) and deploying it on actual robotic hardware to see how the "uncertainty boxes" improve real-world navigation safety. There is also potential to expand the framework to include more sensors, such as mmWave radar, using the same generalizable Gaussian product fusion method.

Summary updated

BibTeX
@inproceedings{stutts2024mutual,
  author = {Alex C. Stutts and Danilo Erricolo and Sathya Ravi and Theja Tulabandhula and Amit Ranjan Trivedi},
  title = {{Mutual Information-calibrated Conformal Feature Fusion for Uncertainty-Aware Multimodal 3D Object Detection at the Edge}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2024},
  doi = {10.1109/icra57147.2024.10609987},
}

Loading related content...

Figure