Multi-Modal Perception with Mismatched Sensor Fields of View

This direction is split into other directions: asymmetric sensor coverage. Its last version stays readable and citable here.

Version 1 of 184 papers103 conceptsfrozen 2026-10-04

In the design of autonomous systems, we often assume that sensors are complementary and co-located. We imagine a world where a camera’s rich texture perfectly overlays a LiDAR’s precise geometry. However, real-world sensor suites are fundamentally asymmetric. A typical LiDAR scanner provides a 360-degree panoramic view of the environment, while a high-resolution camera might only cover a 60-degree forward-facing frustum.

This "field-of-view (FoV) mismatch" creates a spatial inconsistency that breaks traditional fusion techniques. If a model is trained to "decorate" LiDAR points with camera features, what happens to the 80% of the point cloud that the camera never saw?

The Heterogeneity Gap and Spatial Overlap

The core challenge, as identified by Zhipeng Zhao and colleagues in Attention-Enhanced Cross-modal Localization Between 360 Images and Point Clouds, is the "heterogeneity gap." When sensors have different coverage, the number of overlapping features available for comparison is severely limited.

In standard multi-modal fusion, performance is often constrained by the "narrowest" sensor. Georg Krispel and the team behind FuseSeg note that extending architectures to provide full surround-view coverage requires addressing the data imbalance between localized high-resolution imagery and global 3D point clouds. Without explicit handling, the system develops "blind spots" where the geometric sensor (LiDAR) lacks the semantic guidance of the visual sensor (Camera).

Learning from What You Can't See

To solve this, researchers have moved away from simple feature concatenation toward cross-modal distillation. The intuition is that if a model can learn the relationship between 2D textures and 3D structures in the overlapping region, it can "hallucinate" or generalize that understanding to the rest of the 360-degree scan.

Lukas Hoyer and his team proposed IGNet (2D Feature Distillation for Weakly- and Semi-Supervised 3D Semantic Segmentation), which uses a "one-way" contrastive loss. Instead of requiring every 3D point to have a 2D match, the model distills high-level features from a 2D image-trained network into a 3D network. This allows the 3D network to inherit semantic wisdom from the camera even for points that fall outside the camera’s FoV.

Similarly, Jun CEN introduced CMDFusion (CMDFusion: Bidirectional Fusion Network with Cross-modality Knowledge Distillation for LIDAR Semantic Segmentation), which uses distillation during training so that at inference time, the model can achieve high performance using LiDAR alone. This effectively "bakes" the visual context into the geometric weights, bypassing the FoV restriction entirely during deployment.

Geometric Generalization and Pseudo-Labeling

When direct visual supervision is unavailable, the system must rely on the geometric consistency of the scene. Nermin Samet in LOSC explains that points outside the primary sensor's field of view must rely on the geometric generalization of the 3D network.

One powerful mechanism for this is the creation of "semantic parametric maps." In XD-MAP, Frank Bieder and colleagues represent objects like poles or signs as geometric primitives. By building this intermediate map, the system can generate high-quality pseudo-labels for LiDAR data in areas the camera never saw, using the "memory" of the environment's structure.

Classifying the Unknown

A more recent approach to handling FoV asymmetry involves explicitly modeling what is "missing." In Neural Surface Reconstruction and Rendering for LiDAR-Visual Systems, Jianheng Liu and Chunran Zheng propose a spatial classification strategy. They categorize regions into:

  1. Free/Occupied: Known geometry.
  2. Visible Unknown: Captured by images but lacking depth data.
  3. Invisible Unknown: Not captured by either.

By identifying "visible unknown" regions, the system can target these areas for geometric completion using photometric data (like NeRF-based rendering), effectively using the camera to "fill in" the gaps left by the LiDAR's sparsity or range limits.

Beyond Vision: The Omnidirectional Problem

This problem isn't limited to cameras and LiDAR. It appears in any multi-modal system with geometric asymmetry. Yating Xu and Gim Hee Lee explored this in audio-visual parsing (Rethink Cross-Modal Fusion in Weakly-Supervised Audio-Visual Video Parsing). Microphones are omnidirectional, while cameras are directional. This leads to "audible but not visible" events—like a dog barking behind the camera—which can confuse a model that expects perfect spatial alignment. Their solution involves selective fusion that recognizes when one modality should be ignored because it lacks a spatial counterpart in the other.

Summary of the Intuition

The shift in this research area is from alignment-dependent fusion (where sensors must see the same thing at the same time) to representation-level distillation. By treating the overlapping FoV as a "classroom" where the 3D network learns to associate geometry with semantics, researchers enable models to maintain high performance even in the "blind spots" that define real-world sensor configurations.

Papers in this version

Showing 60 of 84 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_multi_modal_perception_with_mismatched_s_ddd5c00f,
  title = {Multi-Modal Perception with Mismatched Sensor Fields of View},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multi-modal-perception-with-mismatched-sensor-fields-of-view/rdr_ddd5c00f49524369b66433c43b317534},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_multi_modal_perception_with_mismatched_s_ddd5c00f_v1,
  title = {Multi-Modal Perception with Mismatched Sensor Fields of View},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/multi-modal-perception-with-mismatched-sensor-fields-of-view/rdr_ddd5c00f49524369b66433c43b317534/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}