Multimodal Fusion and Vision-Language Models: A Survey for Robot Vision

Xiaofeng Han, Shunpeng Chen, Zenghuang Fu, Zhe Feng, Lue Fan, Dong An, Changwei Wang, Li Guo, Weiliang Meng, Xiaopeng Zhang, Rongtao Xu, Shibiao Xu

TL;DR

This comprehensive survey systematically reviews the integration of traditional multimodal fusion techniques and modern Vision-Language Models (VLMs) within robotic vision. By mapping out the architectural evolution across core tasks like SLAM, 3D object detection, navigation, and manipulation, the paper highlights how robotic perception is shifting from passive scene parsing to proactive, language-driven intelligence.

What Problem Does This Paper Address?

Robots operating in the real world face severe perceptual challenges: occlusions, drastic lighting changes, textureless surfaces, and ambiguous environments. Relying on a single sensor modality (like standard RGB cameras) is rarely enough for safe, autonomous operation.

While traditional multimodal fusion combines heterogeneous sensor data (such as pairing cameras with LiDAR or Radars) to improve geometric accuracy, it lacks high-level semantic reasoning. Conversely, modern Vision-Language Models (VLMs) possess incredible reasoning and instruction-following capabilities but are typically trained on static 2D web data. They struggle with real-time constraints, 3D spatial awareness, and direct physical control. This paper addresses this critical gap by providing a task-oriented framework that unifies traditional sensor fusion with modern VLM-driven embodied AI, outlining what it takes to deploy these systems onto physical robotic platforms.

Figure 1: The overview figure illustrates the overall framework of multimodal fusion and VLMs for robot vision. Various sensory inputs (e.g., RGB, Depth, LiDAR, GPS, IMU) are first processed through multimodal fusion strategies, including encoder-decoder frameworks, attention mechanisms, and graph neural networks, to enhance perception. The resulting fused features support core robotic vision tasks such as 3D semantic scene understanding, SLAM, 3D object detection, navigation and localization, and robot manipulation. Vision-language models further bridge perception and reasoning by aligning visual and linguistic information, enabling semantic understanding and action generation. The diagram highlights the integration of traditional fusion pipelines with large vision-language models for complex task execution in robotic systems.

Methods

The authors categorize the technical landscape into two primary paradigms:

1. Traditional Multimodal Fusion

For foundational tasks like semantic scene understanding, fusion strategies are split by their processing stage:

  • Early Fusion: Fusing raw data inputs before feature extraction.
  • Mid (Intermediate) Fusion: Aligning features in an intermediate layer using attention mechanisms, feature weighting, or Graph Neural Networks (GNNs).
  • Late Fusion: Combining independent decision outputs from each modality.
Figure 3: The above diagram intuitively illustrates the basic processes of three main strategies for multimodal fusion: early fusion, mid fusion, and late fusion. In the figure, each modality data (such as images, audio, Lidar, text, etc.) is in- dependently processed by a feature extractor. Early fusion directly fuses data from different modalities before feature extraction; Mid term fusion combines modal features through specific mechanisms such as feature concatenation or weighting after extracting them; Late stage fusion is achieved by integrating the decision results of each modality after independent decision-making is com- pleted. The diagram clearly reflects the key differences and roles of the three fusion methods in the multimodal processing flow.

For 3D Object Detection, the granularity of fusion is further divided into point-level, voxel-level, and ROI-level approaches, heavily relying on Transformer decoders and cross-attention to dynamically map camera features to 3D spaces.

Figure 6: The multimodal fusion framework for 3D object detection includes voxel-level, point-level, and ROI-level fusion to integrate LiDAR and image features and improve detection accuracy.

2. Vision-Language-Action (VLA) Models

The survey tracks the evolution toward unified architectures (like RT-2 and OpenVLA) that ingest vision and language instructions to output discrete action tokens or 3D affordance maps. The authors analyze how techniques like Mixture-of-Experts (MoE) and Low-Rank Adaptation (LoRA) are being utilized to make these massive networks computationally viable for robotics.

Datasets

The paper reviews mainstream benchmarks across three core domains:

  • Scene Understanding: nuScenes and Waymo Open Dataset (autonomous driving); SemanticKITTI (LiDAR sequences); Matterport3D, ScanNet, and NYU Depth V2 (indoor environments).
  • Robot Manipulation: DROID, RT-1, and Open X-Embodiment (cross-robot generalization); Touch and Go and VisGel (vision-tactile interaction).
  • Embodied Navigation: Room-to-Room (R2R), REVERIE, CVDN, and SOON.

Key Insights and Ideas

  • The 3D Spatial Blindspot: Pre-trained VLMs excel at text-image alignment but fundamentally lack an explicit understanding of 3D spatial topology and temporal consistency. They cannot natively perform the causal reasoning across space-time required for complex navigation.
  • Granularity Mismatch: Aligning continuous, high-dimensional sensor data (like tactile grids or LiDAR point clouds) with discrete, abstract language tokens remains a core algorithmic bottleneck.
  • Structured Memory Wins over End-to-End: Pure end-to-end VLM pipelines often underperform in navigation tasks compared to hybrid models that combine language reasoning with explicit topological maps or structured spatial memory.

Key Results

  • 3D Object Detection: On the nuScenes benchmark, Transformer-driven fusion methods dominate. Models like MV2DFusion achieve state-of-the-art results by leveraging modality-specific query mechanisms to accurately detect vehicles, pedestrians, and bicycles.
  • Embodied Navigation: On the Room-to-Room (R2R) benchmark, panoramic semantic mapping methods like PanoGen++ and VISTA achieve a 75% Success Rate (SR) in unseen environments, outperforming traditional sequence-based and pure VLM baselines by explicitly modeling environmental topology.
  • 3D Visual Question Answering: BridgeQA leads the ScanQA benchmark by establishing hierarchical cross-modal bridging channels that successfully connect global 3D scene contexts with fine-grained local object features.

Limitations and Future Work

Despite the promise of VLMs, the authors identify severe bottlenecks blocking real-world robotic deployment:

  1. Inference Latency: Large VLMs are too computationally heavy and slow for edge devices requiring high-frequency control loops.
  2. Data Scarcity and Bias: Most multimodal datasets are synthetic or heavily biased toward Western indoor layouts and clear-weather driving scenarios, causing models to fail in dynamic, low-light, or adverse weather conditions.

Future Directions

The survey points to four critical frontiers:

  • Spatial Intelligence: Integrating explicit 3D topological map embeddings and SLAM components directly into VLM architectures.
  • Lightweight Deployment: Advancing model compression (quantization, pruning) and dynamic modality selection to minimize edge compute overhead.
  • Ethical & Safe AI: Developing standardized adversarial evaluation benchmarks and human-in-the-loop trust modeling to ensure physical safety.
  • Brain-Inspired Design: Borrowing concepts from neuroscience—such as how the human hippocampus consolidates spatiotemporal memory—to build more cognitively plausible robotic architectures.

Summary updated

Abstract

Robot vision has greatly benefited from advancements in multimodal fusion techniques and vision-language models (VLMs). We adopt a task-oriented perspective to systematically review the applications and advancements of multimodal fusion methods and VLMs in the field of robot vision. For semantic scene understanding tasks, we categorize fusion approaches into encoder-decoder frameworks, attention-based architectures, and graph neural networks. Meanwhile, we also analyze the architectural characteristics and practical implementations of these fusion strategies in key tasks such as simultaneous localization and mapping (SLAM), 3D object detection, navigation, and manipulation. We compare the evolutionary paths and applicability of VLMs based on large language models (LLMs) with traditional multimodal fusion methods.Additionally, we conduct an in-depth analysis of commonly used datasets, evaluating their applicability and challenges in real-world robotic scenarios. Building on this analysis, we identify key challenges in current research, including cross-modal alignment, efficient fusion, real-time deployment, and domain adaptation. We propose future directions such as self-supervised learning for robust multimodal representations, structured spatial memory and environment modeling to enhance spatial intelligence, and the integration of adversarial robustness and human feedback mechanisms to enable ethically aligned system deployment. Through a comprehensive review, comparative analysis, and forward-looking discussion, we provide a valuable reference for advancing multimodal perception and interaction in robotic vision. A comprehensive list of studies in this survey is available at https://github.com/Xiaofeng-Han-Res/MF-RV.
BibTeX
@article{han2025multimodal,
  author = {Xiaofeng Han and Shunpeng Chen and Zenghuang Fu and Zhe Feng and Lue Fan and Dong An and Changwei Wang and Li Guo and Weiliang Meng and Xiaopeng Zhang and Rongtao Xu and Shibiao Xu},
  title = {{Multimodal Fusion and Vision-Language Models: A Survey for Robot Vision}},
  journal = {Information Fusion},
  year = {2025},
  url = {https://openreview.net/forum?id=DJXpktHZBZ},
}
Figure