Edge-Based Multimodal Sensor Data Fusion with Vision Language Models (VLMs) for Real-time Autonomous Vehicle Accident Avoidance

Fengze Yang, Bo Yu, Yang Zhou, Xuewen Luo, Zhengzhong Tu, Chenxi LiuarXiv 2025 · DOI · arXiv

Abstract

Traditional autonomous driving (AD) systems are constrained by the line-of-sight of onboard sensors, making them vulnerable to occluded hazards. While Vehicle-to-Everything (V2X) communication provides a pathway for infrastructure-assisted safety, integrating high-dimensional multimodal data in real-time remains a challenge. This paper introduces REACT (Real-time Edge-based Autonomous Co-pilot Trajectory planner), a framework that leverages a lightweight 500M-parameter Vision-Language Model (VLM) to fuse onboard camera data with Roadside Unit (RSU) hazard alerts. By shifting the VLM's task from full path generation to predicting trajectory "residuals," REACT achieves a 77% reduction in collision rates while maintaining sub-second latency on edge hardware like the NVIDIA Jetson AGX Orin.

Introduction: The Challenge of Occluded Hazards

Autonomous vehicles rely heavily on cameras and LiDAR, which are inherently limited by physical obstructions. A vehicle approaching an intersection may be blind to a red-light runner hidden behind a building. Vehicle-to-Everything (V2X) technology, specifically alerts from Roadside Units (RSUs), can provide "bird's-eye" awareness. However, current methods for processing this data often lack the contextual reasoning required for complex traffic scenarios or are too computationally intensive for real-time edge deployment.

The REACT Framework

The REACT framework is designed to bridge the gap between high-level reasoning and real-time execution. It operates as a "co-pilot" that monitors the primary autonomous driving algorithm and intervenes when a downstream accident is detected via V2X alerts.

Figure 1: High-level system architecture of the REACT framework, showing the integration of RSU hazard alerts and onboard sensors with the vehicle control module.

The system consists of five integrated modules:

  1. Data Acquisition & Structuring: Multi-view camera frames are converted into Bird’s-Eye View (BEV) feature maps. Simultaneously, hazard alerts from RSUs are validated and structured.
  2. Task Alignment: This module synchronizes spatial coordinates and timestamps into a unified ego-centric frame, ensuring the VLM understands the hazard's precise location relative to the vehicle.
  3. Task Projection Enhancement: REACT utilizes SmolVLM2-500M, a lightweight VLM fine-tuned with Chain-of-Thought (CoT) reasoning. This allows the model to "think" through spatial relationships before making a decision.
  4. Residual Trajectory Fusion (RTF): Instead of generating a new path from scratch, the model predicts adjustments ($\Delta W$) to the existing planned trajectory.
  5. Edge Adaptation: Optimization techniques like 16-bit quantization and token reduction enable the model to run efficiently on edge hardware.

Technical Innovations: Residuals and Reasoning

A core innovation of REACT is Residual Trajectory Fusion (RTF). Traditional LLM-based planners attempt to generate full coordinate sequences, which is computationally expensive and prone to continuity errors. By predicting only the "residuals" (small deviations from the nominal path), REACT maintains motion smoothness while drastically reducing the number of output tokens required.

Figure 2: Detailed pipeline of the REACT framework, illustrating the flow from BEV extraction and task alignment to VLM-based Chain-of-Thought reasoning and residual trajectory generation.

Furthermore, the use of Chain-of-Thought (CoT) fine-tuning enables the VLM to perform high-level contextual reasoning. The model doesn't just output coordinates; it identifies the hazard, assesses the risk, and concludes how the trajectory should change. This symbolic reasoning allows it to handle complex V2X data more effectively than standard end-to-end neural networks.

Experimental Results

The framework was evaluated using the DeepAccident benchmark, a large-scale CARLA-based dataset featuring nearly 700 accident scenarios across diverse environments.

Key performance metrics include:

  • Safety: A 77% Collision Rate Reduction (CRR) over baseline autonomous models.
  • Perception: Superior Video Panoptic Quality (48.2% VPQ) and mean Intersection over Union (59.1% mIOU).
  • Efficiency: An inference latency of 0.57 seconds on the NVIDIA Jetson AGX Orin, representing a 93.67% reduction in reaction latency compared to unoptimized VLM baselines.
  • Robustness: Consistent performance across adverse weather conditions, including rain and fog.

Conclusion and Future Directions

REACT demonstrates that lightweight VLMs can provide the necessary reasoning for V2X-assisted safety without sacrificing real-time performance. While the system shows remarkable gains in collision avoidance, future work will address precision in yaw angle prediction and the integration of more complex road topologies. By focusing on trajectory residuals, REACT provides a scalable path for deploying advanced AI reasoning in the next generation of autonomous vehicles.

Summary updated

Abstract

Autonomous driving (AD) systems relying solely on onboard sensors may fail to detect distant or obstacle hazards, potentially causing preventable collisions; however, existing transformer-based Vehicle-to-Everything (V2X) approaches, which mitigate AD sensing limitations, either lack effective multimodal fusion and reasoning or struggle to meet real-time performance requirements under complex, high-dimensional traffic conditions. This paper proposes the Real-time Edge-based Autonomous Co-pilot Trajectory planner (REACT), a V2X-integrated trajectory optimization framework for AD based on a fine-tuned lightweight Vision-Language Model (VLM). REACT integrates infrastructure-provided hazard alerts with onboard sensor data, capturing intricate surrounding traffic dynamics and vehicle intents through visual embeddings, interpreting precise numerical data from symbolic inputs, and employing contextual reasoning to generate optimized, safety-oriented trajectories. To ensure robust real-time deployment on edge devices, REACT innovatively employs Residual Trajectory Fusion (RTF) design and specialized edge-adaptation strategies to reduce model complexity and improve inference efficiency. Evaluated on the DeepAccident benchmark, REACT achieves state-of-the-art performance, a 77% collision rate reduction, a 48.2% Video Panoptic Quality (VPQ), and a 0.57-second inference latency on the Jetson AGX Orin. Ablation studies validate the contribution of each input, module, and edge adaptation strategy. These results highlight the effectiveness of lightweight VLMs in enabling real-time cooperative planning on edge platforms and underscore the potential of language-guided contextual reasoning for improving traffic safety and responsiveness.
BibTeX
@misc{yang2025edge,
  author = {Fengze Yang and Bo Yu and Yang Zhou and Xuewen Luo and Zhengzhong Tu and Chenxi Liu},
  title = {{Edge-Based Multimodal Sensor Data Fusion with Vision Language Models (VLMs) for Real-time Autonomous Vehicle Accident Avoidance}},
  year = {2025},
  eprint = {2508.01057},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.01057},
}
Figure