Abstract
Traditional autonomous driving (AD) systems are constrained by the line-of-sight of onboard sensors, making them vulnerable to occluded hazards. While Vehicle-to-Everything (V2X) communication provides a pathway for infrastructure-assisted safety, integrating high-dimensional multimodal data in real-time remains a challenge. This paper introduces REACT (Real-time Edge-based Autonomous Co-pilot Trajectory planner), a framework that leverages a lightweight 500M-parameter Vision-Language Model (VLM) to fuse onboard camera data with Roadside Unit (RSU) hazard alerts. By shifting the VLM's task from full path generation to predicting trajectory "residuals," REACT achieves a 77% reduction in collision rates while maintaining sub-second latency on edge hardware like the NVIDIA Jetson AGX Orin.
Introduction: The Challenge of Occluded Hazards
Autonomous vehicles rely heavily on cameras and LiDAR, which are inherently limited by physical obstructions. A vehicle approaching an intersection may be blind to a red-light runner hidden behind a building. Vehicle-to-Everything (V2X) technology, specifically alerts from Roadside Units (RSUs), can provide "bird's-eye" awareness. However, current methods for processing this data often lack the contextual reasoning required for complex traffic scenarios or are too computationally intensive for real-time edge deployment.
The REACT Framework
The REACT framework is designed to bridge the gap between high-level reasoning and real-time execution. It operates as a "co-pilot" that monitors the primary autonomous driving algorithm and intervenes when a downstream accident is detected via V2X alerts.
The system consists of five integrated modules:
- Data Acquisition & Structuring: Multi-view camera frames are converted into Bird’s-Eye View (BEV) feature maps. Simultaneously, hazard alerts from RSUs are validated and structured.
- Task Alignment: This module synchronizes spatial coordinates and timestamps into a unified ego-centric frame, ensuring the VLM understands the hazard's precise location relative to the vehicle.
- Task Projection Enhancement: REACT utilizes SmolVLM2-500M, a lightweight VLM fine-tuned with Chain-of-Thought (CoT) reasoning. This allows the model to "think" through spatial relationships before making a decision.
- Residual Trajectory Fusion (RTF): Instead of generating a new path from scratch, the model predicts adjustments ($\Delta W$) to the existing planned trajectory.
- Edge Adaptation: Optimization techniques like 16-bit quantization and token reduction enable the model to run efficiently on edge hardware.
Technical Innovations: Residuals and Reasoning
A core innovation of REACT is Residual Trajectory Fusion (RTF). Traditional LLM-based planners attempt to generate full coordinate sequences, which is computationally expensive and prone to continuity errors. By predicting only the "residuals" (small deviations from the nominal path), REACT maintains motion smoothness while drastically reducing the number of output tokens required.
Furthermore, the use of Chain-of-Thought (CoT) fine-tuning enables the VLM to perform high-level contextual reasoning. The model doesn't just output coordinates; it identifies the hazard, assesses the risk, and concludes how the trajectory should change. This symbolic reasoning allows it to handle complex V2X data more effectively than standard end-to-end neural networks.
Experimental Results
The framework was evaluated using the DeepAccident benchmark, a large-scale CARLA-based dataset featuring nearly 700 accident scenarios across diverse environments.
Key performance metrics include:
- Safety: A 77% Collision Rate Reduction (CRR) over baseline autonomous models.
- Perception: Superior Video Panoptic Quality (48.2% VPQ) and mean Intersection over Union (59.1% mIOU).
- Efficiency: An inference latency of 0.57 seconds on the NVIDIA Jetson AGX Orin, representing a 93.67% reduction in reaction latency compared to unoptimized VLM baselines.
- Robustness: Consistent performance across adverse weather conditions, including rain and fog.
Conclusion and Future Directions
REACT demonstrates that lightweight VLMs can provide the necessary reasoning for V2X-assisted safety without sacrificing real-time performance. While the system shows remarkable gains in collision avoidance, future work will address precision in yaw angle prediction and the integration of more complex road topologies. By focusing on trajectory residuals, REACT provides a scalable path for deploying advanced AI reasoning in the next generation of autonomous vehicles.
Summary updated