Abstract
Navigating complex urban environments remains a significant challenge for autonomous driving systems, particularly when dealing with occlusions and unpredictable agent behaviors. This paper introduces ReasonNet, an end-to-end driving framework that integrates temporal memory and global scene reasoning. By maintaining a selective memory bank of past features and modeling object interactions through graph attention networks, ReasonNet effectively anticipates hidden hazards. Evaluated on the CARLA Leaderboard, ReasonNet achieved state-of-the-art performance, demonstrating superior safety and robustness in high-risk occlusion scenarios.
Introduction
While autonomous vehicles have made significant strides on open highways, the "long-tail" of rare and dangerous urban events—such as a pedestrian emerging from behind a parked vehicle—remains a formidable hurdle. Traditional modular pipelines often suffer from error propagation, where a failure in detection leads to catastrophic planning errors downstream. Conversely, many existing end-to-end models lack the temporal depth and global context required to navigate "unobservable spaces."
ReasonNet addresses these limitations by mimicking human-like reasoning. It does not merely react to visible stimuli; it leverages a sophisticated memory architecture and interaction modeling to infer the presence of occluded threats, such as a vehicle running a red light behind a visual obstruction or a sudden brake event caused by an unseen pedestrian.
Methodology
ReasonNet’s architecture is built upon three foundational pillars designed to process multi-modal sensor data into safe, actionable driving commands.
1. Unified Perception
The system fuses inputs from four RGB cameras and a LiDAR sensor into a unified Bird’s-Eye-View (BEV) feature map. This top-down perspective provides a consistent spatial representation of the environment, facilitating better spatial reasoning than standard perspective-view processing.
2. Temporal Reasoning and the Memory Bank
A core innovation of ReasonNet is its dual-stage memory bank, which allows the model to maintain a persistent understanding of the scene over time:
- Short-term Buffer: Stores high-resolution features from recent frames to capture immediate motion dynamics.
- Long-term Buffer: Employs a selective mechanism to store "important" features from the distant past. Importance is determined by the attention weights assigned by the model and the probability of object existence in specific spatial locations. This selective approach allows the model to track objects that are intermittently occluded without excessive computational overhead.
3. Global Reasoning and Occupancy Prediction
To understand the relationships between various agents, ReasonNet utilizes a Graph Attention Network (GAT). This module models the interactions between all detected objects and the static environment. Furthermore, an Occupancy Decoder predicts the future spatial distribution of agents, allowing the model to anticipate where obstacles will be in the coming seconds.
Training with Consistency Loss
ReasonNet is trained via imitation learning, but it introduces a novel "Consistency Loss" to enhance safety. This loss function penalizes the model if its planned trajectory (waypoints) intersects with its own predicted future occupancy map. By enforcing internal consistency between the "planner" and the "predictor," the model learns defensive driving behaviors, such as slowing down when the predicted occupancy suggests a potential collision in an unobservable area.
Experimental Evaluation
The researchers evaluated ReasonNet using the industry-standard CARLA simulator and a new specialized benchmark designed for occlusion handling.
CARLA Leaderboard Performance
ReasonNet achieved 1st place on the CARLA Leaderboard sensor track with a Driving Score of 79.95. This result outperformed previous state-of-the-art models, including InterFuser (76.18), LAV, and TransFuser. The results highlight the framework's ability to handle diverse urban driving conditions with high reliability.
DriveOcclusionSim (DOS)
To specifically test the model's ability to handle hidden threats, the authors introduced DriveOcclusionSim (DOS). This benchmark focuses on four critical scenarios: parked cars, sudden brakes, blocked left turns, and red light infractions. In these high-risk scenarios, ReasonNet’s success rate was nearly double that of models lacking temporal reasoning components. Ablation studies confirmed that removing the temporal module led to a 46% drop in success rates within occlusion-heavy environments.
Conclusion
ReasonNet demonstrates that temporal memory and global interaction modeling are essential for safe end-to-end autonomous driving. By effectively reasoning about what it cannot see based on the behavior of visible agents, the framework sets a new standard for performance on the CARLA benchmark. Future work will focus on bridging the gap between simulation and real-world deployment and integrating explicit traffic rule constraints to further refine the model's decision-making in complex edge cases.
Summary updated