ReasonNet: End-to-End Driving with Temporal and Global Reasoning

Hao Shao, Letian Wang, Ruobing Chen, Steven L. Waslander, Hongsheng Li, Yu Liu

Abstract

Navigating complex urban environments remains a significant challenge for autonomous driving systems, particularly when dealing with occlusions and unpredictable agent behaviors. This paper introduces ReasonNet, an end-to-end driving framework that integrates temporal memory and global scene reasoning. By maintaining a selective memory bank of past features and modeling object interactions through graph attention networks, ReasonNet effectively anticipates hidden hazards. Evaluated on the CARLA Leaderboard, ReasonNet achieved state-of-the-art performance, demonstrating superior safety and robustness in high-risk occlusion scenarios.

Introduction

While autonomous vehicles have made significant strides on open highways, the "long-tail" of rare and dangerous urban events—such as a pedestrian emerging from behind a parked vehicle—remains a formidable hurdle. Traditional modular pipelines often suffer from error propagation, where a failure in detection leads to catastrophic planning errors downstream. Conversely, many existing end-to-end models lack the temporal depth and global context required to navigate "unobservable spaces."

ReasonNet addresses these limitations by mimicking human-like reasoning. It does not merely react to visible stimuli; it leverages a sophisticated memory architecture and interaction modeling to infer the presence of occluded threats, such as a vehicle running a red light behind a visual obstruction or a sudden brake event caused by an unseen pedestrian.

Methodology

ReasonNet’s architecture is built upon three foundational pillars designed to process multi-modal sensor data into safe, actionable driving commands.

1. Unified Perception

The system fuses inputs from four RGB cameras and a LiDAR sensor into a unified Bird’s-Eye-View (BEV) feature map. This top-down perspective provides a consistent spatial representation of the environment, facilitating better spatial reasoning than standard perspective-view processing.

2. Temporal Reasoning and the Memory Bank

A core innovation of ReasonNet is its dual-stage memory bank, which allows the model to maintain a persistent understanding of the scene over time:

  • Short-term Buffer: Stores high-resolution features from recent frames to capture immediate motion dynamics.
  • Long-term Buffer: Employs a selective mechanism to store "important" features from the distant past. Importance is determined by the attention weights assigned by the model and the probability of object existence in specific spatial locations. This selective approach allows the model to track objects that are intermittently occluded without excessive computational overhead.

3. Global Reasoning and Occupancy Prediction

To understand the relationships between various agents, ReasonNet utilizes a Graph Attention Network (GAT). This module models the interactions between all detected objects and the static environment. Furthermore, an Occupancy Decoder predicts the future spatial distribution of agents, allowing the model to anticipate where obstacles will be in the coming seconds.

Training with Consistency Loss

ReasonNet is trained via imitation learning, but it introduces a novel "Consistency Loss" to enhance safety. This loss function penalizes the model if its planned trajectory (waypoints) intersects with its own predicted future occupancy map. By enforcing internal consistency between the "planner" and the "predictor," the model learns defensive driving behaviors, such as slowing down when the predicted occupancy suggests a potential collision in an unobservable area.

Experimental Evaluation

The researchers evaluated ReasonNet using the industry-standard CARLA simulator and a new specialized benchmark designed for occlusion handling.

CARLA Leaderboard Performance

ReasonNet achieved 1st place on the CARLA Leaderboard sensor track with a Driving Score of 79.95. This result outperformed previous state-of-the-art models, including InterFuser (76.18), LAV, and TransFuser. The results highlight the framework's ability to handle diverse urban driving conditions with high reliability.

DriveOcclusionSim (DOS)

To specifically test the model's ability to handle hidden threats, the authors introduced DriveOcclusionSim (DOS). This benchmark focuses on four critical scenarios: parked cars, sudden brakes, blocked left turns, and red light infractions. In these high-risk scenarios, ReasonNet’s success rate was nearly double that of models lacking temporal reasoning components. Ablation studies confirmed that removing the temporal module led to a 46% drop in success rates within occlusion-heavy environments.

Conclusion

ReasonNet demonstrates that temporal memory and global interaction modeling are essential for safe end-to-end autonomous driving. By effectively reasoning about what it cannot see based on the behavior of visible agents, the framework sets a new standard for performance on the CARLA benchmark. Future work will focus on bridging the gap between simulation and real-world deployment and integrating explicit traffic rule constraints to further refine the model's decision-making in complex edge cases.

Summary updated

Abstract

The large-scale deployment of autonomous vehicles is yet to come, and one of the major remaining challenges lies in urban dense traffic scenarios. In such cases, it remains challenging to predict the future evolution of the scene and future behaviors of objects, and to deal with rare adverse events such as the sudden appearance of occluded objects. In this paper, we present ReasonNet, a novel end-to-end driving framework that extensively exploits both temporal and global information of the driving scene. By reasoning on the temporal behavior of objects, our method can effectively process the interactions and relationships among features in different frames. Reasoning about the global information of the scene can also improve overall perception performance and benefit the detection of adverse events, especially the anticipation of potential danger from occluded objects. For comprehensive evaluation on occlusion events, we also release publicly a driving simulation benchmark DriveOcclusionSim consisting of diverse occlusion events. We conduct extensive experiments on multiple CARLA benchmarks, where our model outperforms all prior methods, ranking first on the sensor track of the public CARLA Leaderboard [53].
BibTeX
@inproceedings{shao2023reasonnet,
  author = {Hao Shao and Letian Wang and Ruobing Chen and Steven L. Waslander and Hongsheng Li and Yu Liu},
  title = {{ReasonNet: End-to-End Driving with Temporal and Global Reasoning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
}
Figure