Abstract
We propose and demonstrate a framework called perception as prediction for autonomous driving that uses general value functions (GVFs) to learn predictions. Perception as prediction learns data-driven predictions relating to the impact of actions on the agent's perception of the world. It also provides a data-driven approach to predict the impact of the anticipated behavior of other agents on the world without explicitly learning their policy or intentions. We demonstrate perception as prediction by learning to predict an agent's front safety and rear safety with GVFs, which encapsulate anticipation of the behavior of the vehicle in front and in the rear, respectively. The safety predictions are learned through random interactions in a simulated environment containing other agents. We show that these predictions can be used to produce similar control behavior to an LQR-based controller in an adaptive cruise control problem as well as provide advanced warning when the vehicle behind is approaching dangerously. The predictions are compact policy-based predictions that support prediction of the long term impact on safety when following a given policy. We analyze two controllers that use the learned predictions in a racing simulator to understand the value of the predictions and demonstrate their use in the real-world on a Clearpath Jackal robot and an autonomous vehicle platform.
Introduction
Traditional autonomous driving architectures typically separate perception (detecting objects) from planning (deciding where to go). While effective, this separation often leads to computationally expensive planning phases. For example, standard Adaptive Cruise Control (ACC) systems frequently rely on Model Predictive Control (MPC), which requires simulating thousands of possible future states to find an optimal path. Furthermore, explicitly modeling the intentions and trajectories of every nearby agent is notoriously difficult.
This paper introduces a "perception as prediction" framework that leverages General Value Functions (GVFs) to help autonomous vehicles anticipate safety risks. Instead of merely identifying obstacles, the system learns to predict the long-term impact of its own actions on its perception of safety, implicitly accounting for the behavior of others without needing to model them explicitly.
The Perception as Prediction Framework
The core of the proposed framework is the integration of safety and comfort predictors directly into the control loop. By using GVFs, the vehicle can ask "what if" questions: "What will happen to my future safety if I continue with my current policy?"
General Value Functions (GVFs)
GVFs are used to frame perception as a predictive task. The researchers define safety as a "cumulant" (a pseudo-reward signal). For instance, if a vehicle enters a predefined "unsafe" zone based on headway distance and speed, the cumulant is 1; otherwise, it is 0. Using Temporal Difference (TD) learning, the framework predicts the accumulated future safety over a specific time horizon, typically 1 to 3 seconds. This allows the vehicle to anticipate risks before they manifest as immediate collisions.
Predictive Controllers
Two types of controllers were developed to utilize these GVF predictions:
- Fuzzy Predictive Controller: This controller uses fuzzy logic to balance multiple objectives, including safety, target speed, and passenger comfort. It maps predicted safety values to control actions like braking or acceleration.
- Rule-based Predictive Controller: A simpler implementation that triggers specific acceleration adjustments when predicted future safety thresholds are crossed.
Key Results
The framework was evaluated across several simulated environments (TORCS, Gazebo, Webots) and transferred to real-world platforms. In high-speed highway scenarios within the TORCS simulator, the GVF-based controllers demonstrated performance comparable to industry-standard Linear-Quadratic Regulator (LQR) baselines for ACC.
Computational Efficiency and Safety
A key insight from the results is the computational efficiency of the GVF approach. While a traditional MPC might require 3,000 predictions per time step to plan a path, the GVF-based fuzzy controller achieved similar performance with only 21 predictions.
Additionally, the system proved capable of predicting "rear safety" risks—anticipating when a vehicle behind was approaching too quickly. This provides a foundation for advanced driver warning systems that can react to hazards from all directions, not just the front.
Conclusion
The "perception as prediction" framework demonstrates that action-oriented, data-driven predictions can simplify the autonomous driving stack. By implicitly modeling multi-agent interactions through GVFs, vehicles can navigate complex scenarios efficiently. Future research will focus on extending this longitudinal control to lateral maneuvers like lane changes and training models directly on large-scale real-world driving data.
Summary updated