Self-Adaptive Driving in Nonstationary Environments through Conjectural Online Lookahead Adaptation

Tao Li, Tao Li 0046, Haozhe Lei, Quanyan Zhu

Introduction

Deep reinforcement learning (RL) has advanced autonomous driving, but static policies often fail in nonstationary environments. A model trained for clear skies may struggle when a sudden rainstorm occurs. While meta-learning offers a path to adaptation, traditional methods like Model-Agnostic Meta-Learning (MAML) typically require offline batches of data, which is impractical for real-time driving. This paper presents Conjectural Online Lookahead Adaptation (COLA), a framework for real-time policy adjustment in shifting environments.

The COLA Framework

The core of the proposed approach is the Online Meta-Adaptation Learning (OMAL) framework. COLA operates in three distinct phases to ensure the vehicle adapts its behavior as environmental conditions change:

  1. Belief Calibration: The system first identifies the current environmental "mode" (e.g., weather) using a ResNet-based classifier. To ensure stability and avoid rapid switching of predicted modes due to visual noise, Bayesian filtering is applied. This leverages temporal consistency—the fact that weather conditions usually change gradually.
  2. Conjectural Lookahead: Rather than being purely reactive, the agent performs a "conjectural" lookahead. It estimates its future performance over a horizon $K$ based on its current belief of the environment.
  3. Online Optimization: Using off-policy data from the meta-training phase as a proxy for future trajectories, the agent calculates gradients to update its policy parameters ($\theta$) at every time step. A trust-region constraint is employed to maintain policy stability during these rapid updates.
The COLA framework architecture, illustrating the three stages: Belief Calibration, Lookahead, and Conjecture Lookahead Optimization.

Experimental Evaluation

The authors evaluated COLA using the CARLA simulator (v0.9.4). The agent was meta-trained on CloudyNoon and MidRainyNoon conditions. Testing was conducted on a "Dynamic Weather" benchmark where conditions like cloudiness and precipitation varied periodically, forcing the agent to adapt continuously. The model processes 3-channel RGB images from a front-facing camera to make these real-time decisions.

Results and Discussion

COLA demonstrated significant improvements over standard RL and offline meta-learning baselines. In the dynamic weather scenario, COLA achieved an average episode reward of 159.89, substantially higher than MAML's 100.12.

A critical metric for online learning is "regret"—the cumulative difference in performance between the adaptive agent and an "Oracle" with perfect foresight. COLA's regret was shown to grow sublinearly, indicating that its performance asymptotically approaches that of the Oracle.

Furthermore, the inclusion of Bayesian filtering proved vital. As shown in the accuracy plots, filtering smoothed out perception, preventing the "flickering" of beliefs that can lead to erratic driving. Qualitatively, COLA exhibited human-like adaptation, such as slowing down during heavy rain to maintain lane stability and accelerating once conditions cleared.

Performance metrics for COLA: (a) rewards in static modes, (b) rewards in dynamic weather, (c) cumulative regret over time, and (d) classification accuracy with and without Bayesian filtering.

Conclusion and Future Work

COLA addresses the brittleness of RL in nonstationary driving environments by enabling real-time, conjectural adaptation. By combining belief calibration with lookahead optimization, it achieves near-optimal performance in varying weather.

Despite these successes, limitations remain. The current implementation uses a discrete action space, whereas real-world driving requires continuous control. Additionally, the computational intensity of performing lookahead optimization at every time step remains a challenge for scaling to more complex networks. Future work will look into continuous action spaces and more diverse environmental modes, such as night driving or fog, to further bridge the gap to real-world deployment.

Summary updated

Abstract

Powered by deep representation learning, reinforcement learning (RL) provides an end-to-end learning framework capable of solving self-driving (SD) tasks without manual designs. However, time-varying nonstationary environments cause proficient but specialized RL policies to fail at execution time. For example, an RL-based SD policy trained under sunny days does not generalize well to rainy weather. Even though meta learning enables the RL agent to adapt to new tasks/environments, its offline operation fails to equip the agent with online adaptation ability when facing nonstationary environments. This work proposes an online meta reinforcement learning algorithm based on the \emph{conjectural online lookahead adaptation} (COLA). COLA determines the online adaptation at every step by maximizing the agent's conjecture of the future performance in a lookahead horizon. Experimental results demonstrate that under dynamically changing weather and lighting conditions, the COLA-based self-adaptive driving outperforms the baseline policies in terms of online adaptability. A demo video, source code, and appendixes are available at {\tt https://github.com/Panshark/COLA}
BibTeX
@inproceedings{li2022self,
  author = {Tao Li and Tao Li and Haozhe Lei and Quanyan Zhu},
  title = {{Self-Adaptive Driving in Nonstationary Environments through Conjectural Online Lookahead Adaptation}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2022},
}
Figure