Uncertainty-aware hybrid paradigm of nonlinear MPC and model-based RL for offroad navigation: Exploration of transformers in the predictive model

Faraz Lotfi, Khalil Virji, Farnoosh Faraji, Lucas Berry, Andrew Holliday, David Meger, Gregory Dudek

Abstract

High-speed autonomous navigation in unstructured off-road environments remains a significant challenge due to complex vehicle-terrain interactions and the lack of structured features. This paper proposes a hybrid navigation paradigm that combines model-based Reinforcement Learning (RL) for long-term semantic planning with Nonlinear Model Predictive Control (MPC) for physically constrained throttle management. By utilizing a Transformer-based predictive model instead of traditional LSTMs, the system achieves superior accuracy in early-stage predictions. Additionally, an ensemble-based uncertainty metric is incorporated to modulate vehicle speed in unfamiliar terrains, ensuring safer and more robust navigation.

Introduction

Off-road navigation is notoriously difficult because the "rules" of the road do not exist. There are no lanes, and the terrain—ranging from tall grass to deep mud—drastically affects vehicle dynamics. Existing state-of-the-art systems, such as BADGR, typically rely on image-based predictive models to anticipate future states. However, these models often suffer from two major flaws:

  1. Temporal Lag: Traditional LSTM-based models struggle to make accurate predictions for the immediate future because they require several timesteps to accumulate historical context.
  2. Physics Blindness: Many models ignore the vehicle’s actual kinematics and environmental uncertainty, leading to overconfident or physically impossible maneuvers.

Methodology

The researchers developed a two-module hybrid architecture designed to balance high-level semantic intelligence with low-level physical safety.

Transformer-based Predictive Model

Instead of using an LSTM to process image sequences and actions, the authors implemented a Transformer architecture. Transformers utilize self-attention mechanisms to weigh the importance of different parts of the input sequence simultaneously. This allows the model to provide high-accuracy predictions from the very first timestep, which is critical for real-time re-planning in dynamic environments.

The Transformer-based architecture processes image sequences and actions to predict future terrain events.

Uncertainty-Aware Nonlinear MPC

While the RL component suggests steering angles based on visual data, a nonlinear MPC controller manages the throttle. This controller incorporates a mathematical model of the vehicle's physics (including friction and slope) and a reward function that penalizes high speeds in areas of high epistemic uncertainty. To estimate this uncertainty, the team trained an ensemble of five models. By measuring the Mutual Information (MI) between the models' outputs, the system quantifies model disagreement, signaling the robot to slow down in novel or ambiguous situations.

Key Results and Insights

The proposed system was evaluated using a newly released dataset of over 20,000 manually annotated samples across nine distinct terrain classes.

Predictive Performance

A key finding was the "warm-up" problem inherent in LSTMs. LSTMs often default to common classes like "smooth road" until they have processed enough timesteps. In contrast, the Transformer-based model maintained high Macro-F1 scores across the entire planning horizon.

Comparison of Macro-Precision, Macro-Recall, and Macro-F1 scores between LSTM and Transformer models across various planning horizons.

As shown in the metrics above, the Transformer architecture significantly outperforms the LSTM baseline (BADGR) in the initial timesteps. The Vision Transformer (ViT-L/16) achieved a top accuracy of 87.8% on the classification task.

Safety and Adaptability

In real-world trials, the uncertainty-integrated MPC achieved an average expected return of 64.6, compared to 59.7 for a model that ignored uncertainty. This improvement indicates that the robot was better at selecting safer and faster terrains by proactively reducing speed when the predictive model's confidence was low.

Conclusion

The hybrid synergy between RL-based semantic navigation and MPC-based physical control provides a robust framework for off-road autonomy. While the Transformer architecture offers significant gains in predictive accuracy, future work will focus on reducing the computational overhead of running ensembles and MPC solvers simultaneously. Additionally, exploring sim-to-real transfer could further accelerate the deployment of these systems in diverse real-world environments.

Summary updated

Abstract

In this paper, we investigate a hybrid scheme that combines nonlinear model predictive control (MPC) and model-based reinforcement learning (RL) for navigation planning of an autonomous model car across offroad, unstructured terrains without relying on predefined maps. Our innovative approach takes inspiration from BADGR, an LSTM-based network that primarily concentrates on environment modeling, but distinguishes itself by substituting LSTM modules with transformers to greatly elevate the performance of our model. Addressing uncertainty within the system, we train an ensemble of predictive models and estimate the mutual information between model weights and outputs, facilitating dynamic horizon planning through the introduction of variable speeds. Further enhancing our methodology, we incorporate a nonlinear MPC controller that accounts for the intricacies of the vehicle’s model and states. The model-based RL facet produces steering angles and quantifies inherent uncertainty. At the same time, the nonlinear MPC suggests optimal throttle settings, striking a balance between goal attainment speed and managing model uncertainty influenced by velocity. In the conducted studies, our approach excels over the existing baseline by consistently achieving higher metric values in predicting future events and seamlessly integrating the vehicle’s kinematic model for enhanced decision-making. The code and the evaluation data are available at (Github-repo).
BibTeX
@inproceedings{lotfi2023uncertainty,
  author = {Faraz Lotfi and Khalil Virji and Farnoosh Faraji and Lucas Berry and Andrew Holliday and David Meger and Gregory Dudek},
  title = {{Uncertainty-aware hybrid paradigm of nonlinear MPC and model-based RL for offroad navigation: Exploration of transformers in the predictive model}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
}
Figure