Expanding the Deployment Envelope of Behavior Prediction via Adaptive Meta-Learning

Boris Ivanovic, James Harrison, Marco Pavone

TL;DR

This paper introduces a meta-learning framework that allows behavior prediction models for autonomous vehicles to rapidly adapt to new environments, such as different cities or countries. By replacing the final layer of a neural network with an adaptive Bayesian regression layer, the authors enable efficient online and offline domain transfer, significantly improving both prediction accuracy and uncertainty calibration in unseen settings.

What Problem Does This Paper Address?

Autonomous vehicles (AVs) rely heavily on behavior prediction models to forecast the trajectories of pedestrians and other vehicles. However, these models are typically "brittle"—they are trained on specific geographic regions and struggle to generalize when deployed in a new city with different road geometries, agent types (like Tuk-tuks), or social driving norms (like forming a "Rettungsgasse" in Germany).

Currently, expanding an AV's operational domain requires massive, expensive efforts to collect and manually annotate new data. The authors aim to bridge this gap by creating a model that can adapt "on the fly" from streaming sensor data or from a very small set of pre-annotated samples, effectively expanding the "deployment envelope" of these systems.

Diverse deployment challenges across different countries

Methods

The authors propose a novel architecture that combines recurrent neural networks (RNNs) with ALPaCA (Adaptive Last-Layer-CA), an optimization-based meta-learning technique.

Key technical components include:

  • Adaptive Last Layer: Instead of a standard static weight matrix, the final layer is treated as a Bayesian linear regression problem. The model uses Kalman filtering to update the posterior distribution of these weights efficiently as new data arrives.
  • Aleatoric Uncertainty Modulation: The model explicitly predicts "aleatoric" (irreducible) uncertainty based on the current scene context. Crucially, this uncertainty modulates the adaptation process: if a situation is inherently unpredictable, the model knows to extract less information from it, preventing it from corrupting the learned weights.
  • Hybrid Adaptation: For offline scenarios, the authors combine their analytical Bayesian updates with standard gradient-based fine-tuning. This allows for fast initial adaptation followed by high-capacity refinement.

Datasets

The researchers evaluated their method across several real-world benchmarks:

  • ETH and UCY: Standard pedestrian trajectory datasets from Zurich and Cyprus, used to test scene-to-scene transfer.
  • nuScenes: A large-scale AV dataset featuring data from Boston and Singapore.
  • Lyft Level 5: An AV dataset from Palo Alto.

The transfer from nuScenes to Lyft is particularly challenging because the datasets use different sensor frequencies (2Hz vs 10Hz) and distinct map annotation styles.

Key Insights and Ideas

  • Analytical vs. Gradient Updates: Gradient descent is slow and requires many samples to converge. By using analytical Bayesian updates (Kalman filtering) on the last layer, the model can adapt significantly faster with much less data.
  • Calibration is King: In safety-critical systems like AVs, it’s not enough to be accurate; the model must know when it is uncertain. The authors show that standard fine-tuning often leads to "overfitting" and poor calibration, whereas their Bayesian approach maintains a well-calibrated sense of its own limits.
  • Synergistic Architecture: The paper demonstrates that recurrence (for temporal features) and optimization-based meta-learning (for adaptation) work better together than either does in isolation.

Key Results

  • Rapid Online Adaptation: In the UCY-to-ETH transfer, the proposed method achieved a 90%+ reduction in Average Displacement Error (ADE) in the time it took standard fine-tuning to reach only a 40% reduction.
  • Superior Calibration: As shown in Figure 2 and Figure 5, the method maintains low Expected Calibration Error (ECE) even as it observes more data, whereas standard fine-tuning produces unpredictable and often poorly calibrated results.
  • State-of-the-Art Performance: On the ETH/UCY pedestrian benchmarks, the model's "prior" (before any adaptation) already outperformed several recent specialized architectures like T-GNN.
Comparison of Expected Calibration Error during offline adaptation
ADE reduction during online adaptation

Limitations and Future Work

  • Single-Step Adaptation: To maintain high execution frequencies for real-time use, the authors currently adapt the model based on single-step prediction errors. They acknowledge that adapting to multi-step errors might offer even higher theoretical performance at the cost of complexity.
  • Many-to-One Transfer: The current work focuses on transferring from one source domain to one target domain. The authors suggest that future work should explore "many-to-one" transfer, leveraging diverse data from many cities to create an even more robust and adaptable "universal" prior.
  • Computational Constraints: While the last-layer update is efficient, the underlying graph-structured RNN still carries a computational load that must be managed for deployment on edge hardware.

Summary updated

Abstract

Learning-based behavior prediction methods are increasingly being deployed in real-world autonomous systems, e.g., in fleets of self-driving vehicles, which are beginning to commercially operate in major cities across the world. Despite their advancements, however, the vast majority of prediction systems are specialized to a set of well-explored geographic regions or operational design domains, complicating deployment to additional cities, countries, or continents. Towards this end, we present a novel method for efficiently adapting behavior prediction models to new environments. Our approach leverages recent advances in meta-learning, specifically Bayesian regression, to augment existing behavior prediction models with an adaptive layer that enables efficient domain transfer via offline fine-tuning, online adaptation, or both. Experiments across multiple real-world datasets demonstrate that our method can efficiently adapt to a variety of unseen environments.
BibTeX
@inproceedings{ivanovic2023expanding,
  author = {Boris Ivanovic and James Harrison and Marco Pavone},
  title = {{Expanding the Deployment Envelope of Behavior Prediction via Adaptive Meta-Learning}},
  booktitle = {IEEE International Conference on Robotics and Automation},
  year = {2023},
  doi = {10.1109/icra48891.2023.10161155},
}
Figure