Learning an Urban Air Mobility Encounter Model from Expert Preferences

Sydney M. Katz, Anne-Claire Le Bihan, Mykel J. KochenderferarXiv 2019 · arXiv

Abstract

Developing safety-critical collision avoidance systems for Urban Air Mobility (UAM) requires robust encounter models. However, the absence of operational flight data for UAM vehicles presents a significant barrier. This paper proposes a preference-based learning approach that leverages expert intuition to construct a probabilistic encounter model. By comparing pairs of simulated flight policies, experts can guide the model toward realistic behaviors in approximately 15 minutes of interaction.

Introduction

The safe integration of autonomous aircraft into urban environments depends on rigorous evaluation using encounter models—statistical representations of aircraft behavior during close-proximity events. While traditional models for manned aircraft are derived from vast radar datasets, UAM lacks such historical data. This work addresses this gap by introducing a method to bootstrap encounter models using human expertise through preference-based reinforcement learning.

Methodology

The authors frame UAM trajectory generation as a Markov Decision Process (MDP) with an initially unknown reward function. To recover this function, they employ preference-based learning where experts are presented with pairwise queries. Each query consists of multiple rollouts from two distinct policies, allowing the expert to judge the overall behavioral tendencies rather than isolated paths.

Pairwise comparison of two candidate policies (Policy 1 vs. Policy 2) showing altitude and speed profiles.

A Bayesian preference model updates the distribution over reward parameters based on these choices. The system incorporates prior knowledge, such as the fundamental requirement to land, allowing experts to focus on nuanced trade-offs like jerk penalties and speed regulation near the ground. Active querying methods, specifically Multiobjective Optimization, are used to maximize the information gained from each human interaction.

Results and Discussion

The proposed method demonstrated high efficiency, with the reward function converging after roughly 40 queries. This process is remarkably fast, requiring only about 15 minutes of an expert's time. The system also exhibited significant robustness to human error, maintaining convergence even when experts made "incorrect" choices up to 20% of the time.

A set of trajectories generated from the learned model, demonstrating the stochastic nature of the final encounter model.

Once the reward function is successfully learned, the model can generate a stochastic distribution of trajectories. By adjusting the precision parameter in the softmax policy, researchers can produce the variety of flight paths necessary for comprehensive safety simulations.

Conclusion

Preference-based learning offers a viable path for developing encounter models in data-poor domains like UAM. By effectively capturing expert intuition, this method produces realistic, probabilistic models that are essential for testing next-generation collision avoidance systems. Future research will explore automated feature engineering and the integration of stochasticity directly into the expert querying phase.

Summary updated

BibTeX
@misc{katz2019learning,
  author = {Sydney M. Katz and Anne-Claire Le Bihan and Mykel J. Kochenderfer},
  title = {{Learning an Urban Air Mobility Encounter Model from Expert Preferences}},
  year = {2019},
  eprint = {1907.05575},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1907.05575},
}
Figure