Introduction
Designing safe systems for high-stakes environments like aviation or medicine requires accurate predictions of human behavior. Traditionally, this necessitates Human-In-The-Loop (HITL) simulations. While these "high-fidelity" (Hi-Fi) simulations provide high-quality data from expert participants using realistic hardware, they are prohibitively expensive and time-consuming. Conversely, "low-fidelity" (Lo-Fi) data—such as that collected via crowdsourcing platforms like Amazon Mechanical Turk—is inexpensive and abundant but often lacks the accuracy of expert behavior in real-world scenarios.
This paper, originally published in 2012, addresses this "fidelity gap" by proposing a multi-fidelity framework to fuse these disparate data sources into highly accurate predictive models, even when expert data is scarce.
The Visual Self-Separation Testbed
To evaluate their framework, the authors utilized a visual self-separation game as a testbed. In this scenario, two pilots find themselves on a near-collision course. Each must decide on a heading change to avoid a crash while simultaneously attempting to minimize deviation from their original flight path.
The study compared two distinct levels of fidelity:
- High-Fidelity (Hi-Fi): A simulation featuring a full instrument panel and "expert" behavior (simulated via specific utility weights).
- Low-Fidelity (Lo-Fi): A simplified version where the instrument panel is removed, forcing participants to rely solely on out-the-window visual cues.
Methodology: Model-Free vs. Model-Based Fusion
The core of the research lies in how these data sources are combined. The authors explored two primary approaches:
1. Model-Free Approaches
These methods utilize Locally Weighted (LW) Regression. In the multi-fidelity variant, Lo-Fi data is used to train a "pre-predictor." The output of this pre-predictor then serves as an additional feature for the model trained on the limited Hi-Fi data.
2. Model-Based Approaches
These methods incorporate a structural model of human decision-making, specifically assuming a level-$k$ reasoning strategy. In this context, a level-1 ($k=1$) strategy assumes the other pilot will maintain their current course, and the actor reacts accordingly. The model estimates "utility weights" that represent the trade-off between collision avoidance and path efficiency.
- MAP (Maximum A Posteriori) Multi-fidelity: This approach finds the optimal utility weights by combining the likelihood of observed actions from both Lo-Fi and Hi-Fi sources.
- Bayesian Multi-fidelity: This more robust method accounts for uncertainty by sampling across a distribution of utility weights, weighted by their ability to explain both data sets.
Key Results and Insights
The experimental results highlight a significant advantage for model-based methods. By assuming humans are "bounded rational" agents, the algorithms could learn the underlying logic of the behavior from Lo-Fi data, even when specific actions differed from the Hi-Fi experts.
Key findings include:
- Predictive Efficiency: Model-based multi-fidelity methods achieved high efficiency (near 0.9) with as few as 10 to 20 Hi-Fi samples when supplemented by 1,000 Lo-Fi samples.
- The Failure of Model-Free Methods: Standard regression-based fusion performed poorly, often failing to reach 0.7 efficiency. This suggests that raw data fusion is insufficient without an underlying model of human intent.
- Sufficiency of Level-1 Reasoning: Consistent with previous behavioral research, the authors found that assuming a shallow depth of reasoning ($k=1$) was sufficient to model interactions in these stressful, time-critical scenarios.
Conclusion
The research demonstrates that incorporating a model of human decision-making is crucial for bridging the fidelity gap. Lo-Fi data serves as an effective "prior," narrowing the search space for human-like behavior, which sparse Hi-Fi data can then refine. This multi-fidelity approach offers a path toward developing highly accurate predictive models for complex human-machine systems without the prohibitive cost of large-scale expert simulations.
Summary updated