Interpretable Models for Understanding Immersive Simulations

Nicholas Hoernle, Kobi Gal, Barbara J. Grosz, Leilah Lyons, Ada Ren, Andee RubinIJCAI 2020 · DOI

TL;DR

This paper tackles the challenge of selecting the most interpretable machine learning models for understanding complex, high-dimensional time series data generated by immersive simulations. They introduce novel interpretability tests and demonstrate that models optimizing for human understanding often differ from those optimized for statistical information criteria, with a fully Bayesian approach showing promise for automated model selection.

What Problem Does This Paper Address?

Immersive simulations, often used in education and healthcare training, generate rich, high-dimensional time series data reflecting user interactions and system dynamics. Understanding the causal relationships between user actions and simulation outcomes is crucial for learning, but the complexity of these systems makes it difficult for humans to discern these connections. Machine learning models can help summarize these interactions, but for them to be truly useful, they must not only accurately represent the data but also be understandable to the users (e.g., students, educators). The core problem is how to select the "best" model when "best" means both statistically sound and interpretable by humans, especially when direct human evaluation is costly or infeasible.

Methods

The authors propose an end-to-end paradigm for evaluating the interpretability of unsupervised time-series models. Their approach involves:

  • Model Family: They use a family of Hidden Markov Models (HMMs), specifically "Sticky-HMMs," which are augmented with a hyperparameter (κ) that biases the model to persist in a given state, effectively segmenting the time series into periods of stable dynamics.
  • Interpretability Tests: They adapt two established interpretability tests from Doshi-Velez and Kim (2017):
  • Forward Simulation: Evaluators predict the system's state based on a model's inferred period.
  • Binary Forced Choice: Evaluators choose between two plausible model explanations for a given system state. These tests are operationalized using visualizations of the simulation's state (session-views) extracted from candidate and "intruder" periods.

  • Model Selection: They compare model selection based on statistical information criteria (DIC, WAIC) against selection based on the performance in their human interpretability tests.

Datasets

The primary dataset used is time series log data from the "Connected Worlds" (CW) immersive ecological simulation. This data captures the levels of water in different biomes over 8-minute sessions at a 1Hz frequency, reflecting student interactions. The authors do not introduce a new benchmark dataset but apply their methods to this real-world, complex simulation data.

Key Insights and Ideas

  • Interpretability vs. Statistical Optimality: A key finding is that models optimized for statistical information criteria (like DIC and WAIC) do not necessarily yield the most interpretable representations for humans. This highlights a fundamental trade-off in model selection for complex systems.
  • The "Sticky" Parameter (κ): The value of the κ parameter in Sticky-HMMs significantly impacts interpretability. There appears to be a "sweet spot" for κ (around MK100-MK150) that balances period length and the ability to capture meaningful dynamics, leading to better human understanding. Very high κ values can lead to overly long periods that obscure details, while very low values might capture too much noise.
  • Binary Forced Choice Superiority: The Binary Forced Choice test proved to be a more reliable and sensitive measure of interpretability compared to the Forward Simulation test, showing clearer peaks in performance and less variance.
  • Fully Bayesian Promise: The fully Bayesian (FB) model, which doesn't require manual hyperparameter tuning (including κ), consistently performed well across both statistical and interpretability measures. This suggests it's a strong candidate for situations where human evaluation is not feasible.

Key Results

  • Model Performance: The MK100 and MK150 models achieved the highest accuracy on the Binary Forced Choice test (83% and 82% respectively).
  • Statistical Criteria: The MK5 model was optimal for DIC and WAIC, but it was not among the top performers for interpretability. The FB model ranked in the top 5 for statistical criteria.
  • Interpretability Ranking: In the Binary Forced Choice test, models MK100 and MK150 showed the highest interpretability, outperforming the random baseline significantly. The FB model also showed strong, consistent performance across tests.
  • Random Baseline: The random baseline model performed poorly on both interpretability tests, underscoring the value of the learned models.

Limitations and Future Work

  • Limited Data Instances: The user study was based on a small number of simulation instances (n=2) due to the difficulty in obtaining controlled sessions. However, the authors note that the statistical significance was maintained due to the large number of evaluators and time points used.
  • Future Work: The authors plan to explore alternative interpretability measures, including designing a counterfactual simulation test. They also aim to apply their approach to additional domains and continue developing methods for generating relevant summaries of simulation dynamics for educational purposes.

Summary updated

BibTeX
@inproceedings{hoernle2020interpretable,
  author = {Nicholas Hoernle and Kobi Gal and Barbara J. Grosz and Leilah Lyons and Ada Ren and Andee Rubin},
  title = {{Interpretable Models for Understanding Immersive Simulations}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2020},
  doi = {10.24963/ijcai.2020/321},
}
Figure