Model learning with personalized interpretability estimation (ML-PIE)

Marco Virgolin, Andrea De Lorenzo, Francesca Randone, Eric Medvet, Mattias Wahde

TL;DR

This paper introduces ML-PIE, a novel approach for automatically generating machine learning models that are not only accurate but also interpretable according to an individual user's preferences. By combining evolutionary algorithms with active learning, ML-PIE learns a personalized notion of interpretability, leading to models that users often prefer over those generated with generic interpretability metrics.

What Problem Does This Paper Address?

In high-stakes applications, it's crucial for AI models to be explainable. While many methods exist to make complex "black-box" models understandable (XAI), they often have limitations. An alternative is to directly synthesize models that are inherently interpretable. However, interpretability is subjective and depends on the user's background and the specific context. Existing methods for synthesizing interpretable models often rely on coarse, generic measures of interpretability (like model size) that don't account for individual user preferences. This paper tackles the challenge of creating a system that can learn models tailored to a specific user's notion of interpretability, moving beyond one-size-fits-all solutions.

Methods

The core of the proposed approach, ML-PIE (Model Learning with Personalized Interpretability Estimation), is a bi-objective evolutionary algorithm. This algorithm simultaneously optimizes for model accuracy and a personalized measure of interpretability. The twist is that this personalized interpretability is not predefined but is learned concurrently with the model synthesis process.

Here's how it works:

  • Evolutionary Algorithm (Genetic Programming - GP): ML-PIE uses GP to evolve symbolic functions (models). The GP population is evaluated based on two objectives: accuracy (e.g., mean squared error for regression) and interpretability.
  • Personalized Interpretability Estimator: A neural network acts as an estimator for interpretability. This estimator is trained using feedback provided by the user.
  • Active Learning: To minimize the user's effort, ML-PIE employs an active learning strategy. It identifies models for which the interpretability estimator is most uncertain and presents these to the user for feedback.
  • User Feedback: The user's role is simple: when presented with two models, they indicate which one they find more interpretable. This feedback is used to update the neural network estimator.
  • Concurrent Learning: The model synthesis (GP) and the interpretability estimator training happen in parallel. The updated estimator then influences the GP's search for better models.

Datasets

The experiments were conducted using two real-world datasets:

  • Boston Housing: A regression dataset concerning housing values.
  • German Credit: A binary classification dataset for credit approval.

Both datasets are from the UCI repository and are of moderate size, suitable for user studies. They were chosen for their relevance to fairness in AI, as they contain features related to race and sex.

Key Insights and Ideas

  • Interpretability is Personal: The study strongly suggests that what one user finds interpretable can differ significantly from another's perspective. This highlights the need for personalized approaches.
  • Active Learning for Efficiency: By focusing user feedback on models where the interpretability estimator is most uncertain, ML-PIE efficiently learns a personalized interpretability model with minimal user input.
  • Neural Networks as Uncertainty-Aware Estimators: Using a neural network with dropout at prediction time allows the system to estimate not just interpretability but also its own uncertainty, which is crucial for the active learning strategy.
  • Trade-offs are Key: The bi-objective evolutionary approach naturally explores the trade-off space between accuracy and interpretability, allowing users to select a model that best suits their needs.

Key Results

  • Personalized Models Preferred: In a user study with 61 engineering students, models discovered using ML-PIE (with personalized interpretability) were generally preferred over models found using generic interpretability indices (like model size or a pre-trained estimator).
  • Moderate Preference Gains: While the preference for personalized models was moderate (around 10-20% in some cases), it's considered a promising result, especially given the generic features used and the lack of domain expertise from the student participants.
  • Estimator Performance: The interpretability estimator learned user-specific notions of interpretability effectively, showing low uncertainty and few mispredictions after a warm-up phase.
  • Model Complexity Trends: The study confirmed that as the desired interpretability level increased (moving along the trade-off front), the discovered models generally became simpler (e.g., smaller size, fewer operations).

Limitations and Future Work

  • Generic Features: The current interpretability estimator relies on generic model features (like size, number of operations). The authors suggest that incorporating domain-specific features and involving actual domain experts could significantly improve the personalization and user preference for ML-PIE models.
  • Student Participants: The study used engineering students, not domain experts. While this provided a large cohort, their understanding of the specific datasets (housing, credit) was limited, potentially impacting the depth of personalization.
  • Model Synthesis Algorithm: While ML-PIE is designed to be agnostic to the model synthesis algorithm, it was demonstrated with Genetic Programming. Future work could explore its application with other methods.
  • Concurrency and Halting: The current system allows user feedback to occur concurrently with model synthesis. Future work might investigate strategies for halting the synthesis process based on the estimator's uncertainty to optimize user effort and system performance.
  • Broader Applications: The authors envision adapting ML-PIE to other model types (e.g., decision trees) and exploring its use in guiding model synthesis based on a holistic "trust" metric that combines accuracy, interpretability, and soundness.

Summary updated

Abstract

High-stakes applications require AI-generated models to be interpretable. Current algorithms for the synthesis of potentially interpretable models rely on objectives or regularization terms that represent interpretability only coarsely (e.g., model size) and are not designed for a specific user. Yet, interpretability is intrinsically subjective. In this paper, we propose an approach for the synthesis of models that are tailored to the user by enabling the user to steer the model synthesis process according to her or his preferences. We use a bi-objective evolutionary algorithm to synthesize models with trade-offs between accuracy and a user-specific notion of interpretability. The latter is estimated by a neural network that is trained concurrently to the evolution using the feedback of the user, which is collected using uncertainty-based active learning. To maximize usability, the user is only asked to tell, given two models at the time, which one is less complex. With experiments on two real-world datasets involving 61 participants, we find that our approach is capable of learning estimations of interpretability that can be very different for different users. Moreover, the users tend to prefer models found using the proposed approach over models found using non-personalized interpretability indices.
BibTeX
@inproceedings{virgolin2021model,
  author = {Marco Virgolin and Andrea De Lorenzo and Francesca Randone and Eric Medvet and Mattias Wahde},
  title = {{Model learning with personalized interpretability estimation (ML-PIE)}},
  booktitle = {GECCO Companion},
  year = {2021},
  url = {https://openreview.net/forum?id=obCFItvQCd},
}
Figure