Multi-Objective Optimization of Performance and Interpretability of Tabular Supervised Machine Learning Models

Lennart Schneider, Bernd Bischl, Janek Thomas

Introduction

In high-stakes domains such as finance, medicine, and insurance, machine learning models must be more than just accurate; they must be trustworthy and auditable. While state-of-the-art gradient-boosted trees (e.g., XGBoost) provide exceptional predictive performance on tabular data, they often operate as "black boxes," utilizing hundreds of features in complex, non-linear ways. Conversely, inherently interpretable models like Explainable Boosting Machines (EBMs) or logistic regression often sacrifice accuracy for simplicity.

This research introduces EAGGA (Evolutionary Algorithm for Grouping Genetic Algorithm), a framework designed to automate the search for models that balance this trade-off. By treating interpretability as a multi-objective optimization problem, EAGGA generates a Pareto front of models, allowing users to select the optimal balance between performance and transparency.

The EAGGA Framework

The framework quantifies interpretability through three global sparsity measures:

  1. Feature Sparsity ($NF$): The fraction of available features utilized by the model.
  2. Interaction Sparsity ($NI$): The fraction of possible feature interactions permitted.
  3. Non-monotone Sparsity ($NNM$): The fraction of features allowed to exhibit complex, non-monotone effects (e.g., ensuring that increasing income never negatively impacts a credit score).
The EAGGA framework integrates feature selection, interaction constraints, and monotonicity into a multi-objective evolutionary search.

Unlike standard hyperparameter optimization, which focuses on settings like learning rate, EAGGA evolves the group structure of features. It partitions features into groups that share interaction and monotonicity constraints. To enhance efficiency, the algorithm uses data-driven "detectors" (such as information gain and Spearman correlation) to intelligently initialize the population before evolving these structures toward the Pareto optimal set.

Experimental Evaluation

The framework was evaluated across 20 binary classification tasks from the OpenML CC-18 and AutoML benchmarks, ranging from 4 to 970 features. EAGGA, using XGBoost as its base learner, was compared against a suite of tuned competitors, including EBMs, Random Forests, and Elastic-Nets.

Key Results

The primary metric for evaluation was the Mean Dominated Hypervolume, which measures the quality of the Pareto front—essentially how well a method optimizes both accuracy (AUC) and interpretability simultaneously.

Comparison of Mean Dominated Hypervolume across 20 tasks, showing EAGGA variants consistently outperforming standard competitors.

As illustrated in the results, EAGGA consistently found models that "dominated" the competition. In many instances, EAGGA identified models that matched the AUC of a fully-tuned "black box" XGBoost while utilizing significantly fewer features and interactions. Furthermore, a significant portion of models produced by standard EBMs and Random Forests were found to be Pareto-dominated by EAGGA's solutions, meaning EAGGA could provide a model that was simultaneously more accurate and more interpretable.

Conclusion

EAGGA shifts the paradigm of interpretable machine learning from manual trial-and-error to automated multi-objective discovery. By providing a "menu" of Pareto-optimal models, it empowers domain experts to make informed decisions about the cost of interpretability. Future work will explore extending this framework to deep neural network architectures and implementing self-adaptive mutation rates to further refine the evolutionary search process.

Summary updated

Abstract

We present a model-agnostic framework for jointly optimizing the predictive performance and interpretability of supervised machine learning models for tabular data. Interpretability is quantified via three measures: feature sparsity, interaction sparsity of features, and sparsity of non-monotone feature effects. By treating hyperparameter optimization of a machine learning algorithm as a multi-objective optimization problem, our framework allows for generating diverse models that trade off high performance and ease of interpretability in a single optimization run. Efficient optimization is achieved via augmentation of the search space of the learning algorithm by incorporating feature selection, interaction and monotonicity constraints into the hyperparameter search space. We demonstrate that the optimization problem effectively translates to finding the Pareto optimal set of groups of selected features that are allowed to interact in a model, along with finding their optimal monotonicity constraints and optimal hyperparameters of the learning algorithm itself. We then introduce a novel evolutionary algorithm that can operate efficiently on this augmented search space. In benchmark experiments, we show that our framework is capable of finding diverse models that are highly competitive or outperform state-of-the-art XGBoost or Explainable Boosting Machine models, both with respect to performance and interpretability.
BibTeX
@inproceedings{schneider2023multi,
  author = {Lennart Schneider and Bernd Bischl and Janek Thomas},
  title = {{Multi-Objective Optimization of Performance and Interpretability of Tabular Supervised Machine Learning Models}},
  booktitle = {GECCO},
  year = {2023},
  doi = {10.1145/3583131.3590380},
}
Figure