Multi-Objective Optimization of Performance and Interpretability of Tabular Supervised Machine Learning Models

Lennart Schneider, Bernd Bischl, Janek Thomas

Introduction

In high-stakes domains such as finance, medicine, and insurance, machine learning models must be more than just accurate; they must be trustworthy and auditable. While state-of-the-art gradient-boosted trees (e.g., XGBoost) provide exceptional predictive performance on tabular data, they often operate as "black boxes," utilizing hundreds of features in complex, non-linear ways. Conversely, inherently interpretable models like Explainable Boosting Machines (EBMs) or logistic regression often sacrifice accuracy for simplicity.

This research introduces EAGGA (Evolutionary Algorithm for Grouping Genetic Algorithm), a framework designed to automate the search for models that balance this trade-off. By treating interpretability as a multi-objective optimization problem, EAGGA generates a Pareto front of models, allowing users to select the optimal balance between performance and transparency.

The EAGGA Framework

The framework quantifies interpretability through three global sparsity measures:

  1. Feature Sparsity ($NF$): The fraction of available features utilized by the model.
  2. Interaction Sparsity ($NI$): The fraction of possible feature interactions permitted.
  3. Non-monotone Sparsity ($NNM$): The fraction of features allowed to exhibit complex, non-monotone effects (e.g., ensuring that increasing income never negatively impacts a credit score).
The EAGGA framework integrates feature selection, interaction constraints, and monotonicity into a multi-objective evolutionary search.

Unlike standard hyperparameter optimization, which focuses on settings like learning rate, EAGGA evolves the group structure of features. It partitions features into groups that share interaction and monotonicity constraints. To enhance efficiency, the algorithm uses data-driven "detectors" (such as information gain and Spearman correlation) to intelligently initialize the population before evolving these structures toward the Pareto optimal set.

Experimental Evaluation

The framework was evaluated across 20 binary classification tasks from the OpenML CC-18 and AutoML benchmarks, ranging from 4 to 970 features. EAGGA, using XGBoost as its base learner, was compared against a suite of tuned competitors, including EBMs, Random Forests, and Elastic-Nets.

Key Results

The primary metric for evaluation was the Mean Dominated Hypervolume, which measures the quality of the Pareto front—essentially how well a method optimizes both accuracy (AUC) and interpretability simultaneously.

Comparison of Mean Dominated Hypervolume across 20 tasks, showing EAGGA variants consistently outperforming standard competitors.

As illustrated in the results, EAGGA consistently found models that "dominated" the competition. In many instances, EAGGA identified models that matched the AUC of a fully-tuned "black box" XGBoost while utilizing significantly fewer features and interactions. Furthermore, a significant portion of models produced by standard EBMs and Random Forests were found to be Pareto-dominated by EAGGA's solutions, meaning EAGGA could provide a model that was simultaneously more accurate and more interpretable.

Conclusion

EAGGA shifts the paradigm of interpretable machine learning from manual trial-and-error to automated multi-objective discovery. By providing a "menu" of Pareto-optimal models, it empowers domain experts to make informed decisions about the cost of interpretability. Future work will explore extending this framework to deep neural network architectures and implementing self-adaptive mutation rates to further refine the evolutionary search process.

Summary updated

BibTeX
@inproceedings{schneider2023multi,
  author = {Lennart Schneider and Bernd Bischl and Janek Thomas},
  title = {{Multi-Objective Optimization of Performance and Interpretability of Tabular Supervised Machine Learning Models}},
  booktitle = {GECCO},
  year = {2023},
  doi = {10.1145/3583131.3590380},
}

Loading related content...

Figure