Constrained Generalized Additive 2 Model With Consideration of High-Order Interactions

Akihisa Watanabe, Michiya Kuramata, Kaito Majima, Haruka Kiyohara, Kondo Kensho, Kazuhide Nakata

Introduction

In high-stakes domains such as medicine, finance, and autonomous driving, the demand for machine learning models that are both accurate and interpretable is paramount. While "black box" models like Neural Networks or Gradient Boosting Decision Trees (GBDTs) offer high predictive power, they lack transparency. Conversely, Generalized Additive Models (GAMs) provide interpretability by visualizing feature effects as simple curves, but they often suffer from illogical fluctuations due to data noise and fail to capture complex high-order interactions.

This paper introduces CGA2M+ (Constrained Generalized Additive 2 Model plus), a framework designed to bridge this gap. By enforcing monotonicity constraints and incorporating a specialized term for complex interactions, CGA2M+ achieves state-of-the-art accuracy on tabular data while remaining human-auditable.

Problem Statement: The Interpretability-Accuracy Trade-off

Standard interpretable models like GA2M face two primary challenges:

  1. Illogical Fluctuations: Noise in training data can lead to nonsensical model behavior, such as predicting that increasing a home's cleanliness would decrease its price.
  2. Missing Complexity: Most additive models only account for individual or pairwise feature interactions, missing "high-order" relationships that are critical for predictive performance.
Figure 1: Conceptual illustration of illogical fluctuations where price decreases despite increasing cleanliness, a common issue in unconstrained models.

Methodology

CGA2M+ addresses these issues through three core innovations:

Monotonicity Constraints

The model allows for the integration of domain knowledge by enforcing logical rules. For instance, if "higher cleanliness" should logically lead to "higher satisfaction," the model's internal curves are constrained to move only upward. This is achieved by penalizing decision tree branches in the underlying LightGBM implementation that violate these rules.

High-Order Interaction Term ($f_{high}$)

To capture patterns involving more than two variables, the authors introduce a "catch-all" function, $f_{high}$. This term ensures that the model does not sacrifice accuracy for the sake of simplicity.

Strategic Training Sequence

To maintain interpretability, the authors employ a specific training sequence. The interpretable individual and pairwise terms are trained first using a backfitting algorithm. The $f_{high}$ term is only trained on the remaining residuals, preventing the complex term from overshadowing the human-readable components.

Experimental Evaluation

The model was evaluated on two tabular datasets: California Housing Prices (regression) and Oricon Cafe Satisfaction (customer survey).

Key Results

CGA2M+ consistently outperformed baselines, including Linear Regression, Random Forests, and standard LightGBM, in terms of Root Mean Squared Error (RMSE):

  • California Housing: CGA2M+ achieved a test RMSE of 47,628, surpassing standard GA2M (~50,488) and standard LightGBM (~48,880).
  • Oricon Cafe: CGA2M+ reached a test RMSE of 1.136, the best among all tested models.
Figure 6: Learned monotonic shape functions for the Oricon Cafe dataset, demonstrating smooth and logical relationships between features like Cleanliness and satisfaction.

Discussion and Insights

  • Regularization via Knowledge: Enforcing monotonicity acts as a form of regularization, preventing overfitting to noise and improving generalization on unseen data.
  • Interpretability Diagnostics: The importance of the $f_{high}$ term serves as a diagnostic tool. Low importance for $f_{high}$ confirms that the interpretable curves are capturing the primary signal in the data.
  • Business Utility: The resulting smooth, logical curves are significantly more useful for business decision-making than the erratic outputs of unconstrained models.

Conclusion and Future Work

CGA2M+ provides a robust framework for high-performance, interpretable machine learning. Future research will focus on automating feature selection for monotonicity constraints and exploring methods to further interpret the $f_{high}$ term in datasets where high-order interactions are dominant.

Summary updated

Abstract

In recent years, machine learning and AI have been introduced in many industrial fields. In fields such as finance, medicine, and autonomous driving, where the inference results of a model may have serious consequences, high interpretability, as well as prediction accuracy is required. In this study, we propose Constrained Generalized Additive 2 Model plus (CGA <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> M+), which is based on the Generalized Additive 2 Model (GA <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> M) and differs from it in two major points. The first is the introduction of monotonicity. Enforcing monotonicity constraints on some features using an analyst's knowledge, is expected to improve not only interpretability but also generalization performance. The second is the introduction of a higher-order term: given that GA2M considers only up to second-order interactions, we introduce a higher-order term, which can capture higher-order interactions and improve prediction performance. In this process, the higher-order term does not reduce interpretability by applying our training method. Numerical experiments showed that the proposed model has high prediction accuracy and interpretability. Furthermore, we confirmed that generalization performance is improved by imposing monotonicity constraints.
BibTeX
@inproceedings{watanabe2021constrained,
  author = {Akihisa Watanabe and Michiya Kuramata and Kaito Majima and Haruka Kiyohara and Kondo Kensho and Kazuhide Nakata},
  title = {{Constrained Generalized Additive 2 Model With Consideration of High-Order Interactions}},
  booktitle = {2021},
  year = {2021},
  doi = {10.1109/icecet52533.2021.9698779},
}
Figure