Introduction
In high-stakes domains such as medicine, finance, and autonomous driving, the demand for machine learning models that are both accurate and interpretable is paramount. While "black box" models like Neural Networks or Gradient Boosting Decision Trees (GBDTs) offer high predictive power, they lack transparency. Conversely, Generalized Additive Models (GAMs) provide interpretability by visualizing feature effects as simple curves, but they often suffer from illogical fluctuations due to data noise and fail to capture complex high-order interactions.
This paper introduces CGA2M+ (Constrained Generalized Additive 2 Model plus), a framework designed to bridge this gap. By enforcing monotonicity constraints and incorporating a specialized term for complex interactions, CGA2M+ achieves state-of-the-art accuracy on tabular data while remaining human-auditable.
Problem Statement: The Interpretability-Accuracy Trade-off
Standard interpretable models like GA2M face two primary challenges:
- Illogical Fluctuations: Noise in training data can lead to nonsensical model behavior, such as predicting that increasing a home's cleanliness would decrease its price.
- Missing Complexity: Most additive models only account for individual or pairwise feature interactions, missing "high-order" relationships that are critical for predictive performance.
Methodology
CGA2M+ addresses these issues through three core innovations:
Monotonicity Constraints
The model allows for the integration of domain knowledge by enforcing logical rules. For instance, if "higher cleanliness" should logically lead to "higher satisfaction," the model's internal curves are constrained to move only upward. This is achieved by penalizing decision tree branches in the underlying LightGBM implementation that violate these rules.
High-Order Interaction Term ($f_{high}$)
To capture patterns involving more than two variables, the authors introduce a "catch-all" function, $f_{high}$. This term ensures that the model does not sacrifice accuracy for the sake of simplicity.
Strategic Training Sequence
To maintain interpretability, the authors employ a specific training sequence. The interpretable individual and pairwise terms are trained first using a backfitting algorithm. The $f_{high}$ term is only trained on the remaining residuals, preventing the complex term from overshadowing the human-readable components.
Experimental Evaluation
The model was evaluated on two tabular datasets: California Housing Prices (regression) and Oricon Cafe Satisfaction (customer survey).
Key Results
CGA2M+ consistently outperformed baselines, including Linear Regression, Random Forests, and standard LightGBM, in terms of Root Mean Squared Error (RMSE):
- California Housing: CGA2M+ achieved a test RMSE of 47,628, surpassing standard GA2M (~50,488) and standard LightGBM (~48,880).
- Oricon Cafe: CGA2M+ reached a test RMSE of 1.136, the best among all tested models.
Discussion and Insights
- Regularization via Knowledge: Enforcing monotonicity acts as a form of regularization, preventing overfitting to noise and improving generalization on unseen data.
- Interpretability Diagnostics: The importance of the $f_{high}$ term serves as a diagnostic tool. Low importance for $f_{high}$ confirms that the interpretable curves are capturing the primary signal in the data.
- Business Utility: The resulting smooth, logical curves are significantly more useful for business decision-making than the erratic outputs of unconstrained models.
Conclusion and Future Work
CGA2M+ provides a robust framework for high-performance, interpretable machine learning. Future research will focus on automating feature selection for monotonicity constraints and exploring methods to further interpret the $f_{high}$ term in datasets where high-order interactions are dominant.
Summary updated