Shape Arithmetic Expressions

Krzysztof Kacprzyk, Mihaela van der SchaarAI4Science @ NeurIPS 2023 · OpenReview

TL;DR

This paper introduces Shape Arithmetic Expressions (SHAREs), a novel modeling framework that combines the interpretability of Generalized Additive Models (GAMs) with the expressive power of symbolic regression. SHAREs aim to uncover transparent, yet complex, mathematical expressions from data, addressing limitations of existing methods when dealing with non-closed-form relationships.

What Problem Does This Paper Address?

Symbolic regression (SR) is great at finding concise, interpretable equations, but it struggles when the underlying data doesn't have a neat closed-form expression (e.g., stress-strain curves). In such cases, SR often produces overly complex and uninterpretable equations. On the other hand, Generalized Additive Models (GAMs) are interpretable because they model relationships as sums of univariate functions, but they fail to capture intricate interactions between variables. This paper bridges this gap by proposing SHAREs, which aim to provide both flexibility in modeling complex interactions and interpretability through visualizable "shape functions."

Methods

The core innovation is the Shape Arithmetic Expression (SHARE). SHAREs are mathematical expressions built from:

  • Shape functions (S): Univariate functions (like splines or neural networks) that are fitted to the data. These are the interpretable components.
  • Binary operations (B): Standard arithmetic operations (+, ×, ÷).
  • Variables and constants.

SHAREs are represented as expression trees. The key novelty lies in how they combine GAMs' interpretable univariate functions with SR's ability to model complex interactions. The authors also introduce a set of rules for constructing "transparent" SHAREs, ensuring that the resulting expressions are understandable by focusing on disjoint variable sets for binary operations and disallowing nested shape functions. The implementation uses a nested optimization approach: a genetic programming algorithm searches for the symbolic structure, while gradient descent optimizes the parameters of the neural network-based shape functions.

Datasets

The paper uses several datasets for evaluation:

  • Stress-strain curves of aluminum 6061-T651: Used to demonstrate SR's struggle with non-closed-form data.
  • Feynman Symbolic Regression Database: A collection of equations used to test the ability of GAMs, GA2Ms, and SHAREs to model complex variable interactions.
  • Semi-synthetic dataset inspired by survival analysis: Generated to show how SHAREs extend GAMs.
  • Torque equation dataset: Generated to show how SHAREs extend SR.
  • Temperature dataset: A simulated dataset based on the physics of heating water, designed to be challenging for both GAMs and SR.
  • Boston House Prices dataset and Concrete Compressive Strength dataset: Standard UCI datasets used to compare SHAREs against Linear Regression, GAMs, and XGBoost.

Key Insights and Ideas

  • Bridging the interpretability-expressivity gap: SHAREs offer a compelling way to model complex, non-linear relationships while retaining a degree of interpretability, which is a significant challenge in current machine learning.
  • Transparency beyond size: The paper proposes a novel definition of transparency based on structural rules (disjoint variables for binary ops, no nested shape functions) rather than just the number of terms or tree depth, which are often poor proxies for true comprehensibility.
  • Unifying framework: SHAREs can represent both GAMs (when no interactions are used) and potentially some closed-form expressions (if shape functions are chosen appropriately), acting as a unifying framework.
  • Visualizing complex interactions: By decomposing complex relationships into interpretable shape functions and their interactions, SHAREs allow for a more nuanced understanding than black-box models or overly complex symbolic expressions.

Key Results

  • Improved performance on complex interactions: SHAREs significantly outperform GAMs and GA2Ms on datasets with complex variable interactions, often achieving performance comparable to black-box models like XGBoost. For example, on equation I.8.14, SHAREs achieved an R² score of 0.992, compared to GAM (0.229) and GA2M (0.966).
  • Effective modeling of non-closed-form data: SHAREs successfully fit stress-strain curves with fewer, more interpretable expressions than SR.
  • Accurate physical modeling: In the temperature dataset experiment, SHAREs achieved near-perfect R² scores (0.999) and extracted physically meaningful parameters (like specific heat capacities and heats of fusion/vaporization) from the learned shape functions.
  • Competitive on standard benchmarks: On the Boston and Concrete datasets, SHAREs performed competitively with or surpassed other transparent methods like GAM-S and EBM-1, while maintaining superior interpretability.

Limitations and Future Work

  • Computational cost: The current implementation of SHAREs is time-intensive and does not scale well to larger datasets due to the nested optimization process.
  • Optimization challenges: Optimizing transparent models is generally harder than black-box models.
  • Future work: The authors express confidence that further optimizations will improve scalability and enable wider adoption. They hope future work will address the implementation limitations and enhance the ability to fit SHAREs to even larger and more complex datasets. They also suggest exploring different types of shape functions (e.g., splines) to enforce simplicity if neural networks prove too complex.

Summary updated

Abstract

Symbolic regression has excelled in uncovering equations from physics, chemistry, biology, and related disciplines. However, its effectiveness becomes less certain when applied to experimental data lacking inherent closed-form expressions. Empirically derived relationships, such as entire stress-strain curves, may defy concise closed-form representation, compelling us to explore more adaptive modeling approaches that balance flexibility with interpretability. In our pursuit, we turn to Generalized Additive Models (GAMs), a widely used class of models known for their versatility across various domains. Although GAMs can capture non-linear relationships between variables and targets, they cannot capture intricate feature interactions. In this work, we investigate both of these challenges and propose a novel class of models, Shape Arithmetic Expressions (SHAREs), that fuses GAM's flexible shape functions with the complex feature interactions found in mathematical expressions. SHAREs also provide a unifying framework for both of these approaches. We also design a set of rules for constructing SHAREs that guarantee transparency of the found expressions beyond the standard constraints based on the model's size.
BibTeX
@inproceedings{kacprzyk2023shape,
  author = {Krzysztof Kacprzyk and Mihaela van der Schaar},
  title = {{Shape Arithmetic Expressions}},
  booktitle = {AI4Science @ NeurIPS},
  year = {2023},
  url = {https://openreview.net/forum?id=BybTciafUJ},
}
Figure