Shape Arithmetic Expressions: Advancing Scientific Discovery Beyond Closed-Form Equations

Krzysztof Kacprzyk, Mihaela van der Schaar

TL;DR

This paper introduces Shape Arithmetic Expressions (SHAREs), a novel modeling framework that combines the interpretability of Generalized Additive Models (GAMs) with the expressive power of symbolic regression. SHAREs aim to discover scientific equations that are both accurate and understandable, even when dealing with complex, non-linear relationships that defy traditional closed-form solutions.

What Problem Does This Paper Address?

Scientists have long relied on mathematical equations to describe and understand the natural world. Symbolic Regression (SR) has been a powerful tool for automatically discovering these equations from data. However, SR struggles when the underlying relationships are not easily expressible as concise, closed-form equations, often resulting in overly complex or uninterpretable expressions. On the other hand, Generalized Additive Models (GAMs) offer interpretability by modeling relationships as sums of univariate functions but fail to capture complex interactions between variables. This paper addresses the gap between the need for interpretable models and the ability to discover complex scientific relationships, particularly when dealing with experimental data that lacks inherent closed-form expressions.

Methods

The core contribution is the introduction of Shape Arithmetic Expressions (SHAREs). SHAREs are mathematical expressions that combine:

  • Flexible Shape Functions: These are univariate functions (modeled as neural networks) that can capture complex non-linear relationships for individual variables. They are designed to be interpretable through visualization.
  • Arithmetic Operations: Standard binary operations (+, -, ×, ÷) are used to combine these shape functions.
  • Structure Rules: Crucially, SHAREs adhere to specific rules (Rule 1: Univariate composition, Rule 2: Disjoint binary operations) that ensure transparency. These rules prevent recursive compositions of shape functions and require that binary operations combine functions with disjoint sets of variables. This is a key departure from standard SR, which often prioritizes complexity over interpretability.

The implementation uses a nested optimization approach:

  • Outer Loop (Genetic Programming): A modified genetic programming algorithm searches for the structure of the SHARE (the expression tree).
  • Inner Loop (Gradient Descent): For each candidate expression structure, the shape functions (modeled as neural networks) are optimized using gradient descent.

This approach aims to balance the discovery of the equation's structure with the fitting of its components.

Datasets

The paper uses several datasets for evaluation:

  • Stress-Strain Curves: Experimental data for aluminum at different temperatures was used to demonstrate SR's struggle with non-closed-form relationships.
  • Feynman Symbolic Regression Database: A collection of 100 equations was used to evaluate the performance of GAMs, GA2Ms, and SHAREs in capturing complex interactions.
  • Temperature Dataset: A semi-synthetic dataset was generated to simulate the temperature of water based on physics laws, demonstrating SHAREs' ability to extract quantitative insights.
  • PMLB Classification Datasets: Several small classification datasets were used for additional experiments to compare SHAREs with other methods.

Key Insights and Ideas

  • Bridging the Gap: SHAREs effectively bridge the gap between interpretable models like GAMs and powerful but often opaque models like those from SR. They offer a way to discover complex relationships while retaining interpretability.
  • Transparency through Structure: The defined rules for SHAREs (disjoint variable sets for binary operations) are crucial for ensuring transparency. This is a more principled approach to interpretability than simply limiting expression size.
  • Quantitative Insights from Shape Functions: The shape functions in SHAREs are not just fitting curves; they can reveal physical properties. For example, in the temperature dataset experiment, the slopes and plateaus of the shape functions corresponded to specific heat capacities and heats of fusion/vaporization.
  • SR's Limitation with Non-Closed Forms: The paper clearly demonstrates that standard SR, while good for closed-form equations, produces overly complex and less interpretable results for empirical data that doesn't fit neat mathematical forms.
  • GAMs' Interaction Limitation: GAMs, while interpretable, are shown to struggle significantly with capturing interactions between variables, leading to lower performance on datasets with complex dependencies.

Key Results

  • Improved Performance: SHAREs consistently achieved high R² scores, often outperforming GAMs and sometimes matching or exceeding SR on complex datasets. For instance, on the temperature dataset, SHAREs achieved R² scores significantly higher than GAMs and SR.
  • Interpretability: SHAREs provided interpretable shape functions that allowed for the extraction of physical insights, a capability lacking in standard SR and reduced in GA2Ms.
  • Rule-Based Transparency: The paper demonstrates that 86% of equations from the Feynman Symbolic Regression database satisfy Rule 2, suggesting that the transparency constraints imposed by SHAREs do not severely limit their expressiveness for many real-world scientific phenomena.
  • Robustness to Noise: SHAREs showed comparable robustness to noise as XGBoost on the temperature dataset.

Limitations and Future Work

  • Computational Cost: The current implementation of SHAREs is time-intensive due to nested optimization (genetic programming for structure, neural networks for shape functions). This limits scalability to very large datasets.
  • Optimization Bottleneck: The necessity of fitting a separate neural network for every equation is a major bottleneck.
  • Future Work: The authors hope future work will address the computational limitations through more efficient optimization algorithms. They also suggest exploring ways to explicitly incorporate information about units into the model and developing methods for selecting the "best" transparent SHARE from a Pareto frontier. Further research into "nicer" shape functions and more systematic notions of transparency is also proposed.

Summary updated

BibTeX
@inproceedings{kacprzyk2024shape,
  author = {Krzysztof Kacprzyk and Mihaela van der Schaar},
  title = {{Shape Arithmetic Expressions: Advancing Scientific Discovery Beyond Closed-Form Equations}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2024},
  url = {https://openreview.net/forum?id=KMJOgVp14o},
}
Figure