Shape Arithmetic Expressions: Advancing Scientific Discovery Beyond Closed-Form Equations

Krzysztof Kacprzyk, Mihaela van der Schaar

Abstract

Symbolic regression has excelled in uncovering equations from physics, chemistry, biology, and related disciplines. However, its effectiveness becomes less certain when applied to experimental data lacking inherent closed-form expressions. Empirically derived relationships, such as entire stress-strain curves, may defy concise closed-form representation, compelling us to explore more adaptive modeling approaches that balance flexibility with interpretability. In our pursuit, we turn to Generalized Additive Models (GAMs), a widely used class of models known for their versatility across various domains. Although GAMs can capture non-linear relationships between variables and targets, they cannot capture intricate feature interactions. In this work, we investigate both of these challenges and propose a novel class of models, Shape Arithmetic Expressions (SHAREs), that fuses GAM’s flexible shape functions with the complex feature interactions found in mathematical expressions. SHAREs also provide a unifying framework for both of these approaches. We also design a set of rules for constructing SHAREs that guarantee transparency of the found expressions beyond the standard constraints based on the model’s size.

Introduction

In scientific discovery, the goal is often to find the "underlying equation" of a system. Traditionally, researchers rely on two primary tools: Symbolic Regression (SR) and Generalized Additive Models (GAMs). SR attempts to find closed-form expressions like $F=ma$. While effective for fundamental physics, it often fails on experimental data that lacks a neat formula, resulting in "bloated," unreadable equations that attempt to compensate for a lack of fit. Conversely, GAMs are flexible and easily visualized but struggle with complex feature interactions; they can model $f(x) + g(y)$ but fail at relationships like $f(x \times y)$.

This research identifies a "missing middle": models capable of handling complex variable interactions while remaining decomposable and visually interpretable, even when the ground truth is not a simple textbook equation.

Methodology: Shape Arithmetic Expressions (SHAREs)

The core innovation is the SHARE framework. Instead of constructing equations from basic operators and fixed functions (e.g., $\sin$, $\exp$), SHAREs utilize univariate shape functions ($s(x)$) implemented as small neural networks.

Structure of a Shape Arithmetic Expression (SHARE) tree.

To maintain model transparency, the authors propose two fundamental construction rules:

  1. Univariate Composition: A transparent function can be wrapped in a shape function while remaining transparent, as the resulting composition can still be plotted.
  2. Disjoint Binary Operations: Arithmetic operations (e.g., $+$, $\times$, $\div$) are permitted only between functions that utilize different sets of variables. This ensures the model can be decomposed into independent, understandable components.

The discovery process employs a "nested optimization" approach. An outer loop uses Genetic Programming to evolve the mathematical tree structure, while an inner loop utilizes Gradient Descent (Adam) to train the neural networks serving as shape functions within that structure.

Experimental Evaluation

The authors evaluated SHAREs across several challenging scenarios:

  • Feynman Symbolic Regression Database: 100 benchmark equations from physics.
  • Experimental Stress-Strain Curves: Real-world data for Aluminum 6061-T651, exhibiting non-linear elastic and plastic deformation.
  • Water Temperature Simulation: Modeling phase changes (latent heat) from ice to steam.
  • PMLB Classification: Testing general machine learning performance on real-world datasets like breast cancer and diabetes.

Key Insights and Results

A central argument of the paper is that transparency stems from decomposition. By breaking a complex system into parts that can be analyzed individually (e.g., via 1D plots), researchers can understand how components combine. Mathematically, "Transparent SHAREs" have a naturally bounded size based on the number of variables, eliminating the need for arbitrary "max depth" constraints.

Experimental stress-strain curves for Aluminum 6061-T651 at different temperatures.

Key performance metrics include:

  • Expressivity: SHAREs successfully modeled 86 out of 100 Feynman equations under strict transparency rules.
  • Performance: On complex interactions where GAMs failed (e.g., the Law of Cosines), SHAREs matched the accuracy of black-box models like XGBoost.
  • Scientific Insight: In the stress-strain example, SHAREs achieved an $R^2$ of 0.999 with just 3 shape functions. In contrast, standard SR required an 18-term equation to reach an $R^2$ of 0.970. In the water temperature experiment, SHAREs clearly identified plateaus corresponding to latent heat—physical properties that standard symbolic regression failed to capture concisely.
  • Robustness: The models maintained performance levels comparable to XGBoost even as data noise increased.

Conclusion and Future Work

While promising, the current SHARE implementation faces limitations. The "nested optimization" is computationally expensive, making it slow for massive or high-dimensional datasets. Additionally, neural network shape functions can sometimes produce "jagged" results; future work may explore splines or stronger regularization for smoother, more physically intuitive shapes. Despite these challenges, SHAREs provide a powerful new framework for scientific discovery where traditional closed-form equations fall short.

Summary updated

Abstract

Symbolic regression has excelled in uncovering equations from physics, chemistry, biology, and related disciplines. However, its effectiveness becomes less certain when applied to experimental data lacking inherent closed-form expressions. Empirically derived relationships, such as entire stress-strain curves, may defy concise closed-form representation, compelling us to explore more adaptive modeling approaches that balance flexibility with interpretability. In our pursuit, we turn to Generalized Additive Models (GAMs), a widely used class of models known for their versatility across various domains. Although GAMs can capture non-linear relationships between variables and targets, they cannot capture intricate feature interactions. In this work, we investigate both of these challenges and propose a novel class of models, Shape Arithmetic Expressions (SHAREs), that fuses GAM’s flexible shape functions with the complex feature interactions found in mathematical expressions. SHAREs also provide a unifying framework for both of these approaches. We also design a set of rules for constructing SHAREs that guarantee transparency of the found expressions beyond the standard constraints based on the model’s size.
BibTeX
@inproceedings{kacprzyk2024shape,
  author = {Krzysztof Kacprzyk and Mihaela van der Schaar},
  title = {{Shape Arithmetic Expressions: Advancing Scientific Discovery Beyond Closed-Form Equations}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2024},
  url = {https://openreview.net/forum?id=KMJOgVp14o},
}
Figure