Active learning improves performance on symbolic regression tasks in StackGP

Nathan Haut, Wolfgang Banzhaf, Bill Punch

TL;DR

This paper introduces an active learning strategy for symbolic regression that significantly reduces the amount of data needed to discover mathematical models. By intelligently selecting the most informative data points to query, their StackGP system achieves comparable or better performance than existing methods, even without domain expertise.

What Problem Does This Paper Address?

Symbolic regression aims to discover mathematical equations that best fit a given dataset. This is a powerful tool for creating interpretable AI models, especially in scientific discovery where understanding the underlying relationships is crucial. However, a major bottleneck is the sheer amount of data required to train these models effectively. Existing methods, like AIFeynman, often rely on domain expertise, complex pre-processing (e.g., dimensional analysis), or large datasets, limiting their applicability in real-world scenarios where data is scarce or units are unknown. This paper tackles the challenge of making symbolic regression more data-efficient and accessible by employing active learning.

Methods

The core of this work is an active learning strategy integrated with a StackGP symbolic regression system. Here's how it works:

  1. Initialization: The process starts with a very small, randomly generated dataset (3 points).
  2. Model Evolution: StackGP evolves a population of candidate symbolic regression models using genetic programming techniques (mutation, crossover, cloning). The fitness of these models is evaluated based on correlation with the data and model complexity.
  3. Ensemble Selection: A diverse ensemble of the best-performing models is selected.
  4. Uncertainty Maximization: The key innovation is how new data points are chosen. The system identifies points where the ensemble of models exhibits the highest prediction uncertainty. This uncertainty is quantified using the standard deviation of ensemble responses, normalized by a trimmed mean to avoid outliers.
  5. Data Augmentation & Iteration: The selected "most informative" data point is added to the training set, and the StackGP system re-evolves its models with the expanded dataset. This cycle repeats until a perfect model is found or a termination criterion is met.
  6. Handling Redundant Points: If the uncertainty maximization suggests a point already in the training set, the system searches a subspace to find a new, informative point.

The StackGP system itself uses a stack-based representation for programs and employs Pareto tournament selection for model selection.

Datasets

The primary dataset used is the Feynman Symbolic Regression Database, consisting of 100 physics equations. This benchmark is notable because the equations are physically meaningful, and previous work has established baseline performance metrics (e.g., data points needed) for comparison. The authors generated synthetic data for these equations.

Key Insights and Ideas

  • Active Learning for Data Efficiency: The most significant insight is that actively selecting data points based on model uncertainty dramatically reduces the data requirements for symbolic regression. This makes the technique viable in data-scarce environments.
  • Domain Expertise Independence: The proposed method achieves good results without relying on domain-specific knowledge like dimensional analysis, making it more generalizable.
  • Ensemble-Based Uncertainty: Using an ensemble of models to gauge uncertainty is more robust than relying on a single model, as it captures a broader range of potential model behaviors and disagreements.
  • Potential for Scientific Discovery: The ability to guide data collection intelligently opens doors for designing experiments and accelerating scientific research by focusing on the most critical data points.

Key Results

  • High Success Rate: StackGP with active learning successfully rediscovered 72 out of 100 Feynman equations.
  • Outperforming AIFeynman: For 37 equations, the system solved them with just the initial 3 data points, significantly outperforming AIFeynman's minimum of 10 points. For another 16 equations, it required fewer data points than AIFeynman.
  • Comparable to Eureqa: The system achieved a success rate (72/100) comparable to Eureqa (71/100), a commercial symbolic regression tool, while using no domain expertise.
  • Data Reduction: For some equations, like equation 3, StackGP needed only 42.5 data points on average, compared to 1000 for AIFeynman.

Limitations and Future Work

  • Problematic Equations: The paper acknowledges that 28 problems were not solved within the given iteration limit. Analysis suggests that equations with complex denominators might pose challenges, potentially due to the relative nature of the uncertainty metric or inherent difficulties in symbolic regression for such structures.
  • Redundant Point Selection: While a subspace search was introduced to handle already selected points, the authors suggest that further work is needed to ensure that new points are sufficiently distinct from existing ones.
  • Uncertainty Metric: The authors propose exploring alternative uncertainty metrics, as the current relative metric might be influenced by ensemble response magnitudes, potentially biasing selection.
  • Hybrid Approaches: Future work could involve hybrid strategies combining random and informed point selection.
  • Broader Application: The authors plan to explore other active learning techniques and classify problem types that are particularly well-suited or challenging for active learning in symbolic regression.

Summary updated

Abstract

In this paper we introduce an active learning method for symbolic regression using StackGP. The approach begins with a small number of data points for StackGP to model. To improve the model the system incrementally adds a data point such that the new point maximizes prediction uncertainty as measured by the model ensemble. Symbolic regression is re-run with the larger data set. This cycle continues until the system satisfies a termination criterion. We use the Feynman AI benchmark set of equations to examine the ability of our method to find appropriate models using fewer data points. The approach was found to successfully rediscover 72 of the 100 Feynman equations using as few data points as possible, and without use of domain expertise or data translation.
BibTeX
@inproceedings{haut2022active,
  author = {Nathan Haut and Wolfgang Banzhaf and Bill Punch},
  title = {{Active learning improves performance on symbolic regression tasks in StackGP}},
  booktitle = {GECCO Companion},
  year = {2022},
  url = {https://openreview.net/forum?id=ar4MEcVyEF},
}
Figure