Abstract
Symbolic Regression (SR) is a powerful tool for discovering mathematical relationships in data, but modern Transformer-based models often require massive datasets to generalize. This paper investigates the application of K-Fold Cross-Validation (KFCV) to the SymbolicGPT architecture to improve performance on small datasets. We demonstrate that using 5-fold KFCV on a dataset of 15,000 indices—a 97% reduction from standard training sizes—results in a 53.31% relative improvement in validation loss. Remarkably, this approach allows the model to achieve performance parity with a baseline trained on 500,000 indices, suggesting that KFCV is a robust strategy for data-efficient symbolic regression.
Introduction
Symbolic Regression (SR) aims to find the underlying mathematical equation that describes a set of data points, offering greater interpretability than traditional black-box models. While Transformer-based models like SymbolicGPT have significantly improved the state-of-the-art in SR, they are notoriously data-hungry. In many scientific fields, obtaining hundreds of thousands of data points is impractical. When trained on small datasets, these complex models frequently overfit to noise rather than learning the true mathematical structure. This study evaluates whether K-Fold Cross-Validation (KFCV) can serve as a generalization engine to bridge the performance gap between small and large datasets.
Methodology
The researchers employed SymbolicGPT, a three-stage Transformer architecture. The model first uses a "T-net" to create embeddings of input data, then a GPT-based language model generates a mathematical "skeleton" (the equation structure), and finally, a constant optimizer fills in the numerical values.
The core intervention was the implementation of 5-Fold Cross-Validation (KFCV). Unlike a standard 80/20 train-test split, KFCV partitions the data into five subsets. The model is trained five times; in each iteration, a different fold is held out for validation while the other four are used for training. The final performance is calculated as the average across all five iterations. This strategy forces the Transformer to learn more generalized patterns by exposing it to different subsets of the limited data.
Experimental Setup
The study utilized a synthetic dataset for 1-variable symbolic regression. Two primary configurations were compared:
- Baseline: A large-scale dataset consisting of 500,000 indices.
- Experimental: A significantly reduced dataset of 15,000 indices, representing a 97% reduction in data volume.
The primary evaluation metric was the validation loss, comparing the KFCV-trained small model against a standard-split small model and the large-scale baseline.
Results
The application of KFCV yielded a 53.31% relative improvement in validation loss compared to the standard split on the same small dataset. Most notably, the performance of the model trained on 15,000 points with KFCV was nearly identical to the baseline model trained on 500,000 points.
Quantitative analysis showed that the average validation loss across the five folds was 0.27858, a significant reduction from the 0.59664 observed in the non-KFCV small-data trial. This suggests that the "data floor" for effective Transformer training in SR is much lower than previously assumed when robust training strategies are employed.
Discussion
The results indicate that KFCV acts as a powerful generalization engine. By preventing the "memorization" trap common in small-sample deep learning, it allows the model to identify the correct mathematical structure more reliably. The authors also noted improved consistency in the model's outputs across different trials.
However, several limitations warrant further investigation. The study was restricted to 1-variable equations; the effectiveness of KFCV in multi-variable search spaces, where complexity scales exponentially, remains unknown. Additionally, the computational overhead of training multiple folds must be balanced against the time saved by using smaller datasets.
Conclusion
This research demonstrates that K-Fold Cross-Validation can drastically improve the data efficiency of Transformer-based symbolic regression models. By achieving "big data" results with a 97% smaller footprint, this approach provides a viable path for applying advanced SR techniques in resource-constrained environments.
Summary updated