Evaluating K-Fold Cross Validation for Transformer Based Symbolic Regression Models

Kaustubh Kislay, Shlok Singh, Soham Joshi, Rohan Dutta, Jay Shim George Flint, Kevin ZhuarXiv 2024 · DOI · arXiv

Abstract

Symbolic Regression (SR) is a powerful tool for discovering mathematical relationships in data, but modern Transformer-based models often require massive datasets to generalize. This paper investigates the application of K-Fold Cross-Validation (KFCV) to the SymbolicGPT architecture to improve performance on small datasets. We demonstrate that using 5-fold KFCV on a dataset of 15,000 indices—a 97% reduction from standard training sizes—results in a 53.31% relative improvement in validation loss. Remarkably, this approach allows the model to achieve performance parity with a baseline trained on 500,000 indices, suggesting that KFCV is a robust strategy for data-efficient symbolic regression.

Introduction

Symbolic Regression (SR) aims to find the underlying mathematical equation that describes a set of data points, offering greater interpretability than traditional black-box models. While Transformer-based models like SymbolicGPT have significantly improved the state-of-the-art in SR, they are notoriously data-hungry. In many scientific fields, obtaining hundreds of thousands of data points is impractical. When trained on small datasets, these complex models frequently overfit to noise rather than learning the true mathematical structure. This study evaluates whether K-Fold Cross-Validation (KFCV) can serve as a generalization engine to bridge the performance gap between small and large datasets.

Methodology

The researchers employed SymbolicGPT, a three-stage Transformer architecture. The model first uses a "T-net" to create embeddings of input data, then a GPT-based language model generates a mathematical "skeleton" (the equation structure), and finally, a constant optimizer fills in the numerical values.

The core intervention was the implementation of 5-Fold Cross-Validation (KFCV). Unlike a standard 80/20 train-test split, KFCV partitions the data into five subsets. The model is trained five times; in each iteration, a different fold is held out for validation while the other four are used for training. The final performance is calculated as the average across all five iterations. This strategy forces the Transformer to learn more generalized patterns by exposing it to different subsets of the limited data.

The K-Fold Cross-Validation process used for parameter optimization and final evaluation.

Experimental Setup

The study utilized a synthetic dataset for 1-variable symbolic regression. Two primary configurations were compared:

  • Baseline: A large-scale dataset consisting of 500,000 indices.
  • Experimental: A significantly reduced dataset of 15,000 indices, representing a 97% reduction in data volume.

The primary evaluation metric was the validation loss, comparing the KFCV-trained small model against a standard-split small model and the large-scale baseline.

Results

The application of KFCV yielded a 53.31% relative improvement in validation loss compared to the standard split on the same small dataset. Most notably, the performance of the model trained on 15,000 points with KFCV was nearly identical to the baseline model trained on 500,000 points.

Quantitative analysis showed that the average validation loss across the five folds was 0.27858, a significant reduction from the 0.59664 observed in the non-KFCV small-data trial. This suggests that the "data floor" for effective Transformer training in SR is much lower than previously assumed when robust training strategies are employed.

Quantitative results showing training and validation losses for each of the five folds and the overall average.

Discussion

The results indicate that KFCV acts as a powerful generalization engine. By preventing the "memorization" trap common in small-sample deep learning, it allows the model to identify the correct mathematical structure more reliably. The authors also noted improved consistency in the model's outputs across different trials.

However, several limitations warrant further investigation. The study was restricted to 1-variable equations; the effectiveness of KFCV in multi-variable search spaces, where complexity scales exponentially, remains unknown. Additionally, the computational overhead of training multiple folds must be balanced against the time saved by using smaller datasets.

Conclusion

This research demonstrates that K-Fold Cross-Validation can drastically improve the data efficiency of Transformer-based symbolic regression models. By achieving "big data" results with a 97% smaller footprint, this approach provides a viable path for applying advanced SR techniques in resource-constrained environments.

Summary updated

Abstract

Symbolic Regression remains an NP-Hard problem, with extensive research focusing on AI models for this task. Transformer models have shown promise in Symbolic Regression, but performance suffers with smaller datasets. We propose applying k-fold cross-validation to a transformer-based symbolic regression model trained on a significantly reduced dataset (15,000 data points, down from 500,000). This technique partitions the training data into multiple subsets (folds), iteratively training on some while validating on others. Our aim is to provide an estimate of model generalization and mitigate overfitting issues associated with smaller datasets. Results show that this process improves the model's output consistency and generalization by a relative improvement in validation loss of 53.31%. Potentially enabling more efficient and accessible symbolic regression in resource-constrained environments.
BibTeX
@misc{kislay2024evaluating,
  author = {Kaustubh Kislay and Shlok Singh and Soham Joshi and Rohan Dutta and Jay Shim George Flint and Kevin Zhu},
  title = {{Evaluating K-Fold Cross Validation for Transformer Based Symbolic Regression Models}},
  year = {2024},
  eprint = {2410.21896},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.21896},
}
Figure