ViSymRe: Vision-guided Multimodal Symbolic Regression

Da Li, Junping Yin, Jin Xu, Xinxin Li, Juan Zhang

TL;DR

This paper introduces ViSymRe, a novel framework for Symbolic Regression (SR) that leverages visual information to improve the accuracy and interpretability of discovered equations. By transforming complex equations into visual representations and using a "virtual vision" approach, ViSymRe can perform inference even without direct visual input, bridging the modality gap between data and equations.

What Problem Does This Paper Address?

Symbolic Regression (SR) aims to discover simple, interpretable mathematical equations that explain observed data. While traditional methods like Genetic Programming (GP) are robust but slow, recent Transformer-based models offer faster inference by learning direct mappings from data to equations. However, these models struggle with the inherent modality difference between numerical data and symbolic equations, leading to convergence issues. Existing multimodal approaches often require paired data, which isn't always available during inference. This paper tackles the challenge of effectively integrating visual information into SR to improve performance and address the "modality gap" without requiring visual inputs during inference.

Methods

ViSymRe introduces several key innovations:

  • Multi-View Random Slicing (MVRS): This technique projects multivariate equations into 2D visual representations. By using random affine transformations and controllable sampling points, MVRS avoids common visualization pitfalls like variable degradation and exponential sampling complexity, making high-dimensional equations visually tractable at low computational cost.
  • Dual-Visual Pipeline: ViSymRe employs two visual pipelines: a "real" pipeline for training (using actual visual representations) and a "virtual" pipeline for inference. The virtual pipeline uses a Visual Decoder to generate "virtual vision" from dataset embeddings, enabling dataset-only inference.
  • Biased Cross-Attention: To handle noise in the virtual vision, a Biased Cross-Attention module is introduced. This module selectively suppresses attention weights on noisy components, improving the robustness of multimodal fusion.
  • Syntax Constraint Algorithm: To ensure generated equations are physically plausible and interpretable, a syntax constraint algorithm is integrated into the decoding process, pruning invalid structures.

Datasets

The paper uses several datasets for training and evaluation:

  • Training Data: Generated from diverse equation skeletons and sampled constants, covering a wide range of numerical scales.
  • Evaluation Benchmarks:
    • Low-dimensional benchmarks: Including "Nguyen," "Keijzer," "Korns," "Livermore," "Jin," and "Neat."
    • SRBench 1.0: Covering "Feynman" and "ODE-Strogatz" benchmarks.
  • SRSD-Feynman benchmark: With Easy, Medium, and Hard difficulty levels.
  • SRBench 2.0: Including black-box and phenomenological & first-principles problems.

Key Insights and Ideas

  • Visualizing Equations is Powerful: The core idea is that transforming equations into visual representations can provide a richer signal for SR models, helping to bridge the gap between numerical data and symbolic structures.
  • "Virtual Vision" Solves the Inference Problem: The ability to generate "virtual vision" from data alone is crucial. This allows the model to leverage visual priors learned during training without needing actual visual inputs at inference time, a significant practical advantage.
  • Noise Suppression is Key for Multimodal Fusion: The Biased Cross-Attention mechanism highlights that simply fusing multimodal information isn't enough; actively mitigating noise in one modality (like the virtual vision) is essential for robust performance.
  • Simplicity Matters: The paper emphasizes that good SR models should not only fit the data but also produce simple, interpretable equations. The syntax constraint algorithm and evaluation metrics like Complexity directly address this.

Key Results

  • ViSymRe achieves competitive performance across various SR benchmarks, often outperforming state-of-the-art dataset-only baselines.
  • It demonstrates strong performance in low-complexity and rapid-inference scenarios.
  • The model shows robust performance under varying noise levels, maintaining high R² scores and Symbolic Solution Rates where other models degrade.
  • ViSymRe generates equations with significantly lower complexity compared to many baselines, indicating better interpretability.
  • The framework exhibits superior extrapolation capabilities, especially when dealing with datasets of varying magnitudes, thanks to the proposed Adaptive Magnitude Scaling (AMS) algorithm.

Limitations and Future Work

  • Out-of-Distribution Generalization: While AMS helps, ViSymRe's generalization capability on datasets significantly different from the training distribution remains a challenge.
  • Reliance on Analytical Equations for MVRS: MVRS, while efficient, still requires analytical equations for visualization during training. The paper suggests this is a trade-off for feasibility in large-scale pre-training.
  • Future Work: The authors suggest that the Biased Cross-Attention mechanism could be a valuable component for other multimodal learning tasks, particularly in noisy environments. The dual-visual pipeline approach also offers a promising direction for transferring knowledge from data-rich training to resource-constrained inference settings.

Summary updated

Abstract

Extracting simple mathematical expression from an observational dataset to describe complex natural phenomena is one of the core objectives of artificial intelligence (AI). This field is known as symbolic regression (SR). Traditional SR models are based on genetic programming (GP) or reinforcement learning (RL), facing well-known challenges, such as low efficiency and overfitting. Recent studies have integrated SR with large language models (LLMs), enabling fast zero-shot inference by learning mappings from millions of dataset-expression pairs. However, since the input and output are inherently different modalities, such models often struggle to converge effectively. In this paper, we introduce ViSymRe, a vision-guided multimodal SR model that incorporates the third resource, expression graph, to bridge the modality gap. Different from traditional multimodal models, ViSymRe is trained to extract vision, termed virtual vision, from datasets, without relying on the global availability of expression graphs, which addresses the essential challenge of visual SR, i.e., expression graphs are not available during inference. Evaluation results on multiple mainstream benchmarks show that ViSymRe achieves more competitive performance than the state-of-the-art dataset-only baselines. The expressions predicted by ViSymRe not only fit the dataset well but are also simple and structurally accurate, goals that SR models strive to achieve.
BibTeX
@misc{li2024visymre,
  author = {Da Li and Junping Yin and Jin Xu and Xinxin Li and Juan Zhang},
  title = {{ViSymRe: Vision-guided Multimodal Symbolic Regression}},
  year = {2024},
  eprint = {2412.11139},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.11139},
}
Figure