From Embeddings to Equations: Genetic-Programming Surrogates for Interpretable Transformer Classification

Mohammad Sadegh Khorshidi, Navid Yazdanjue, Hassan Gharoun, Mohammad Reza Nikoo, Fang Chen, Amir H. GandomiarXiv 2025 · DOI · arXiv

Introduction

Transformers have revolutionized artificial intelligence, achieving state-of-the-art performance across language, vision, and multimodal tasks. However, their internal mechanisms remain largely opaque, characterized by high-dimensional embedding spaces that defy direct human interpretation. The paper "From Embeddings to Equations" addresses this "interpretability gap" by proposing a novel pipeline that distills complex Transformer embeddings into compact, human-readable algebraic equations using Genetic Programming (GP).

Methodology

The proposed approach treats the Transformer as a "frozen" encoder, extracting high-dimensional output vectors without modifying the underlying model weights. To bridge the gap between these dense embeddings and symbolic logic, the authors introduce a multi-stage pipeline:

  1. Embedding Extraction: Utilizing state-of-the-art encoders such as ModernBERT (text), DINOv2 (images), and SigLIP (multimodal).
  2. Semantic-Preserving Feature Partitioning (SPFP): High-dimensional vectors (ranging from 768 to 1152 dimensions) are partitioned into disjoint "views" to manage the search space complexity for symbolic regression.
  3. Multi-population Ensemble GP (MEGP): An evolutionary engine where distinct populations evolve symbolic programs for different data views. The final classification is determined by the sum of these programs' logits.
  4. Canonical Model Selection: Employs a "1-standard-error (1-SE) rule" to identify the most parsimonious model that maintains performance parity with more complex alternatives.
  5. Calibration: Post-hoc Temperature Scaling is applied to align the model's confidence with its empirical accuracy.

Datasets and Experimental Setup

The researchers evaluated the MEGP pipeline across five diverse benchmarks spanning text, image, and multimodal classification. The table below details the embedding dimensions and feature partitioning used for each dataset, alongside a summary of performance metrics.

Dataset characteristics and performance overview across multiple metrics including AUC, Brier score, Complexity, ECE, F1 score, and Log-loss.

Key Insights

  • Antagonistic Decision Axes: Analysis of the evolved equations revealed that Transformer logic often relies on "antagonistic pairs" of dimensions. These pairs act as primary decision axes, where one dimension increases and another decreases the probability of a specific class.
  • Extreme Sparsity: The surrogate models demonstrate remarkable efficiency. For instance, the MNIST surrogate achieved near-perfect accuracy using only 1.7% of the available embedding dimensions, highlighting that much of the embedding space is redundant for specific classification tasks.
  • Modality Agnosticism: The pipeline's success across text, image, and multimodal data suggests that the geometric structure of latent spaces in modern Transformers shares fundamental properties that symbolic regression can effectively exploit.
  • The Role of Calibration: Symbolic models can exhibit overconfidence. The authors found that temperature scaling significantly reduced Expected Calibration Error (ECE) by 58% to 76%, enhancing the reliability of the model's probabilistic outputs.

Results and Discussion

The symbolic surrogates achieved performance levels comparable to the original black-box models while offering significantly higher interpretability and compactness.

Model complexity (node count) across the five benchmark datasets, illustrating the relative parsimony of the evolved symbolic equations.
  • Accuracy: The models reached $F_1$ scores of approximately 0.99 on MNIST and CIFAR-10, and 0.95 on SST2G. The 20 Newsgroups dataset proved more complex, with an $F_1$ score of roughly 0.81.
  • Compactness: The MNIST model was distilled into just 867 mathematical nodes, representing a massive reduction in complexity compared to the original neural network.
  • Reliability: Post-calibration ECE scores were consistently low, making these models suitable for applications where trust and uncertainty quantification are paramount.

Conclusion and Future Work

By converting "frozen" embeddings into symbolic equations, this research provides a path toward global, mechanistic interpretability of Transformer models. While the current equations operate in embedding space—requiring further work to ground specific dimensions in input tokens or pixels—the ability to distill a billion-parameter model into a handful of algebraic operations is a significant step forward. Future research may explore expanding the primitive set of mathematical operators or integrating input-level grounding directly into the evolutionary process.

Summary updated

BibTeX
@misc{khorshidi2025from,
  author = {Mohammad Sadegh Khorshidi and Navid Yazdanjue and Hassan Gharoun and Mohammad Reza Nikoo and Fang Chen and Amir H. Gandomi},
  title = {{From Embeddings to Equations: Genetic-Programming Surrogates for Interpretable Transformer Classification}},
  year = {2025},
  eprint = {2509.21341},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.21341},
}
Figure