Symbolic Brittleness in Sequence Models: On Systematic Generalization in Symbolic Mathematics

Sean Welleck, Peter West, Jize Cao, Yejin Choi

Abstract

Neural sequence-to-sequence models have demonstrated remarkable performance on symbolic integration tasks, often surpassing traditional rule-based solvers. However, this paper investigates whether such performance reflects a genuine understanding of mathematical rules or merely sophisticated pattern matching. By evaluating state-of-the-art models through the lenses of robustness, compositionality, and out-of-distribution generalization, we reveal significant "symbolic brittleness." Models fail on simple perturbations, struggle with basic linear properties like the sum rule, and collapse when faced with out-of-distribution constants. These findings suggest that current Transformer architectures lack the inductive biases necessary for systematic mathematical reasoning.

Introduction

In 2019, research demonstrated that standard sequence-to-sequence Transformers could outperform traditional symbolic solvers like Mathematica at symbolic integration. While this was a landmark result, it raised a fundamental question: Has the model actually learned the rules of calculus, or has it simply memorized a massive lookup table of patterns?

In mathematics, generalization is inherently systematic. A true understanding of calculus implies that if a model can integrate $f(x)$ and $g(x)$, it should inherently know how to integrate their sum, $f(x) + g(x)$. Furthermore, the specific value of a constant $k$ in $\int \cos(kx) dx$ should not affect the model's ability to apply the rule. This paper investigates the gap between high test-set accuracy and the lack of systematic generalization in symbolic domains.

Methodology

The authors evaluate a state-of-the-art neural sequence integrator using three primary metrics: robustness, compositionality, and out-of-distribution (OOD) generalization.

Conceptual framework for evaluating symbolic brittleness through robustness, compositionality, and out-of-distribution generalization.

To probe these boundaries, they introduce SAGGA (Symbolic Archive Generation with Genetic Algorithms), a gradient-free evolutionary algorithm designed to find adversarial mathematical problems. SAGGA takes a "seed" equation the model solves correctly and applies mutations—such as changing constants or swapping operators—to find the minimal change that causes a model failure. This approach allows for an active exploration of the model's knowledge boundaries beyond static test sets.

Analysis of Symbolic Brittleness

The Illusion of Mastery

A model can exhibit near-perfect accuracy on a test set while remaining remarkably brittle. For instance, a model might correctly integrate $26x^{42}$ but fail on $88x^{42}$. This suggests the model has not learned the general power rule ($\int x^n dx$) but has instead memorized specific instances or "templates" encountered during training.

The Compositionality Gap

One of the most striking findings is the failure of compositionality. While the model can integrate individual functions $f(x)$ and $g(x)$ perfectly, the failure rate for the sum $f(x) + g(x)$ can skyrocket to 85% for certain primitive types. The model fails to apply the most basic linear property of integration, indicating it does not treat the operation as a combination of independent rules.

Confidently Incorrect

When the model fails, it often assigns a high probability (exceeding 90%) to a complex, nonsensical string. The correct, simple answer is frequently found at the very bottom of the probability distribution, suggesting the model is biased toward fitting inputs into long, convoluted expressions seen during training rather than applying logical simplification.

Empirical Results

  • Robustness: Even within the training range of coefficients, simple primitives like $k_1 \sin(k_2 x)$ showed failure rates between 24% and 45%.
  • Compositionality: For functions solved individually with 0% failure, adding two terms led to failure rates of 29.2% for coefficients and 84.9% for exponents.
  • Extrapolation: As equation complexity (number of operator nodes) increased from the training limit of 15 to 35, the failure rate jumped from 1.6% to 59.2%.
  • Integer OOD: Performance on trigonometric functions collapsed to near 0% accuracy as coefficients moved from the training range (1-100) to larger values (10,000+).
Heatmap of failure rates (Fail@1) for various function types as coefficients move beyond the training distribution (1-100).

Conclusion

The study concludes that current neural sequence models are prisoners of their training data distribution. The Transformer architecture lacks the inductive biases required to enforce mathematical logic, such as linearity and symmetry. Future work must move beyond pure sequence-to-sequence learning toward architectures that can explicitly represent and reason with mathematical properties, ensuring that generalization is a product of logic rather than pattern matching.

Summary updated

Abstract

Neural sequence models trained with maximum likelihood estimation have led to breakthroughs in many tasks, where success is defined by the gap between training and test performance. However, their ability to achieve stronger forms of generalization remains unclear. We consider the problem of symbolic mathematical integration, as it requires generalizing systematically beyond the training set. We develop a methodology for evaluating generalization that takes advantage of the problem domain's structure and access to a verifier. Despite promising in-distribution performance of sequence-to-sequence models in this domain, we demonstrate challenges in achieving robustness, compositionality, and out-of-distribution generalization, through both carefully constructed manual test suites and a genetic algorithm that automatically finds large collections of failures in a controllable manner. Our investigation highlights the difficulty of generalizing well with the predominant modeling and learning approach, and the importance of evaluating beyond the test set, across different aspects of generalization.
BibTeX
@inproceedings{welleck2022symbolic,
  author = {Sean Welleck and Peter West and Jize Cao and Yejin Choi},
  title = {{Symbolic Brittleness in Sequence Models: On Systematic Generalization in Symbolic Mathematics}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i8.20841},
}
Figure