Learning advanced mathematical computations from examples

François Charton, Amaury Hayat, Guillaume Lample

Executive Summary

Researchers from Facebook AI and Ecole des Ponts Paristech have demonstrated that Transformer models can learn to solve advanced mathematical problems—like predicting the stability of differential equations and designing control systems—simply by observing examples. Without any built-in knowledge of calculus or linear algebra, these models achieve near-perfect accuracy on qualitative tasks and provide significant speedups over classical symbolic solvers.

The Problem: Bridging Symbolic and Numerical Math

In the world of mathematics, there are two main ways to solve a problem: symbolic computation (applying rigid, theory-derived rules like differentiation) and numerical approximation (crunching numbers to find a "good enough" answer). Traditionally, deep learning has struggled with the former; while neural networks excel at pattern recognition, they often stumble on basic arithmetic because they lack a "mental model" of mathematical rules.

The authors tackle a significant gap: Can a neural network learn to perform "advanced" mathematics—tasks that usually require a deep understanding of stability theory, control theory, and Partial Differential Equations (PDEs)—without being programmed with a single mathematical rule? They focus on properties like local stability (will a system return to equilibrium after a nudge?) and controllability (can we steer a system to a desired state?), which are foundational to engineering and physics but computationally expensive to solve.

Methodology: Math as a Translation Task

The researchers treated mathematics as a translation problem. They used a standard Transformer architecture (the same technology behind large language models) with an encoder-decoder setup.

The "language" of math was handled through two clever representations:

  1. Prefix Notation: Mathematical expressions were converted into trees and then into sequences (e.g., $2 + x$ becomes [+, 2, x]). This avoids the ambiguity of parentheses.
  2. Tokenized Reals: Floating-point numbers were represented as sequences of tokens (sign, mantissa, and exponent), allowing the model to process numerical values with high precision.

The model was trained to map a problem description (a system of equations) directly to a solution (a classification label or a numerical value) without intermediate steps like calculating Jacobians or eigenvalues.

Massive Synthetic Datasets

The study relies on massive, high-quality synthetic datasets. For each of the three main problem areas, the authors generated over 50 million examples:

  • Local Stability: Systems of 2 to 6 equations.
  • Control Theory: Autonomous and non-autonomous systems, including tasks to predict the "feedback matrix" needed to stabilize a system.
  • PDEs: Linear PDEs where the model predicts if a solution exists and its behavior at infinity.

These datasets were carefully balanced using rejection sampling to ensure the model couldn't "cheat" by simply guessing the most frequent class (e.g., always guessing a system is unstable).

Key Insights and Discoveries

  • The "Shortcut" Discovery: The most striking finding is that the model doesn't seem to be "simulating" the classical algorithms (like computing a Jacobian and then its eigenvalues). Instead, it finds mathematical "shortcuts" or patterns in the symbolic structure of the equations that humans haven't formally described yet.
  • Creative Problem Solving: In the control theory task, the model was asked to predict a feedback matrix $K$. Remarkably, the model often predicted a different valid matrix than the one in the training set. This proves the model isn't just memorizing; it has learned the underlying "physics" of the problem well enough to find alternative solutions.
  • Out-of-Distribution Generalization: The model showed surprising resilience. When tested on equations longer than those in the training set, or systems with more variables (e.g., training on 5 equations and testing on 6), the accuracy remained high. This suggests it learned a generalized property of differential systems rather than just specific templates.

Key Results

The Transformer achieved 96.4% accuracy in predicting system stability and 97.4% for controllability. This far outperformed a "bag of words" baseline (fastText), which only hit ~60-70%.

Accuracy of stability prediction across system degrees compared to FastText

For quantitative tasks, like predicting the speed of convergence, the model was accurate within a 10% margin in 86.6% of cases.

Accuracy of quantitative predictions (within 10% margin) across different model architectures

Furthermore, the Transformer proved to be highly efficient. It solved stability problems in 0.0008 seconds, compared to 0.02 seconds for optimized mathematical libraries—a 25x speedup. For complex control matrices, the speedup was even more dramatic (0.002s vs 0.4s).

Limitations and Future Work

Despite the impressive results, the authors note several hurdles:

  • The "Black Box" Problem: While the model is accurate, we don't know how it's making these predictions. Analysis of the attention heads showed a very uniform distribution, making it difficult to extract the "rules" the model discovered.
  • Numerical Limits: While the model is great at qualitative "yes/no" questions, its precision on purely numerical regression (like exact eigenvalue values) still lags behind dedicated numerical solvers.
  • Scope: The study is limited to specific classes of differential systems and linear PDEs. Extending this to highly non-linear, chaotic systems remains a frontier for future research.

In conclusion, this paper suggests that "mathematical intuition"—the ability to look at a complex system and sense its behavior—might be a pattern-recognition task that neural networks are uniquely suited to master.

Summary updated

Abstract

Using transformers over large generated datasets, we train models to learn mathematical properties of differential systems, such as local stability, behavior at infinity and controllability. We achieve near perfect prediction of qualitative characteristics, and good approximations of numerical features of the system. This demonstrates that neural networks can learn to perform complex computations, grounded in advanced theory, from examples, without built-in mathematical knowledge.
BibTeX
@inproceedings{charton2021learning,
  author = {François Charton and Amaury Hayat and Guillaume Lample},
  title = {{Learning advanced mathematical computations from examples}},
  booktitle = {International Conference on Learning Representations},
  year = {2021},
  url = {https://openreview.net/forum?id=hbeNX932HhG},
}
Figure