Abstract
Using transformers over large generated datasets, we train models to learn mathematical properties of differential systems, such as local stability, behavior at infinity and controllability. We achieve near perfect prediction of qualitative characteristics, and good approximations of numerical features of the system. This demonstrates that neural networks can learn to perform complex computations, grounded in advanced theory, from examples, without built-in mathematical knowledge.
Introduction
In classical mathematics, determining the properties of a differential system—such as its stability or controllability—requires the application of rigorous, theory-driven rules. A mathematician typically calculates a Jacobian matrix, derives its eigenvalues, and analyzes their real parts. While symbolic engines like Mathematica or MATLAB are precise, they are often computationally expensive and struggle with the scaling demands of high-dimensional or non-linear systems.
This research explores whether neural networks can learn to "shortcut" this advanced mathematical theory. By treating mathematical expressions as a translation task, we investigate if a Transformer model can learn to map the symbolic structure of a complex differential equation directly to its properties, effectively "feeling" the solution based on patterns learned from millions of examples.
Methodology
The study utilizes a standard Transformer architecture with an encoder-decoder setup. Mathematical functions are represented as trees and flattened into sequences using prefix notation (e.g., 2 + x becomes + 2 x) to eliminate the ambiguity of parentheses.
To train these models, a massive synthetic dataset of over 50 million examples per task was generated using symbolic math libraries like SymPy and SciPy. The tasks included:
- Local Stability: Systems of 2 to 6 equations.
- Controllability: Both autonomous and non-autonomous systems.
- Partial Differential Equations (PDEs): Linear PDEs in 2 to 6 dimensions.
Generalization and Extrapolation
A critical finding of this work is the model's ability to generalize beyond its training distribution. The models were tested on out-of-distribution (OOD) sets containing longer expressions and higher-degree equations than those encountered during training.
As shown in the data, models trained on lower-degree systems maintain significant accuracy when extrapolated to higher-degree systems. For instance, a model trained on systems of degrees 2 to 5 can still predict the stability of degree 6 systems with high accuracy (up to 77.3% for the largest model), suggesting that the network is learning underlying mathematical properties rather than simply memorizing templates.
Experimental Results
The Transformer models achieved near-perfect accuracy on qualitative tasks. In stability prediction, the model reached an overall accuracy of 96.4%, significantly outperforming baseline models like FastText.
Key performance metrics across other tasks include:
- Controllability: 97.4% accuracy for autonomous systems and 99.7% for non-autonomous systems.
- PDE Behavior: 98.4% accuracy in predicting the existence and behavior of solutions at infinity.
- Computational Speed: The Transformer is 10 to 100 times faster than traditional libraries. For example, predicting a control feedback matrix took 0.002 seconds, compared to 0.4 seconds for standard symbolic solvers.
While qualitative "Yes/No" classification is highly successful, quantitative regression (e.g., exact convergence speed) remains more challenging, though the models still provide respectable approximations within a 10% error margin.
Discussion and Limitations
Despite their performance, these models function largely as "black boxes." Analysis of the Transformer's attention heads revealed nearly uniform weight distributions, making it difficult to interpret the decision-making process. Furthermore, while these models excel at rapid approximation, they do not yet match the infinite precision of symbolic solvers for purely numerical tasks.
These findings suggest that deep learning models could serve as powerful "fast filters," providing near-instant approximations for scientists before they commit to more computationally intensive symbolic computations.
Summary updated