Int2Int: a framework for mathematics with transformers

François ChartonarXiv 2025 · DOI · arXiv

Abstract

While deep learning has transformed natural language processing, its application to pure mathematics remains hindered by a high technical barrier to entry. This paper introduces Int2Int, an open-source PyTorch framework designed to bridge this gap. By framing mathematical problems as sequence-to-sequence translation tasks, Int2Int provides a modular pipeline for researchers to apply transformer models to complex mathematical domains. We demonstrate the framework's efficacy through two primary tasks: computing greatest common divisors (GCD) and predicting the rank of elliptic curves. Our results show that with appropriate numerical representation, transformers can achieve high accuracy in specialized mathematical tasks, such as identifying rank 1 elliptic curves with 88.7% precision.

1. Introduction

In recent years, mathematicians have increasingly recognized the potential of artificial intelligence to assist in discovery, from identifying patterns in number theory to performing symbolic integration. However, the "cost of entry" for applying deep learning to mathematics is significant. Researchers must often build custom tokenizers for mathematical notation, manage complex GPU training loops, and format non-textual data into sequences suitable for attention mechanisms.

Int2Int aims to flatten this learning curve. It provides a standardized, modular codebase specifically tailored for "integer-to-integer" problems. By treating mathematical logic as a language, the framework allows researchers to focus on the underlying mathematics rather than the technical plumbing of deep learning implementation.

2. Framework Architecture and Methodology

Int2Int frames mathematical problems as sequence-to-sequence (seq2seq) translation tasks. In this paradigm, a mathematical query (e.g., "12 + 23") is treated as a source sentence, and the solution ("35") is the target translation.

2.1 Supported Architectures

While the framework defaults to standard encoder-decoder transformers, it is designed for architectural flexibility. It supports:

  • Standard Transformers: Utilizing multi-head attention for sequence mapping.
  • Recurrent Neural Networks (RNNs): Including LSTMs and GRUs for comparative baselines.
  • Universal Transformers: Models that utilize shared layers to iterate on a problem, potentially better mimicking the iterative nature of mathematical proofs.

2.2 Specialized Tokenization and Representation

A core insight of the Int2Int framework is that the representation of integers significantly affects model performance. The framework includes encoders for:

  • Positional Notation: Integers can be encoded in various bases (e.g., base 10 or base 1000). High-base encoding allows large numbers to be processed as shorter sequences, which often improves learning efficiency.
  • Complex Structures: Beyond simple integers, the framework supports the tokenization of matrices (via adjacency matrices), graphs, and polynomials.

2.3 Data Management

Mathematical datasets are often infinite in scope. Int2Int supports on-the-fly generation, allowing training data to be produced programmatically during the training loop. Furthermore, the data loader supports curriculum learning, where the model is initially exposed to simpler problems (e.g., small integers) before progressing to higher-complexity tasks.

3. Experimental Results

To validate the framework, we conducted experiments on two distinct mathematical tasks: synthetic arithmetic and algebraic geometry.

3.1 Greatest Common Divisor (GCD)

Using a dataset generated on the fly, we trained a model to compute the GCD of two integers. The model achieved 84.6% accuracy on the test set. Analysis revealed that the model's performance was closely tied to its numerical representation; accuracy approached 100% when the GCD was a product of the divisors of the base (e.g., powers of 2 or 5 in a base-10 system).

3.2 Elliptic Curve Rank Prediction

Predicting the rank of an elliptic curve—a fundamental property in algebraic geometry—is a notoriously difficult computational task. Using a dataset of 1.01 million curves from the LMFDB (L-functions and Modular Forms Database), the Int2Int model was tasked with predicting the rank based on the curve's coefficients.

After 100 epochs, the model achieved 49.9% overall accuracy. While this figure reflects the inherent difficulty of the task, it significantly outperformed the modal predictor baseline. Most notably, the model predicted rank 1 curves with 88.7% accuracy, suggesting that transformers can effectively identify specific mathematical signatures within raw coefficient data.

4. Discussion and Limitations

Int2Int demonstrates that mathematical logic can be captured by attention mechanisms similar to those used in natural language. However, several limitations remain for future development:

  • Scaling: Multi-GPU support is currently limited to Slurm-based environments, and model sharding for massive architectures is not yet implemented.
  • Decoding Strategies: While transformers benefit from beam search—allowing the model to propose and rank multiple candidate solutions—the RNN implementations are currently limited to greedy decoding.
  • Architecture Evolution: Support for decoder-only configurations (similar to the GPT family) is currently in development to broaden the framework's utility.

5. Conclusion

Int2Int provides a robust starting point for mathematicians looking to leverage the power of transformers. By standardizing the pipeline for mathematical seq2seq tasks, the framework facilitates more rapid experimentation and discovery in the digital age of mathematics.

Summary updated

BibTeX
@misc{charton2025int,
  author = {François Charton},
  title = {{Int2Int: a framework for mathematics with transformers}},
  year = {2025},
  eprint = {2502.17513},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.17513},
}
Figure