Introduction
MLFMF is a massive new open-source benchmark designed to bridge the gap between machine learning and formal mathematics. By extracting over 250,000 entries from the Agda and Lean proof assistants, researchers have created the largest dataset of its kind, representing mathematical knowledge as both structured syntax trees and complex relational networks.
The Challenge of Premise Selection
Formalizing mathematics—writing proofs that a computer can verify with 100% certainty—is notoriously difficult and time-consuming. While "proof assistants" like Lean and Agda help, users often struggle with premise selection: out of tens of thousands of existing theorems, which one is needed to prove the current goal?
Existing datasets for this task are often limited to a single programming language or are too small for modern deep learning. Furthermore, learning from the "vernacular" (the human-written code) is messy because it includes meta-information and "tactics" that obscure the underlying logic. MLFMF addresses this by providing a standardized, multi-language dataset based on the internal, fully elaborated logic of mathematical libraries.
Data Extraction Pipeline
To transform raw mathematical libraries into machine-learnable formats, the authors developed a two-stage pipeline:
- Extraction (
lib2sexp): A language-dependent tool extracts the internal representation of every theorem, definition, and proof into S-expressions (Lisp-like nested lists). This captures the Abstract Syntax Tree (AST) of the math, providing a cleaner view of the logic than human-written code.
- Graph Construction (
sexp2graph): A Python-based tool converts these S-expressions into a directed multi-graph. In this graph, nodes represent entries (theorems, axioms, modules) and edges represent relationships (e.g., a theorem referencing a specific lemma in its proof).
Dataset Composition
The MLFMF suite includes four major libraries, totaling over 250,000 entries:
- Agda Standard Library: The foundational library for Agda.
- Agda-unimath: A large library for univalent mathematics.
- TypeTopology: A library focused on topology and constructive mathematics.
- Lean Mathlib4: The massive, community-driven library for Lean 4, currently considered the "gold standard" in the formalization community.
Experimental Results and Insights
The authors evaluated several baseline models on a link prediction task, aiming to predict which lemma a theorem will reference. They tested simple "Bag of Words" and TF-IDF approaches alongside sophisticated graph-based methods.
- Structure is Signal: The researchers found that the structural graph of how theorems relate is often more important than the textual content of the theorems themselves.
- Node2vec Success: The graph-based Node2vec model was the clear winner. On Agda libraries, it achieved an accuracy of 0.96–0.98. The "true" reference was usually found within the top 5 recommendations (Mean Minimal Rank between 3.2 and 5.8).
- The Lean Challenge: Performance on Lean’s
Mathlib4was significantly lower, with a Mean Minimal Rank of 195. The authors attribute this to the sheer scale of Mathlib4 and Lean's heavy reliance on automated "tactics," which result in a more complex reference graph compared to Agda. - The "Math Jargon" Problem: Standard NLP embeddings like FastText performed poorly. Words like "group" or "ring" have highly specific meanings in a mathematical context that general-purpose models fail to capture.
Limitations and Future Directions
The authors note a few limitations that present opportunities for future work:
- The "Arrow of Time": The current train-test split is random. In reality, a recommendation system can only suggest theorems written before the current one. Future versions aim to include chronological metadata.
- Claim Detection: The dataset does not explicitly distinguish between a basic definition and a major theorem. Adding these labels would enable more nuanced recommendation tasks.
- Beyond Baselines: The provided results serve as a baseline floor. The authors invite the ML community to apply more advanced Graph Neural Networks (GNNs) and Transformers to improve upon these initial scores.
Summary updated