Abstract
Recent progress in automated theorem proving (ATP) relies heavily on the miniF2F benchmark. While autoformalization models report near-perfect accuracy, end-to-end performance remains low. This paper investigates this discrepancy, identifying widespread errors and semantic mismatches in the original dataset. We introduce miniF2F-v2, a human-verified benchmark in Lean 4, and demonstrate that current LLM-based evaluation metrics are significantly inflated compared to human expert judgment.
Introduction: The Autoformalization Gap
The pursuit of AI capable of competing in the International Math Olympiad (IMO) typically involves a two-stage pipeline: autoformalization (translating informal problems into formal languages like Lean) and automated theorem proving (ATP). While recent models claim ~97% accuracy on the standard miniF2F benchmark, integrated systems solve only about 36% of problems.
This research identifies the root cause: the original miniF2F benchmark is riddled with inconsistencies. Formal statements often fail to match their informal counterparts, problems are frequently over-simplified by providing the answer within the theorem statement, and some are mathematically unprovable due to coding errors.
Methods: The miniF2F-v2 Dataset
To address these limitations, we conducted a manual audit of all 488 theorems in miniF2F. We corrected 16 unprovable statements and resolved semantic mismatches by creating two distinct versions of the dataset:
- miniF2F-v2s (Simplified): Informal statements were adjusted to match existing simplified formal statements, ensuring a clean 1-to-1 mapping where the solution is often explicitly stated.
- miniF2F-v2c (Competition-level): Formal statements were rewritten to match the original difficult Olympiad problems, restoring multiple-choice formats and removing "spoilers" to force the AI to derive the answer independently.
Key Insights and Results
The Fallacy of LLM-as-a-Judge
A critical finding is the unreliability of using LLMs to evaluate formal code. While automated judges reported 97.5% accuracy for the Herald translator, human expert review placed it at only 62.7%. LLMs often overlook semantic mismatches that prevent a formal statement from being a valid translation of the original problem.
The Simplification Trap
Over half of the original miniF2F problems were "excessively simplified." When these problems were restored to their true competition difficulty in miniF2F-v2c, prover performance dropped sharply. For example, Deepseek-Prover-V1.5-RL's accuracy fell from 46.7% to 22.2% on the simplified subset.
End-to-End Performance
Evaluating translation and proving in isolation masks compounding errors. In a strict Olympiad setting—where no credit is given for proving a simplified version of a problem—the best end-to-end pipeline achieved only 36.9% on the original dataset. However, on the cleaned miniF2F-v2s, performance rose to 50.0%, indicating that high-quality, discrepancy-free data is essential for training and evaluating robust reasoning models.
Conclusion and Future Work
The miniF2F-v2 dataset provides a "platinum-quality" benchmark for Lean 4, offering a more accurate measure of progress toward human-level mathematical reasoning. Future work should focus on expanding these corrections to other formal languages like Isabelle and Metamath and utilizing this clean data to conduct ablation studies on model architectures and reasoning mechanisms.
Summary updated