miniF2F-Lean Revisited: Reviewing Limitations and Charting a Path Forward

Azim Ospanov, Farzan Farnia, Roozbeh Yousefzadeh

Abstract

Recent progress in automated theorem proving (ATP) relies heavily on the miniF2F benchmark. While autoformalization models report near-perfect accuracy, end-to-end performance remains low. This paper investigates this discrepancy, identifying widespread errors and semantic mismatches in the original dataset. We introduce miniF2F-v2, a human-verified benchmark in Lean 4, and demonstrate that current LLM-based evaluation metrics are significantly inflated compared to human expert judgment.

Introduction: The Autoformalization Gap

The pursuit of AI capable of competing in the International Math Olympiad (IMO) typically involves a two-stage pipeline: autoformalization (translating informal problems into formal languages like Lean) and automated theorem proving (ATP). While recent models claim ~97% accuracy on the standard miniF2F benchmark, integrated systems solve only about 36% of problems.

This research identifies the root cause: the original miniF2F benchmark is riddled with inconsistencies. Formal statements often fail to match their informal counterparts, problems are frequently over-simplified by providing the answer within the theorem statement, and some are mathematically unprovable due to coding errors.

The automated theorem proving pipeline and potential failure modes including statement mismatches and prover failures.

Methods: The miniF2F-v2 Dataset

To address these limitations, we conducted a manual audit of all 488 theorems in miniF2F. We corrected 16 unprovable statements and resolved semantic mismatches by creating two distinct versions of the dataset:

  1. miniF2F-v2s (Simplified): Informal statements were adjusted to match existing simplified formal statements, ensuring a clean 1-to-1 mapping where the solution is often explicitly stated.
  2. miniF2F-v2c (Competition-level): Formal statements were rewritten to match the original difficult Olympiad problems, restoring multiple-choice formats and removing "spoilers" to force the AI to derive the answer independently.
Comparison of problem statements across miniF2F-v1, v2s (simplified), and v2c (competition-level), showing how 'spoilers' are handled.

Key Insights and Results

The Fallacy of LLM-as-a-Judge

A critical finding is the unreliability of using LLMs to evaluate formal code. While automated judges reported 97.5% accuracy for the Herald translator, human expert review placed it at only 62.7%. LLMs often overlook semantic mismatches that prevent a formal statement from being a valid translation of the original problem.

The Simplification Trap

Over half of the original miniF2F problems were "excessively simplified." When these problems were restored to their true competition difficulty in miniF2F-v2c, prover performance dropped sharply. For example, Deepseek-Prover-V1.5-RL's accuracy fell from 46.7% to 22.2% on the simplified subset.

End-to-End Performance

Evaluating translation and proving in isolation masks compounding errors. In a strict Olympiad setting—where no credit is given for proving a simplified version of a problem—the best end-to-end pipeline achieved only 36.9% on the original dataset. However, on the cleaned miniF2F-v2s, performance rose to 50.0%, indicating that high-quality, discrepancy-free data is essential for training and evaluating robust reasoning models.

Conclusion and Future Work

The miniF2F-v2 dataset provides a "platinum-quality" benchmark for Lean 4, offering a more accurate measure of progress toward human-level mathematical reasoning. Future work should focus on expanding these corrections to other formal languages like Isabelle and Metamath and utilizing this clean data to conduct ablation studies on model architectures and reasoning mechanisms.

Summary updated

BibTeX
@inproceedings{ospanov2025minif,
  author = {Azim Ospanov and Farzan Farnia and Roozbeh Yousefzadeh},
  title = {{miniF2F-Lean Revisited: Reviewing Limitations and Charting a Path Forward}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=KtaHv0YUyh},
}
Figure