TL;DR
MathEye is a new framework that transforms imperfect, LLM-generated mathematical proof outlines into fully verified Lean 4 proofs. By using "angelic execution" to bypass minor errors and MaxSAT optimization to minimize human intervention, MathEye solves 93.3% of graduate-level math benchmarks, reducing the required human effort by 55% compared to naive approaches.
What Problem Does This Paper Address?
Formalizing mathematics in proof assistants like Lean is a "high-stakes, high-effort" endeavor. While Large Language Models (LLMs) have become surprisingly good at sketching the high-level logic of a proof, they are notoriously brittle when it comes to the details. A single missing side condition (like forgetting to prove a function is continuous before integrating it) or a slightly misapplied library lemma causes a standard proof assistant to abort the entire script.
For a human, this is frustrating: the LLM might have 90% of the logic right, but the computer rejects the whole thing. Existing automated provers often fail on complex, multi-step graduate-level theorems because they can't bridge these small symbolic gaps. MathEye tackles this by allowing the proof assistant to be "optimistic"—accepting the high-level structure while isolating exactly which parts need a human's "sanity check" to proceed.
Methods
The MathEye workflow consists of four main stages:
- Sketching and Angelic Checking: MathEye first asks an LLM (like ChatGPT) to generate a Lean proof script. Instead of failing at the first error, MathEye uses Angelic Execution. If a specific tactic fails, the system replaces that fragment with a
sorryplaceholder. This preserves the overall proof structure, resulting in an "angelic proof" that is structurally valid but contains holes. - Proof Dependence Graph (PDG): The system builds a directed hypergraph where nodes are mathematical symbols (hypotheses, lemmas) and edges represent how proof fragments connect them. This captures the "data flow" of the logic.
- Residual Analysis via MaxSAT: Not all
sorryplaceholders are created equal. Some are low-level technicalities, while others are core logical leaps. MathEye formulates the selection of which holes to ask a human to fill as a MaxSAT optimization problem. It seeks the "minimal residual core"—the smallest, easiest-to-verify set of claims that, if true, would logically imply that all other holes in the proof can be filled. - Minimal-Interaction Refinement: MathEye translates these essential obligations from Lean code into ordinary mathematical language (LaTeX). A human mathematician simply answers "yes" or "no" to these questions. Once confirmed, MathEye automatically reconstructs the final, fully verified Lean proof.
Datasets
The authors evaluated MathEye on a suite of 30 advanced mathematical benchmarks formalized in Lean 4.
- Source: These problems were drawn from graduate-level qualifying exams at UCLA and CU Boulder.
- Complexity: Unlike "competition-style" problems (like those in the IMO) which often rely on clever tricks, these benchmarks require multi-step reasoning in analysis (measure theory, integration) and algebra (group and ring theory).
- Baselines: MathEye was compared against five state-of-the-art LLMs, including DeepSeek-Prover-V2 and ChatGPT 5.1 (Extended Thinking), in both one-shot and iterative repair modes.
Key Insights and Ideas
- Structure is more robust than Syntax: The core "aha" moment is that LLMs are often better at "global" reasoning than "local" symbolic execution. By isolating local failures, we can salvage the global logic.
- The Power of Logical Interdependence: In a complex proof, many errors are redundant. If a human confirms a high-level intermediate lemma, the system can often infer that the low-level "side conditions" required for that lemma must also be true, or are at least trivial enough for the human to have implicitly verified.
- Plain Language is the Bridge: By presenting goals in standard math notation rather than Lean syntax, MathEye allows mathematicians who aren't experts in formal methods to contribute to formal verification.
Key Results
- High Success Rate: MathEye solved 28 out of 30 (93.3%) problems. In contrast, the union of all five automated baseline systems solved at most 30% of the problems, even when given multiple attempts.
- Efficiency: Residual analysis reduced the number of required user interactions by 55% compared to a naive approach of asking about every single failure.
- Low Human Overhead: On average, a problem required only ~6 interactions and took a human roughly 3 minutes to resolve.
Limitations and Future Work
- Sketch Dependency: MathEye relies on the initial LLM sketch. If the LLM provides a fundamentally flawed proof strategy (e.g., trying to prove a true statement using a false lemma), MathEye cannot "hallucinate" a fix.
- Explanatory Value: The current cost model focuses on minimizing the number of queries, but it doesn't yet account for the explanatory value of a query. Some queries might be harder to answer but provide more insight into the proof's validity.
- Closing the Loop: The authors suggest that the human judgments collected by MathEye could be used as high-quality training data to fine-tune future LLMs, potentially reducing the need for human intervention over time.
Summary updated