From Sorry to QED: Refining LLM Math Proofs via Angelic Execution

Yanju Chen, Hanzhi Liu, Ruizhe Qian, Jiaming Shan, Xu Yang, Yufei Ding, Osbert Bastani, Yu FengarXiv 2026 · arXiv

TL;DR

MathEye is a new framework that transforms imperfect, LLM-generated mathematical proof outlines into fully verified Lean 4 proofs. By using "angelic execution" to bypass minor errors and MaxSAT optimization to minimize human intervention, MathEye solves 93.3% of graduate-level math benchmarks, reducing the required human effort by 55% compared to naive approaches.

What Problem Does This Paper Address?

Formalizing mathematics in proof assistants like Lean is a "high-stakes, high-effort" endeavor. While Large Language Models (LLMs) have become surprisingly good at sketching the high-level logic of a proof, they are notoriously brittle when it comes to the details. A single missing side condition (like forgetting to prove a function is continuous before integrating it) or a slightly misapplied library lemma causes a standard proof assistant to abort the entire script.

For a human, this is frustrating: the LLM might have 90% of the logic right, but the computer rejects the whole thing. Existing automated provers often fail on complex, multi-step graduate-level theorems because they can't bridge these small symbolic gaps. MathEye tackles this by allowing the proof assistant to be "optimistic"—accepting the high-level structure while isolating exactly which parts need a human's "sanity check" to proceed.

Methods

The MathEye workflow consists of four main stages:

  1. Sketching and Angelic Checking: MathEye first asks an LLM (like ChatGPT) to generate a Lean proof script. Instead of failing at the first error, MathEye uses Angelic Execution. If a specific tactic fails, the system replaces that fragment with a sorry placeholder. This preserves the overall proof structure, resulting in an "angelic proof" that is structurally valid but contains holes.
  2. Proof Dependence Graph (PDG): The system builds a directed hypergraph where nodes are mathematical symbols (hypotheses, lemmas) and edges represent how proof fragments connect them. This captures the "data flow" of the logic.
  3. Residual Analysis via MaxSAT: Not all sorry placeholders are created equal. Some are low-level technicalities, while others are core logical leaps. MathEye formulates the selection of which holes to ask a human to fill as a MaxSAT optimization problem. It seeks the "minimal residual core"—the smallest, easiest-to-verify set of claims that, if true, would logically imply that all other holes in the proof can be filled.
  4. Minimal-Interaction Refinement: MathEye translates these essential obligations from Lean code into ordinary mathematical language (LaTeX). A human mathematician simply answers "yes" or "no" to these questions. Once confirmed, MathEye automatically reconstructs the final, fully verified Lean proof.
The MathEye workflow: from an initial LLM sketch to a verified proof via angelic checking and human interaction.

Datasets

The authors evaluated MathEye on a suite of 30 advanced mathematical benchmarks formalized in Lean 4.

  • Source: These problems were drawn from graduate-level qualifying exams at UCLA and CU Boulder.
  • Complexity: Unlike "competition-style" problems (like those in the IMO) which often rely on clever tricks, these benchmarks require multi-step reasoning in analysis (measure theory, integration) and algebra (group and ring theory).
  • Baselines: MathEye was compared against five state-of-the-art LLMs, including DeepSeek-Prover-V2 and ChatGPT 5.1 (Extended Thinking), in both one-shot and iterative repair modes.

Key Insights and Ideas

  • Structure is more robust than Syntax: The core "aha" moment is that LLMs are often better at "global" reasoning than "local" symbolic execution. By isolating local failures, we can salvage the global logic.
  • The Power of Logical Interdependence: In a complex proof, many errors are redundant. If a human confirms a high-level intermediate lemma, the system can often infer that the low-level "side conditions" required for that lemma must also be true, or are at least trivial enough for the human to have implicitly verified.
  • Plain Language is the Bridge: By presenting goals in standard math notation rather than Lean syntax, MathEye allows mathematicians who aren't experts in formal methods to contribute to formal verification.

Key Results

  • High Success Rate: MathEye solved 28 out of 30 (93.3%) problems. In contrast, the union of all five automated baseline systems solved at most 30% of the problems, even when given multiple attempts.
  • Efficiency: Residual analysis reduced the number of required user interactions by 55% compared to a naive approach of asking about every single failure.
  • Low Human Overhead: On average, a problem required only ~6 interactions and took a human roughly 3 minutes to resolve.
Ablation study showing how residual analysis significantly reduces the number of human interactions across different benchmarks.

Limitations and Future Work

  • Sketch Dependency: MathEye relies on the initial LLM sketch. If the LLM provides a fundamentally flawed proof strategy (e.g., trying to prove a true statement using a false lemma), MathEye cannot "hallucinate" a fix.
  • Explanatory Value: The current cost model focuses on minimizing the number of queries, but it doesn't yet account for the explanatory value of a query. Some queries might be harder to answer but provide more insight into the proof's validity.
  • Closing the Loop: The authors suggest that the human judgments collected by MathEye could be used as high-quality training data to fine-tune future LLMs, potentially reducing the need for human intervention over time.

Summary updated

Abstract

Large language models (LLMs) can now generate Lean proof scripts that capture the high-level structure of many mathematical arguments. However, these scripts frequently fail on complex theorems: a single missing side condition or misapplied lemma can cause the prover to abort early and discard an otherwise viable proof outline. This paper presents MathEye, a framework that converts imperfect LLM-generated proof scripts into fully verified Lean proofs by combining symbolic reasoning with minimal human interaction. At the core of MathEye is angelic proof checking, an optimistic execution model that replaces failing proof fragments with placeholders while preserving the overall structure of the argument. This optimistic check produces an angelic proof that isolates the exact proof obligations that the LLM fails to discharge. Because these unresolved obligations are often logically interdependent, asking a mathematician to address every placeholder directly is inefficient. To minimize human effort, MathEye performs residual analysis over a proof dependence graph that captures symbol-level data flow in the angelic proof. By formulating obligation selection as a MaxSAT optimization problem, MathEye selects a minimal-cost set of essential obligations whose resolution suffices to complete the entire proof. MathEye presents these essential obligations to the mathematician in ordinary mathematical language and uses the mathematician's answers to automatically reconstruct the complete Lean proof. Experiments on 30 real-world benchmarks across analysis and algebra show that MathEye solves 93.3% of the problems while reducing the number of required user interactions by 55%.
BibTeX
@misc{chen2026from,
  author = {Yanju Chen and Hanzhi Liu and Ruizhe Qian and Jiaming Shan and Xu Yang and Yufei Ding and Osbert Bastani and Yu Feng},
  title = {{From Sorry to QED: Refining LLM Math Proofs via Angelic Execution}},
  year = {2026},
  eprint = {2611.22103},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.22103},
}
Figure