TL;DR
Autoformalization—translating natural language math into machine-verifiable code like Lean 4—is bottlenecked by code that fails to compile (syntactic errors) or subtly alters the problem's meaning (semantic drift). The authors introduce Autoformalizer with Tool Feedback (ATF), an agentic framework that allows an LLM to iteratively write, verify, and repair formal statements using both a Lean 4 compiler and an ensemble LLM consistency checker. By pairing this multi-turn tool interaction with expert iteration and Direct Preference Optimization (DPO), ATF dramatically improves formalization accuracy and inference-time scaling.
What Problem Does This Paper Address?
Automated Theorem Proving (ATP) systems (like DeepSeek-Prover or Lean-based provers) require formal mathematical specifications to search for and verify proofs. However, formal statement data is scarce. While LLMs can autoformalize informal competition math into Lean 4, they struggle with two major failure modes:
- Syntactic Invalidity: Formal proof assistants are unforgiving. Even leading formalizers fail compilation around 40% of the time due to syntax nuances across library versions.
- Semantic Inconsistency: Passing compiler checks does not mean the translation is faithful. Models often swap quantifiers ($\forall$ vs. $\exists$), change domains, or alter preconditions, creating problems that no longer match the original query.
Existing approaches rely on one-shot prompting or training single-pass models that have no mechanism to test or reflect on their outputs.
Methods
ATF turns formalization into an interactive generation-and-repair loop powered by two dedicated tools:
- Syntax Check Tool: Feeds statements to the Lean 4 (v4.15) compiler. To handle massive execution overheads, the authors deploy an initial rule-based pre-check (catching unclosed parentheses or missing
by sorry) and a grouped execution pipeline that batches multiple statements into isolated Lean namespaces in a single file—cutting average execution latency by 87% (from 6.2s to 0.8s per statement). - Consistency Check Tool: Evaluates whether the formal code strictly preserves the mathematical semantics of the natural language problem. Because single LLM judges exhibit high false-positive rates on subtly perturbed statements, ATF uses an ensemble vote between QWQ-32B and Qwen3-32B, flagging statements as consistent only when both agree.
The training framework follows three stages:
- Cold Start: Claude 3.5 Sonnet generates multi-turn reasoning and tool-calling trajectories enforcing strict interaction protocols (e.g., consistency is only checked after syntax passes; modifications require re-checking).
- Expert Iteration: The model (initialized from Qwen3-32B) generates trajectories on 178k competition problems from NuminaMath-1.5. Successful trajectories (<8 revision rounds) are collected into training data for subsequent fine-tuning rounds.
- DPO for Efficient Revisions: To discourage endless, repetitive tool-call loops, the authors use DPO with a negative log-likelihood penalty, treating trajectories that reach valid statements in fewer turns as positive pairs over longer, redundant revision paths.
Datasets
- Training Data: A curated subset of 178k challenging competition problems from NuminaMath-1.5 (including AMC/AIME, Olympiads, and Chinese math contests).
- Consistency Benchmark: 800 math queries across 8 domains (Algebra, Calculus, Combinatorics, etc.) paired with Lean statements and adversarial semantic perturbations generated via Gemini-2.5-Pro (character similarity >0.95).
- Evaluation Benchmarks: FormalMath-Lite and ProverBench (in-distribution), alongside CombiBench (an out-of-distribution benchmark focused on complex combinatorial problems).
- Released Dataset: Numina-ATF, containing 752,553 synthetic formal Lean 4 statements verified for both syntax and semantic consistency.
Key Insights and Ideas
- Compiler feedback is not enough: A Lean compiler only guarantees logical well-formedness, not faithful translation. Coupling a formal compiler with semantic LLM judges prevents models from taking "shortcuts" that satisfy the type checker by trivializing the theorem statement.
- Ensemble judging catches subtle perturbations: Individual LLM evaluators misclassified ~9% of subtly corrupted statements as correct. Requiring consensus between two different reasoning models reduced false positives to below 6%.
- DPO regularizes agentic revision loops: Left unchecked, iterative tool loops can produce repeating compiler errors. Applying preference optimization on trajectory length directly trains the policy to make targeted, high-leverage corrections.
- Inference-time scaling applies to formalization: Allowing the model more revision attempts (inference search depth) and parallel samples (breadth) steadily improves semantic consistency, even extending beyond the maximum revision counts seen during training.
Key Results
- Substantial SOTA Gains: On CombiBench (OOD), ATF-32B reached 65.38% Pass@1 consistency—a 29.13% absolute increase over Goedel-V2-Formalizer-32B (36.25%) and StepFun-Formalizer-32B (25.50%).
- Consistent Wins Across Benchmarks: ATF-32B achieved Pass@1 consistency of 94.51% on FormalMath-Lite and 89.78% on ProverBench.
- Distillation Efficiency: The smaller ATF-8B-Distilled outperformed all previous 32B baselines on FormalMath-Lite (91.12% consistency) and ProverBench (85.16%).
- Parallel Sampling (Pass@32): ATF achieved a 100% success rate on CombiBench when scaled to 32 parallel candidates.
- Human Validation: Expert human evaluations confirmed these improvements (human-judge Pearson correlation $r = 0.746$), with ATF-32B rated 49.0% consistent on CombiBench versus 22.0% for Goedel-V2.
Limitations and Future Work
- Diminishing Revision Returns: Success rates on the consistency check drop precipitously with each revision attempt (from 69.5% on attempt 1 down to 8.8% on attempt 8), showing that the model struggles once its most confident recovery strategies fail.
- Inference Latency & Cost: Calling an external Lean compiler and two 32B judge models at test time introduces non-trivial latency compared to single-pass generation.
- Future Work: Utilizing the 750K open-sourced Numina-ATF formal statements to train theorem provers, expanding tools to support other interactive proof assistants (e.g., Isabelle, Coq), and designing smarter backtracking strategies for complex revisions.
Summary updated