Autoformalizer with Tool Feedback

Qi Guo, Jianing Wang, Jianfei Zhang, Deyang Kong, Xiangzhou Huang, Xiangyu Xi, Wei Wang, Jingang Wang, Xunliang Cai, Shikun Zhang, Wei YeSubmission to ICLR 2026 · Rejected

TL;DR

Autoformalization—translating natural language math into machine-verifiable code like Lean 4—is bottlenecked by code that fails to compile (syntactic errors) or subtly alters the problem's meaning (semantic drift). The authors introduce Autoformalizer with Tool Feedback (ATF), an agentic framework that allows an LLM to iteratively write, verify, and repair formal statements using both a Lean 4 compiler and an ensemble LLM consistency checker. By pairing this multi-turn tool interaction with expert iteration and Direct Preference Optimization (DPO), ATF dramatically improves formalization accuracy and inference-time scaling.

What Problem Does This Paper Address?

Automated Theorem Proving (ATP) systems (like DeepSeek-Prover or Lean-based provers) require formal mathematical specifications to search for and verify proofs. However, formal statement data is scarce. While LLMs can autoformalize informal competition math into Lean 4, they struggle with two major failure modes:

  1. Syntactic Invalidity: Formal proof assistants are unforgiving. Even leading formalizers fail compilation around 40% of the time due to syntax nuances across library versions.
  2. Semantic Inconsistency: Passing compiler checks does not mean the translation is faithful. Models often swap quantifiers ($\forall$ vs. $\exists$), change domains, or alter preconditions, creating problems that no longer match the original query.

Existing approaches rely on one-shot prompting or training single-pass models that have no mechanism to test or reflect on their outputs.

Methods

ATF turns formalization into an interactive generation-and-repair loop powered by two dedicated tools:

  • Syntax Check Tool: Feeds statements to the Lean 4 (v4.15) compiler. To handle massive execution overheads, the authors deploy an initial rule-based pre-check (catching unclosed parentheses or missing by sorry) and a grouped execution pipeline that batches multiple statements into isolated Lean namespaces in a single file—cutting average execution latency by 87% (from 6.2s to 0.8s per statement).
  • Consistency Check Tool: Evaluates whether the formal code strictly preserves the mathematical semantics of the natural language problem. Because single LLM judges exhibit high false-positive rates on subtly perturbed statements, ATF uses an ensemble vote between QWQ-32B and Qwen3-32B, flagging statements as consistent only when both agree.

The training framework follows three stages:

  1. Cold Start: Claude 3.5 Sonnet generates multi-turn reasoning and tool-calling trajectories enforcing strict interaction protocols (e.g., consistency is only checked after syntax passes; modifications require re-checking).
  2. Expert Iteration: The model (initialized from Qwen3-32B) generates trajectories on 178k competition problems from NuminaMath-1.5. Successful trajectories (<8 revision rounds) are collected into training data for subsequent fine-tuning rounds.
  3. DPO for Efficient Revisions: To discourage endless, repetitive tool-call loops, the authors use DPO with a negative log-likelihood penalty, treating trajectories that reach valid statements in fewer turns as positive pairs over longer, redundant revision paths.

Datasets

  • Training Data: A curated subset of 178k challenging competition problems from NuminaMath-1.5 (including AMC/AIME, Olympiads, and Chinese math contests).
  • Consistency Benchmark: 800 math queries across 8 domains (Algebra, Calculus, Combinatorics, etc.) paired with Lean statements and adversarial semantic perturbations generated via Gemini-2.5-Pro (character similarity >0.95).
  • Evaluation Benchmarks: FormalMath-Lite and ProverBench (in-distribution), alongside CombiBench (an out-of-distribution benchmark focused on complex combinatorial problems).
  • Released Dataset: Numina-ATF, containing 752,553 synthetic formal Lean 4 statements verified for both syntax and semantic consistency.

Key Insights and Ideas

  • Compiler feedback is not enough: A Lean compiler only guarantees logical well-formedness, not faithful translation. Coupling a formal compiler with semantic LLM judges prevents models from taking "shortcuts" that satisfy the type checker by trivializing the theorem statement.
  • Ensemble judging catches subtle perturbations: Individual LLM evaluators misclassified ~9% of subtly corrupted statements as correct. Requiring consensus between two different reasoning models reduced false positives to below 6%.
  • DPO regularizes agentic revision loops: Left unchecked, iterative tool loops can produce repeating compiler errors. Applying preference optimization on trajectory length directly trains the policy to make targeted, high-leverage corrections.
  • Inference-time scaling applies to formalization: Allowing the model more revision attempts (inference search depth) and parallel samples (breadth) steadily improves semantic consistency, even extending beyond the maximum revision counts seen during training.

Key Results

  • Substantial SOTA Gains: On CombiBench (OOD), ATF-32B reached 65.38% Pass@1 consistency—a 29.13% absolute increase over Goedel-V2-Formalizer-32B (36.25%) and StepFun-Formalizer-32B (25.50%).
  • Consistent Wins Across Benchmarks: ATF-32B achieved Pass@1 consistency of 94.51% on FormalMath-Lite and 89.78% on ProverBench.
  • Distillation Efficiency: The smaller ATF-8B-Distilled outperformed all previous 32B baselines on FormalMath-Lite (91.12% consistency) and ProverBench (85.16%).
  • Parallel Sampling (Pass@32): ATF achieved a 100% success rate on CombiBench when scaled to 32 parallel candidates.
  • Human Validation: Expert human evaluations confirmed these improvements (human-judge Pearson correlation $r = 0.746$), with ATF-32B rated 49.0% consistent on CombiBench versus 22.0% for Goedel-V2.

Limitations and Future Work

  • Diminishing Revision Returns: Success rates on the consistency check drop precipitously with each revision attempt (from 69.5% on attempt 1 down to 8.8% on attempt 8), showing that the model struggles once its most confident recovery strategies fail.
  • Inference Latency & Cost: Calling an external Lean compiler and two 32B judge models at test time introduces non-trivial latency compared to single-pass generation.
  • Future Work: Utilizing the 750K open-sourced Numina-ATF formal statements to train theorem provers, expanding tools to support other interactive proof assistants (e.g., Isabelle, Coq), and designing smarter backtracking strategies for complex revisions.

Summary updated

Abstract

Autoformalization addresses the scarcity of data for Automated Theorem Proving (ATP) by translating mathematical problems from natural language into formal statements. Efforts in recent work shift from directly prompting large language models to training an end-to-end formalizer model from scratch, achieving remarkable advancements. However, existing formalizer still struggles to consistently generate valid statements that meet syntactic validity and semantic consistency. To address this issue, we propose the Autoformalizer with Tool Feedback (ATF), a novel approach that incorporates syntactic and consistency information as tools into the formalization process. By integrating Lean 4 compilers for syntax corrections and employing a multi-LLMs-as-judge approach for consistency validation, the model is able to adaptively refine generated statements according to the tool feedback, enhancing both syntactic validity and semantic consistency. The training of ATF involves a cold-start phase on synthetic tool-calling data, an expert iteration phase to improve formalization capabilities, and Direct Preference Optimization to reduce ineffective revisions. Experimental results show that ATF markedly outperforms a range of baseline formalizer models, with its superior performance further validated by human evaluations. Subsequent analysis reveals that ATF demonstrates excellent inference scaling properties. Moreover, we open-source Numina-ATF, a dataset containing 750K synthetic formal statements to facilitate advancements in autoformalization and ATP research.
BibTeX
@misc{guo2025autoformalizer,
  author = {Qi Guo and Jianing Wang and Jianfei Zhang and Deyang Kong and Xiangzhou Huang and Xiangyu Xi and Wei Wang and Jingang Wang and Xunliang Cai and Shikun Zhang and Wei Ye},
  title = {{Autoformalizer with Tool Feedback}},
  year = {2025},
  eprint = {2510.06857},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.06857},
}

Loading related content...

Figure