TL;DR
PaperFormalizer is an agentic pipeline that helps researchers transform their LaTeX-based papers into Lean-verified mathematical proofs. By keeping the author "in the loop" to resolve natural language ambiguities and approve error repairs, it makes rigorous formal verification a practical part of the scientific writing process.
What Problem Does This Paper Address?
Even the most meticulously proofread research papers often contain "small gaps"—overlooked boundary cases, missing regularity conditions, or implicit assumptions. While proof assistants like Lean can catch these errors, manual formalization is incredibly time-consuming, often taking weeks for a single paper.
Current AI-driven "autoformalization" tools face two major hurdles:
- Ambiguity: Natural language is imprecise. A machine might formalize a statement in a way the author never intended.
- Repair Choice: When a statement is found to be false, there are many ways to fix it. A machine cannot know which "repair" preserves the author's core scientific contribution.
PaperFormalizer addresses these by treating formalization as a collaborative process between an AI agent and the human author.
Methods
The PaperFormalizer pipeline consists of two main stages, as illustrated in the workflow below:
Stage I: Natural Language Review
- Transcription: An agent converts the LaTeX source into a dependency graph of definitions, assumptions, and theorems.
- Review: A reviewer agent checks each node for faithfulness to the text, ambiguity, and mathematical correctness. If it finds a potential error, a "fixer" agent proposes a minimal repair.
Stage II: Autoformalization
- Planning & Scaffolding: The system decides which parts to reuse from existing libraries (like Mathlib) and which to formalize from scratch. It then writes the Lean "scaffolding" (the statements of the theorems).
- Proof Filling: AI "workers" attempt to fill in the proofs, querying Lean's language server to ensure every step is valid.
- Linking: Finally, the system links every Lean declaration back to the specific sentence in the paper it formalizes, allowing the author to verify the translation.
The system runs inside coding agents like Claude Code or Codex and provides a local web interface for the author to review findings and approve changes.
Datasets
The authors evaluated PaperFormalizer on three of their own theoretical papers in fields like statistics and machine learning:
- A paper on Treatment Effects on Extreme Events.
- A paper on Optimal Transport.
- A paper on Neural Causal Models.
Key Insights and Ideas
- Author-in-the-Loop is Essential: The system is designed to stop and ask the author whenever a statement has multiple readings or a repair might change a main result. This ensures the final verified version actually reflects the author's intent.
- The "Personal Toolkit": As an author formalizes more papers, they build a library of reusable "Foundations" and "Tricks." This creates a feedback loop that makes each subsequent paper easier and cheaper to verify.
- Formalization as a Debugger: The process isn't just about getting a "stamp of approval"; it's a powerful way to find and fix subtle errors during the writing process itself.
Key Results
In their primary case study (a paper on extreme treatment effects), PaperFormalizer revealed that 5 out of 7 results did not hold as printed.
The pipeline identified the root causes—such as missing exogeneity conditions and incorrect radial limits—and proposed repairs that the authors accepted. Ultimately, all seven results were successfully verified in Lean.
In other tests:
- Optimal Transport: All 12 results were proved exactly as printed.
- Neural Causal Models: 20 of 29 results were proved; two non-main results were refuted (shown to be false with no simple repair).
Limitations and Future Work
- Human Time: While much faster than manual formalization, the author still needs to spend time reviewing AI findings and resolving ambiguities.
- Cost: Running high-reasoning LLMs (like GPT-4 or Claude 3 Opus) through many iterations can be expensive.
- Future Directions: The authors hope PaperFormalizer becomes a routine part of the research workflow, allowing scientists to share "Lean-verified" papers that others can build upon with absolute confidence in the underlying mathematics.
Summary updated