Abstract
Yue Fang, Shaohan Huang, Xin Yu, Haizhen Huang, Zihan Zhang, Weiwei Deng, Furu Wei, Feng Sun, Qi Zhang, Zhi Jin. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.
2025.
Introduction
While Large Language Models (LLMs) have become proficient at informal mathematical reasoning in plain English, they struggle with auto-formalization—the process of translating informal logic into machine-verifiable languages like Lean 4. Formal languages are unforgiving; a single misplaced character or logical misalignment can render a proof useless.
Traditional approaches relying on Supervised Fine-Tuning (SFT) often produce Lean code that looks correct but fails to compile or subtly alters the original problem's meaning. To bridge this gap, the authors introduce ReLean, a reinforcement learning framework that provides fine-grained, multi-dimensional feedback during training.
Methodology: The ReLean Framework
ReLean moves beyond simple imitation by using Reinforcement Learning from Model Feedback. The framework is built on three core pillars:
1. Multi-Aspect Dense Rewards
Instead of a single binary signal, ReLean utilizes four distinct reward models to guide the translator:
- Semantic Alignment ($r_s$): Uses back-translation to compare the embedding similarity of the generated Lean code (translated back to English) against the original input.
- Term-Level Alignment ($r_t$): Measures precision in capturing mathematical sub-terms (constants, variables) using Longest Common Subsequence (LCS).
- Global-Level Alignment ($r_g$): Evaluates structural resemblance to reference statements via edit distance.
- Compile-Checking ($r_c$): A binary reward verifying if the code passes the Lean 4 compiler.
2. Curriculum Learning
To stabilize training, the model follows an "easy-to-hard" progression. It first learns from examples with high semantic similarity and shorter sub-term sequences before tackling complex, multi-step theorems.
3. Optimization
These rewards are aggregated to guide a LLaMA-3-8B base model using Proximal Policy Optimization (PPO).
Key Results
ReLean demonstrated superior performance across major formal mathematics benchmarks:
- Lean Workbook: ReLean achieved 26.58% accuracy, significantly outperforming the specialized Herald model (22.36%) and general-purpose models like GPT-4o (19.78%).
- ProofNet: On this rigorous undergraduate-level benchmark, it reached 19.20% accuracy, nearly double that of InternLM2-Math.
Discussion and Insights
A critical insight from the study is that compilation does not guarantee correctness. A model can generate code that compiles perfectly but describes a completely different mathematical problem. ReLean’s semantic reward is essential for maintaining "intent."
Furthermore, the authors found that training on difficult examples too early leads to instability, making the multi-dimensional curriculum strategy essential for steady improvement. By separating structural feedback from semantic feedback, the framework prevents the model from "gaming" any single metric.
Limitations and Future Work
The primary limitation is the evaluation bottleneck. While training is automated, final assessments of semantic correctness still rely on human experts or expensive "judge" LLMs. Future work aims to develop fully automated formal solvers to verify logical equivalence without human intervention.
Summary updated