NL2Lean: Translating Natural Language into Lean 4 through Multi-Aspect Reinforcement Learning

Yue Fang, Shaohan Huang, Yu Xin, Haizhen Huang, Zihan Zhang, Weiwei Deng, Furu Wei, Feng Sun, Qi Zhang, Zhi Jin2025 · DOI

Abstract

Yue Fang, Shaohan Huang, Xin Yu, Haizhen Huang, Zihan Zhang, Weiwei Deng, Furu Wei, Feng Sun, Qi Zhang, Zhi Jin. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

2025.

Introduction

While Large Language Models (LLMs) have become proficient at informal mathematical reasoning in plain English, they struggle with auto-formalization—the process of translating informal logic into machine-verifiable languages like Lean 4. Formal languages are unforgiving; a single misplaced character or logical misalignment can render a proof useless.

Traditional approaches relying on Supervised Fine-Tuning (SFT) often produce Lean code that looks correct but fails to compile or subtly alters the original problem's meaning. To bridge this gap, the authors introduce ReLean, a reinforcement learning framework that provides fine-grained, multi-dimensional feedback during training.

Methodology: The ReLean Framework

ReLean moves beyond simple imitation by using Reinforcement Learning from Model Feedback. The framework is built on three core pillars:

1. Multi-Aspect Dense Rewards

Instead of a single binary signal, ReLean utilizes four distinct reward models to guide the translator:

  • Semantic Alignment ($r_s$): Uses back-translation to compare the embedding similarity of the generated Lean code (translated back to English) against the original input.
  • Term-Level Alignment ($r_t$): Measures precision in capturing mathematical sub-terms (constants, variables) using Longest Common Subsequence (LCS).
  • Global-Level Alignment ($r_g$): Evaluates structural resemblance to reference statements via edit distance.
  • Compile-Checking ($r_c$): A binary reward verifying if the code passes the Lean 4 compiler.

2. Curriculum Learning

To stabilize training, the model follows an "easy-to-hard" progression. It first learns from examples with high semantic similarity and shorter sub-term sequences before tackling complex, multi-step theorems.

3. Optimization

These rewards are aggregated to guide a LLaMA-3-8B base model using Proximal Policy Optimization (PPO).

Overview of the ReLean framework, illustrating (a) the curriculum data sorting process and (b) the multi-aspect reinforcement learning loop.

Key Results

ReLean demonstrated superior performance across major formal mathematics benchmarks:

  • Lean Workbook: ReLean achieved 26.58% accuracy, significantly outperforming the specialized Herald model (22.36%) and general-purpose models like GPT-4o (19.78%).
  • ProofNet: On this rigorous undergraduate-level benchmark, it reached 19.20% accuracy, nearly double that of InternLM2-Math.
Accuracy improvements of ReLean (Reward Model RL) compared to metric-based RL over training iterations on Lean Workbook and ProofNet.

Discussion and Insights

A critical insight from the study is that compilation does not guarantee correctness. A model can generate code that compiles perfectly but describes a completely different mathematical problem. ReLean’s semantic reward is essential for maintaining "intent."

Furthermore, the authors found that training on difficult examples too early leads to instability, making the multi-dimensional curriculum strategy essential for steady improvement. By separating structural feedback from semantic feedback, the framework prevents the model from "gaming" any single metric.

Limitations and Future Work

The primary limitation is the evaluation bottleneck. While training is automated, final assessments of semantic correctness still rely on human experts or expensive "judge" LLMs. Future work aims to develop fully automated formal solvers to verify logical equivalence without human intervention.

Summary updated

BibTeX
@misc{fang2025nl,
  author = {Yue Fang and Shaohan Huang and Yu Xin and Haizhen Huang and Zihan Zhang and Weiwei Deng and Furu Wei and Feng Sun and Qi Zhang and Zhi Jin},
  title = {{NL2Lean: Translating Natural Language into Lean 4 through Multi-Aspect Reinforcement Learning}},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.1586},
}
Figure