Synthetic Proofs with Tool-Integrated Reasoning: Contrastive Alignment for LLM Mathematics with Lean

Mark Obozov, Michael Diskin, Aleksandr Beznosikov, Alexander Gasnikov, Serguei BarannikovOpenReview 2025 · OpenReview

Abstract

Modern mathematical reasoning benchmarks primarily focus on answer finding rather than proof verification, creating a gap in evaluating the proving capabilities of large language models (LLMs). We present a methodology for generating diverse mathematical proof tasks using formal tools. Our approach combines Lean-based synthetic problem generation with a novel Tool-Integrated Reasoning (TiR) framework for partial proof validation, and it uses contrastive preference optimization to align the model's proof outputs. Experiments on the Qwen-2.5 family of models demonstrate meaningful improvements in mathematical reasoning, particularly for smaller models. Our aligned models achieve up to a 57% higher success rate than baselines on the MiniF2F benchmark (across 0.5B, 1.5B, and 7B parameter models). These results highlight the potential of synthetic data and integrated validation for advancing LLM-based mathematical reasoning.

Introduction

While Large Language Models (LLMs) have become increasingly proficient at finding numerical answers to mathematical problems, they often struggle to provide rigorous, step-by-step proofs. Formal theorem proving—using languages like Lean—is the gold standard for accuracy, but it is computationally expensive and difficult for models to learn due to an effectively infinite search space. This research seeks to bridge the gap between informal natural language reasoning and formal verification by creating a scalable pipeline for high-quality synthetic proof data generation and model alignment.

Methodology

The researchers developed a three-stage pipeline to enhance mathematical reasoning capabilities:

1. Tree-Based Conjecture Generation

Using the Lean theorem prover, the system performs a "random walk" on existing proof graphs. By taking a known theorem and moving several steps away to find a related but more distant statement, the system can reverse the path to create a new, harder problem with a valid proof skeleton.

2. Tool-Integrated Reasoning (TiR)

Rather than requiring a full, formal Lean proof, the model is prompted to write a Python-based verification function. This function is tested against $N$ random trials. If it holds true for all samples, the proof is considered "validated" via a probabilistic filter, allowing for faster verification than full formalization.

3. Contrastive Alignment (SimPO)

The team creates pairs of "preferred" (correct) and "rejected" (incorrect) proofs to fine-tune the model using Simple Preference Optimization (SimPO). This objective encourages the model to assign a higher probability to correct reasoning paths while maintaining a specific confidence margin over incorrect ones.

Experimental Results

The methodology was tested on the Qwen-2.5 family of models, ranging from 0.5B to 7B parameters, using the MiniF2F benchmark for formal mathematics.

Performance comparison of aligned Qwen-2.5 models versus baselines on the MiniF2F benchmark.

As shown in the results, the "Aligned" models consistently outperformed the baselines. The smallest model (0.5B parameters) saw the most significant relative improvement, with its success rate jumping from 0.14 to 0.22 (a ~57% increase). The 1.5B and 7B models also showed marked improvements, reaching success rates of 0.37 and 0.53, respectively.

Key Insights and Discussion

  • Efficiency of Scale: The substantial gains in the 0.5B model suggest that high-quality synthetic data and alignment can compensate for lower parameter counts in specialized tasks.
  • Probabilistic Validation: Using Python-based sampling (TiR) as a filter allows for the verification of conjectures at a scale much faster than full formalization in Lean would permit.
  • Current Limitations: The TiR approach offers probabilistic rather than absolute guarantees. Furthermore, the current synthetic dataset is skewed toward algebra and number theory; future work is required to expand these methods into geometry and combinatorics.

Conclusion

This work demonstrates that combining structured synthetic data generation with contrastive alignment significantly improves the mathematical proving capabilities of LLMs. By integrating formal tools into the reasoning process, researchers can create a curriculum of increasingly difficult problems that push the boundaries of automated mathematical reasoning.

Summary updated

Abstract

Modern mathematical reasoning benchmarks primarily focus on answer finding rather than proof verification, creating a gap in evaluating the proving capabilities of large language models (LLMs). We present a methodology for generating diverse mathematical proof tasks using formal tools. Our approach combines Lean-based synthetic problem generation with a novel Tool-Integrated Reasoning (TiR) framework for partial proof validation, and it uses contrastive preference optimization to align the model's proof outputs. Experiments on the Qwen-2.5 family of models demonstrate meaningful improvements in mathematical reasoning, particularly for smaller models. Our aligned models achieve up to a 57% higher success rate than baselines on the MiniF2F benchmark (across 0.5B, 1.5B, and 7B parameter models). These results highlight the potential of synthetic data and integrated validation for advancing LLM-based mathematical reasoning.
BibTeX
@misc{obozov2025synthetic,
  author = {Mark Obozov and Michael Diskin and Aleksandr Beznosikov and Alexander Gasnikov and Serguei Barannikov},
  title = {{Synthetic Proofs with Tool-Integrated Reasoning: Contrastive Alignment for LLM Mathematics with Lean}},
  year = {2025},
  url = {https://openreview.net/forum?id=8uDKpvMg5h},
  howpublished = {OpenReview},
}
Figure