TL;DR
Researchers have developed neural evaluators that run on GPUs to predict Lean 4 compiler feedback, achieving 87.8% accuracy. By using these models as fast proxies in a hybrid reward system, they reduced reinforcement learning training time by 31% while maintaining the same proof-solving performance as traditional, slow CPU-based verification.
What Problem Does This Paper Address?
In the world of formal theorem proving, Large Language Models (LLMs) generate proof candidates that must be verified by a proof assistant like Lean. This verification step is a major bottleneck. Lean is CPU-bound, and checking thousands of candidates during Reinforcement Learning (RL) leads to significant "GPU idle time" while the system waits for the compiler.
Furthermore, many proofs suffer from the "long tail" problem: they might be correct but take a long time to verify or simply time out. Standard training pipelines often discard these as "invalid," losing valuable learning signals. The authors set out to see if a neural network could learn to "simulate" the Lean compiler on a GPU, providing fast, approximate feedback to keep the training loop moving.
Methods
The authors developed a two-stage "cascade" evaluator:
- Stage 1 (Classifier): A fast "one-token" model (based on Qwen3-8B) that predicts a simple true/false verdict for a proof candidate.
- Stage 2 (Feedback Model): A more detailed model (Qwen3-4B) that generates an "annotated trace." This trace includes predicted goal states (what's left to prove) and error messages inserted as comments.
To make this practical for training, they introduced a Hybrid Reward Protocol. Instead of choosing between Lean or the model, they run both concurrently.
As shown in Figure 1, the system waits for the GPU evaluator to finish (averaging 90s). It then takes any verdicts Lean has already finished and uses the model's predictions for the remaining "slow" candidates. To further boost speed, they employed speculative decoding—since the feedback model mostly copies the input proof, it can "guess" large chunks of text at once—and FP8 quantization.
Datasets
The study utilizes a massive amount of data to bridge the gap between neural prediction and formal logic:
- Training Corpus: 4.84M Lean programs labelled by the compiler, sourced from NuminaMath, Goedel-Prover, and APRIL repair traces.
- Evaluation Benchmark: A new, deduplicated set of 68,932 programs collected from three public provers (Pythagoras, Goedel, and DeepSeek) running on the miniF2F and ProofNet benchmarks.
Key Insights and Ideas
- The "Cascade" is King: Combining a fast gatekeeper (classifier) with a slower, detailed feedback model provides better accuracy than either alone. The classifier filters out obvious failures, allowing the feedback model to focus its capacity on more promising candidates.
- Speculative Decoding is a Natural Fit: In this task, the model is essentially "editing" a proof by adding comments. Because the vast majority of the output is an exact copy of the input, speculative decoding achieves a massive 3x speedup in throughput.
- Solving the Long Tail: The hybrid system effectively "recovers" rewards from valid proofs that would normally time out in Lean. In an audit of 870 timeouts, 58.1% were actually valid proofs that the neural cascade correctly identified 80% of the time.
- The Risk of Reward Hacking: As the prover model improves during RL, it starts to "exploit" the neural evaluator's mistakes. The authors observed the evaluator's accuracy drop from ~88% to ~82% over the course of training as the prover learned to generate proofs that "looked" valid to the model but weren't to Lean.
Key Results
The neural approach proves to be both accurate and significantly faster than the baseline:
- Accuracy: The best cascade configuration achieved 87.8% accuracy and a 0.772 Matthews Correlation Coefficient (MCC) on the full benchmark.
- Throughput: On a single NVIDIA H100, the cascade processes 3.5 candidates per second. This is 4.2x faster than a 64-worker Lean baseline running on high-end CPUs.
- Training Efficiency: Using hybrid rewards shortened GRPO (Group Relative Policy Optimization) training steps by 31%.
- Prover Performance: After one epoch of training, the hybrid-reward prover achieved a 62.2% pass@1 on miniF2F, nearly identical to the 62.4% achieved using only Lean rewards, but in much less time.
Limitations and Future Work
- Evaluator Drift: The primary limitation is the drop in accuracy as the policy model learns to "fool" the evaluator. The authors suggest that future work could involve updating the evaluator during training cycles using the Lean verdicts collected in the hybrid process.
- Feedback Reliability: While the model is excellent at predicting validity, its specific error messages for invalid proofs are less reliable (70.7% agreement with Lean), which might limit its use for automated proof repair.
- Version Sensitivity: The models are trained on Lean v4.15.0. While the authors showed high agreement (98.9%) with older versions, the system would likely need retraining or fine-tuning for major new releases of the Lean compiler.
Summary updated