h1: A Taste of the Bitter Lesson in Automated Theorem Proving

Skander Moalla, Ahmad Rammal, Niket Patel, Remi Munos, Julia KempearXiv 2026 · arXiv

TL;DR

The "Bitter Lesson" of AI—that general methods scaling with compute eventually outperform human-engineered domain knowledge—has arrived for automated theorem proving. The authors introduce h1, a simple agentic loop that eschews complex, human-designed workflows in favor of a streamlined interface to the Lean 4 proof assistant and test-time scaling. h1 achieves state-of-the-art results on major benchmarks and settles eight previously open mathematical problems, demonstrating that a single, persistent attempt can often outperform specialized multi-agent systems.

What Problem Does This Paper Address?

Formal theorem proving is the "hard mode" of mathematics, requiring proofs to be written in code (like Lean 4) that can be mechanically verified. As frontier Large Language Models (LLMs) have improved, researchers have built increasingly complex "harnesses" around them—systems that enforce specific planning stages, decompose problems into sub-lemmas, or orchestrate multiple specialized agents.

The authors ask a fundamental question: Are these complex, human-designed workflows actually necessary? Or are their gains simply a byproduct of using stronger models and more compute? By testing whether a simple, unstructured loop can match or beat these sophisticated systems, the paper explores whether the "Bitter Lesson" applies to the design of theorem-proving agents.

Methods

The authors propose h1, a "simple harness" that provides a model with an efficient interface to Lean 4 without imposing a specific search or decomposition structure.

The h1 loop consists of three main stages:

  1. Proof Search (ReAct): The model uses a multi-turn "Reasoning and Acting" loop to edit a persistent working file, call search tools (like Mathlib search), and receive immediate compiler feedback.
  2. Validation: An external validator (SafeVerify) checks the final submission for correctness and ensures no "cheating" (like using sorry placeholders) occurred.
  3. Reflection: If validation fails, a separate "Reflection LLM" compacts the entire interaction transcript into a concise summary of verified progress and lessons learned. This summary becomes the "memory" for the next iteration, preventing the model from repeating mistakes.
The h1 harness architecture: a simple loop of reasoning, tool use, validation, and reflection.

Crucially, h1 focuses on interface efficiency. It uses targeted file edits rather than full-proof regeneration and leverages heavy caching to make long-horizon reasoning affordable.

Datasets

The authors evaluated h1 across a wide range of benchmarks:

  • PutnamBench: 672 problems from the prestigious Putnam undergraduate competition.
  • FATE-X: 100 algebra problems at the PhD qualifying-exam level.
  • FormalConjectures100-Solved: A subset of research-level conjectures with known solutions.
  • IMO 2026: Problems from the International Mathematical Olympiad.
  • Open Problems: 236 targets from the Formal Conjectures dataset that were previously unsolved.

Key Insights and Ideas

  • The Bitter Lesson Reaffirmed: Simple scaling Pareto-dominates specialized systems. When using the same underlying model, h1 often achieves higher solve rates at lower costs than complex systems like Goedel-Architect or AlphaProof Nexus.
  • Interface over Workflow: The authors found that how the model interacts with the environment matters more than the structure of its thoughts. Features like persistent workspaces, immediate compiler feedback, and efficient library search are the primary drivers of success.
  • Depth beats Breadth: In theorem proving, a single long, persistent attempt (pass@1 with high budget) is often more efficient than many independent short attempts (pass@k). This challenges the common practice of "repeated sampling" as the primary way to scale test-time compute.
  • Reflection Prevents "Context Rot": Without the reflection step to compact history, long agentic runs eventually suffer from performance degradation as the context window fills with irrelevant trial-and-error.

Key Results

  • State-of-the-Art Performance: Using GPT-5.5, h1 solved 670 out of 672 problems on PutnamBench. On FATE-X, it solved 90/100 problems, surpassing specialized provers.
  • Settling Open Problems: h1 successfully proved eight open problems in the Formal Conjectures dataset, including variants of Erdős conjectures and OEIS sequences.
  • Efficiency Gains: On a 100-problem PutnamBench subset, h1 with Gemini 3 Flash significantly outperformed AxProverBase (a more complex framework) by solving 69 problems compared to 18 at a matched $20 budget.
h1 Pareto-dominates specialized harnesses. On the right, h1 overtakes other loops on a PutnamBench subset through more efficient tool use and caching.

Limitations and Future Work

  • Single-File Constraint: h1 currently operates on a single Lean file. Proving massive theorems (like Fermat’s Last Theorem) that require managing thousands of lines across multiple files will likely require new forms of dependency management.
  • Black-Box Models: The authors acknowledge that because they use proprietary frontier models, it is difficult to perfectly account for potential data leakage during the models' pre-training.
  • Human-Designed Tools: While the workflow is simple, the tools (like search engines for Mathlib) are still human-designed. The authors suggest that in the future, models might even design and deploy their own custom harnesses for specific problems.

Summary updated

Abstract

While progress in informal mathematics has come largely from scaling the chain-of-thought and agentic reasoning of frontier language models, formal theorem proving has increasingly relied on human-designed workflows around these models, such as planning, decomposition, and orchestrating specialized provers. This raises a bitter-lesson question: can a simple agentic loop that scales test-time compute match these workflows? The answer matters as formal verification must scale to keep up with frontier models producing research mathematics, yet new workflows rarely settle it, as their gains are often confounded with stronger models or more compute. We study this question with h1, a simple harness that gives a model an efficient interface to Lean 4 and keeps it working through verifier feedback and self-reflection, without imposing a decomposition or search structure. h1 reaches state-of-the-art results on PutnamBench, FATE-X, and FormalConjectures100-Solved, settles eight open problems in FormalConjectures, and, at matched models, matches or outperforms specialized workflows such as Goedel-Architect and AlphaProof Nexus, surprisingly showing that a single attempt (pass@1) can often scale to solve open problems. Through ablations and matched-model comparisons with Claude Code and AxProverBase, we find that simplicity alone is not enough: an efficient Lean interface largely determines cost and solve rate. We thus recommend that future work evaluate frontier models with efficient execution feedback before prescribing workflows, at the risk of bitter lessons.
BibTeX
@misc{moalla2026h,
  author = {Skander Moalla and Ahmad Rammal and Niket Patel and Remi Munos and Julia Kempe},
  title = {{h1: A Taste of the Bitter Lesson in Automated Theorem Proving}},
  year = {2026},
  eprint = {2610.35962},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.35962},
}
Figure