Advancing Mathematical Research via Human-AI Interactive Theorem Proving

Chenyi Li, Zhijian Lai, Dong An, Jiang Hu, Zaiwen WenarXiv 2025 · arXiv · OpenReview

Introduction

While Large Language Models (LLMs) have demonstrated remarkable proficiency in solving competition-level mathematics, frontier mathematical research presents a unique set of challenges. Research at the cutting edge requires long-horizon coherence, high levels of abstraction, and absolute logical rigor. Current AI systems often suffer from "hallucinations" or logical gaps when tackling unsolved problems. Conversely, fully automated formal verification tools, such as Lean, possess steep learning curves that can hinder rapid exploration.

This research introduces a structured, human-in-the-loop workflow designed to bridge this gap. By transforming LLMs from simple chatbots into sophisticated research partners, the authors demonstrate how human intuition can be combined with AI-driven "prove-or-disprove" branches and numerical verification to accelerate mathematical discovery.

The Human-AI Research Pipeline

The core of this approach is a four-stage interactive pipeline that leverages the creative, broad-search capabilities of LLMs while maintaining mathematical soundness through human oversight and numerical checks.

Overview of the AI-assisted research workflow, spanning conceptualization, proving, and discovery.
  1. Topic Conceptualization: The process begins with a human providing a high-level idea. The LLM then maps the relevant mathematical domain, identifies key operators, and proposes concrete research directions.
  2. Goal-Guided Proving: For known targets, structured prompts force the LLM into a "line-by-line" reasoning mode, requiring it to cite standard theorems and avoid logical leaps.
  3. Open-Ended Discovery: When the conclusion is unknown, the workflow employs a "prove-or-disprove" protocol. The AI runs parallel branches: one searching for a constructive proof and another actively hunting for counterexamples.
  4. Property-Constrained Synthesis: In this "generator-tester" loop, the human defines the necessary properties of a mathematical object. The AI generates candidates and writes Python code to numerically verify them before any formal proof is attempted.

Key Insights and Ideas

The researchers identified several critical advantages to this interactive methodology:

  • Mining "Rich Traces": LLMs often produce auxiliary lemmas or invariants during the reasoning process that they may not initially flag as significant. Human experts can mine these traces to find crucial insights, such as the 2D invariant "Grover plane" discovered in this study.
  • Numerical Sanity Checks: By tasking the AI with writing its own verification code, researchers can quickly discard false conjectures. This prevents hours of wasted effort on proving statements that fail simple numerical tests.
  • Cross-Domain Lifts: LLMs excel at adapting concepts between fields—for instance, suggesting how a convergence proof from Euclidean space might be adapted to the curved geometry of a unitary manifold.

Case Study: Quantum Search and Riemannian Optimization

The workflow was validated through a case study on the intersection of Riemannian optimization and Grover’s quantum search algorithm. This collaboration led to several concrete mathematical outcomes:

  • Discovery of Length-5 Retraction: The AI successfully navigated a complex combinatorial search to identify a specific product of five exponential terms satisfying the requirements for a Riemannian retraction.
  • Exponential Speedup: While standard worst-case analysis suggested a sublinear convergence rate of $O(\epsilon^{-2})$, the interactive process helped identify that the problem satisfied the Polyak-Łojasiewicz (PL) inequality. This led to the proof of a much faster, "problem-specific" exponential convergence rate of $O(\log(1/\epsilon))$.
  • Operational Efficiency: The AI effectively handled routine proof work and combinatorial searches, allowing human researchers to focus on high-level strategy and final verification.

Limitations and Future Directions

Despite its successes, the system remains an assistant rather than an "auto-mathematician." It heavily depends on human experts to filter trivial suggestions, catch subtle logical errors, and maintain overall rigor. LLMs still pose risks of fabricating results or citing non-existent literature, necessitating that all AI output be treated as raw material for human refinement.

Future work aims to integrate this natural language workflow with formal proof assistants like Lean or Coq. This would create a seamless pipeline from a vague human intuition to a machine-verified formal proof, further pushing the boundaries of automated mathematical research.

Summary updated

Abstract

We investigate how large language models can be used as research tools in scientific computing while preserving mathematical rigor. We propose a human-in-the-loop workflow for interactive theorem proving and discovery with LLMs. Human experts retain control over problem formulation and admissible assumptions, while the model searches for proofs or contradictions, proposes candidate properties and theorems, and helps construct structures and parameters that satisfy explicit constraints, supported by numerical experiments and simple verification checks. Experts treat these outputs as raw material, further refine them, and organize the results into precise statements and rigorous proofs. We instantiate this workflow in a case study on the connection between manifold optimization and Grover's quantum search algorithm, where the pipeline helps identify invariant subspaces, explore Grover-compatible retractions, and obtain convergence guarantees for the retraction-based gradient method. The framework provides a practical template for integrating large language models into frontier mathematical research, enabling faster exploration of proof space and algorithm design while maintaining transparent reasoning responsibilities. Although illustrated on manifold optimization problems in quantum computing, the principles extend to other core areas of scientific computing.
BibTeX
@misc{li2025advancing,
  author = {Chenyi Li and Zhijian Lai and Dong An and Jiang Hu and Zaiwen Wen},
  title = {{Advancing Mathematical Research via Human-AI Interactive Theorem Proving}},
  year = {2025},
  eprint = {2512.09443},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2512.09443},
}
Figure