ProofDiff: Auditable Argument-Level Comparison of AI-Generated Mathematical Proofs

Matan Tsafrir, Esdras Medeiros, Jorge Poco, Kevin Gomez, Eduardo V. Teixeira, Claudio SilvaarXiv 2026 · arXiv

TL;DR

ProofDiff is a new system designed to compare the logical structure of informal mathematical proofs, particularly those generated or revised by AI. By constructing "argument maps" that capture claims and dependencies, it allows researchers to audit whether two documents follow the same mathematical route, even when their wording, notation, or level of detail differs significantly.

What Problem Does This Paper Address?

As generative AI becomes a common tool in mathematical research, we face a growing challenge: how do we verify that an AI-revised proof is actually the same argument?

Standard text-comparison tools (like git diff) operate on prose; they can tell you if words changed, but they can't tell you if the underlying logic shifted. Conversely, formal verification tools (like Lean or Coq) can confirm a proof is correct, but they don't necessarily reveal if two correct proofs use the same strategy. If an AI "cleans up" a proof, did it just fix the grammar, or did it subtly swap a constructive argument for a non-constructive one? ProofDiff fills this gap by focusing on the "mathematical route"—the specific sequence of hypotheses, lemmas, and inferences that lead to a conclusion.

Methods

ProofDiff uses a multi-stage pipeline to move from raw text to a structured comparison:

  1. Ingestion & Grounding: The system accepts TeX or PDF files, parsing them into structural blocks while maintaining links back to the original source lines.
  2. Node Construction: Using an LLM (the paper uses GPT-5.5), ProofDiff identifies "argument units"—theorems, lemmas, definitions, and individual proof steps. Each node is enriched with a semantic description of its role and mathematical content.
  3. Relation Discovery: The system identifies how these nodes connect. It classifies relations into types like proves, uses_result, or unfolds_definition. This creates a directed graph called an Argument Map.
  4. Cross-Document Alignment: ProofDiff compares the maps of two different documents. It proposes "candidate alignments" between nodes or groups of nodes. Crucially, it can handle many-to-one mappings, where several detailed steps in one proof are compressed into a single concise step in another.
  5. Human-in-the-Loop Review: The output is presented in an interactive browser interface (flow.html). Reviewers can inspect the AI’s rationales, adjust alignments, and maintain an auditable history of the comparison.
Figure 1: The automatic ProofDiff pipeline. Each document is processed independently before cross- document candidate alignment; the resulting maps and proposals are presented in flow.html.

Datasets

The authors evaluated ProofDiff on a "Toy proof inputs" set consisting of eight AI-generated informal proofs. These were generated using models like Claude Opus 5, Gemini 3.1 Pro, and GPT-5.6 Sol. They focused on four illustrative pairs:

  • Infinitude of Primes: Testing preservation under complete reformulation.
  • Cauchy–Schwarz: Comparing two different strategies (discriminant vs. normalization).
  • Uncountability of $\mathbb{R}$: Comparing different mathematical machinery (diagonal argument vs. measure theory).
  • Mean Value Theorem: Testing alignment across different levels of granularity (detailed vs. concise).

Key Insights and Ideas

  • Argument over Prose: The core philosophy is that the "route" matters more than the wording. ProofDiff can align a proof even if every sentence has been rewritten, provided the logical dependencies remain the same.
  • Handling Granularity: One of the most impressive features is the ability to recognize "grouped compression." In the Mean Value Theorem test, ProofDiff successfully mapped five distinct argument units from a detailed proof to a single concise node in the revised version.
  • Strategy Sensitivity: When comparing proofs with different strategies (like the Cauchy-Schwarz example), the system correctly aligns the shared theorem statement but leaves the method-specific lemmas "unresolved." This highlights exactly where the two arguments diverge.
  • Auditable AI: Rather than providing a black-box "similarity score," ProofDiff provides a textual rationale for every proposed link, allowing a human to judge if the AI's correspondence makes mathematical sense.
Figure 2: An annotated synthetic comparison rendered by the production flow.html interface. A1–A5 mark candidate alignments and D1–D5 mark within-document dependencies; the key explains each arc. The additional final lemma has no candidate correspondence and remains unresolved.

Key Results

The system demonstrated high feasibility across its test cases:

  • Infinitude of Primes: Successfully aligned all 4 major components with high confidence ($c \ge 0.75$), despite the proofs being generated by different models (Claude vs. Gemini).
  • Mean Value Theorem: Handled a complex many-to-one alignment, linking secant-slope notation and auxiliary function parts in the original to a single "reduction" step in the concise version.
  • Uncountability of $\mathbb{R}$: Corrected for different formal labels, matching an "original corollary" to a "final theorem" because they shared the same argumentative role.
Figure 3: Automatic ProofDiff flow view for the infinitude-of-primes pair with proof nodes expanded and confidence filter c ≥0.75. The original’s explicit prime definition is the only unresolved node.

Limitations and Future Work

The authors note that this is an early-stage implementation. Key limitations include:

  • Small Scale: The evaluation was limited to short, AI-generated proofs. Performance on long, human-written mathematical papers remains to be seen.
  • LLM Dependency: The accuracy of the argument maps depends on the underlying LLM's ability to understand mathematical roles. Errors in node extraction can cascade into the alignment phase.
  • No Correctness Check: ProofDiff compares arguments, not truth. It can tell you if two proofs are the same, but not if both are logically flawed.
  • Future Directions: The team aims to incorporate "structural agreement" (graph topology) into the alignment scoring and to test the system on a wider variety of human-authored mathematical literature.

Summary updated

Abstract

Mathematical understanding depends on the route from hypotheses to conclusion: how the parts of a proof function and which dependencies carry the argument forward. This structure can persist when wording or notation changes, while a small alteration to a hypothesis or inference can produce a different argument beneath similar prose. Formal verification can establish that a rewrite proves the same theorem, but not whether it follows the same mathematical route. Comparing informal proofs, especially those produced or revised by generative models, therefore requires reconstructing which claims, dependencies, and reasoning steps are preserved, reorganized, or replaced. We present ProofDiff, a system for comparing informal mathematical documents. It is designed to support human review, not to certify mathematical correctness. Given two documents, ProofDiff independently constructs a source-grounded argument map for each and proposes correspondences between the resulting maps. Each map captures the document's claims and reasoning steps and records the typed relations that connect them. An integrated human-review layer allows reviewers to revise the maps and correspondences while preserving an auditable history. Comparing the two argument maps provides a basis for inspecting whether the documents follow the same mathematical route and where their claims or dependencies differ. We demonstrate ProofDiff on four illustrative pairs of AI-generated proofs, ranging from a detailed proof and its concise revision to proofs using different mathematical strategies.
BibTeX
@misc{tsafrir2026proofdiff,
  author = {Matan Tsafrir and Esdras Medeiros and Jorge Poco and Kevin Gomez and Eduardo V. Teixeira and Claudio Silva},
  title = {{ProofDiff: Auditable Argument-Level Comparison of AI-Generated Mathematical Proofs}},
  year = {2026},
  eprint = {2611.10233},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.10233},
}
Figure