TL;DR
ProofDiff is a new system designed to compare the logical structure of informal mathematical proofs, particularly those generated or revised by AI. By constructing "argument maps" that capture claims and dependencies, it allows researchers to audit whether two documents follow the same mathematical route, even when their wording, notation, or level of detail differs significantly.
What Problem Does This Paper Address?
As generative AI becomes a common tool in mathematical research, we face a growing challenge: how do we verify that an AI-revised proof is actually the same argument?
Standard text-comparison tools (like git diff) operate on prose; they can tell you if words changed, but they can't tell you if the underlying logic shifted. Conversely, formal verification tools (like Lean or Coq) can confirm a proof is correct, but they don't necessarily reveal if two correct proofs use the same strategy. If an AI "cleans up" a proof, did it just fix the grammar, or did it subtly swap a constructive argument for a non-constructive one? ProofDiff fills this gap by focusing on the "mathematical route"—the specific sequence of hypotheses, lemmas, and inferences that lead to a conclusion.
Methods
ProofDiff uses a multi-stage pipeline to move from raw text to a structured comparison:
- Ingestion & Grounding: The system accepts TeX or PDF files, parsing them into structural blocks while maintaining links back to the original source lines.
- Node Construction: Using an LLM (the paper uses GPT-5.5), ProofDiff identifies "argument units"—theorems, lemmas, definitions, and individual proof steps. Each node is enriched with a semantic description of its role and mathematical content.
- Relation Discovery: The system identifies how these nodes connect. It classifies relations into types like
proves,uses_result, orunfolds_definition. This creates a directed graph called an Argument Map. - Cross-Document Alignment: ProofDiff compares the maps of two different documents. It proposes "candidate alignments" between nodes or groups of nodes. Crucially, it can handle many-to-one mappings, where several detailed steps in one proof are compressed into a single concise step in another.
- Human-in-the-Loop Review: The output is presented in an interactive browser interface (
flow.html). Reviewers can inspect the AI’s rationales, adjust alignments, and maintain an auditable history of the comparison.
Datasets
The authors evaluated ProofDiff on a "Toy proof inputs" set consisting of eight AI-generated informal proofs. These were generated using models like Claude Opus 5, Gemini 3.1 Pro, and GPT-5.6 Sol. They focused on four illustrative pairs:
- Infinitude of Primes: Testing preservation under complete reformulation.
- Cauchy–Schwarz: Comparing two different strategies (discriminant vs. normalization).
- Uncountability of $\mathbb{R}$: Comparing different mathematical machinery (diagonal argument vs. measure theory).
- Mean Value Theorem: Testing alignment across different levels of granularity (detailed vs. concise).
Key Insights and Ideas
- Argument over Prose: The core philosophy is that the "route" matters more than the wording. ProofDiff can align a proof even if every sentence has been rewritten, provided the logical dependencies remain the same.
- Handling Granularity: One of the most impressive features is the ability to recognize "grouped compression." In the Mean Value Theorem test, ProofDiff successfully mapped five distinct argument units from a detailed proof to a single concise node in the revised version.
- Strategy Sensitivity: When comparing proofs with different strategies (like the Cauchy-Schwarz example), the system correctly aligns the shared theorem statement but leaves the method-specific lemmas "unresolved." This highlights exactly where the two arguments diverge.
- Auditable AI: Rather than providing a black-box "similarity score," ProofDiff provides a textual rationale for every proposed link, allowing a human to judge if the AI's correspondence makes mathematical sense.
Key Results
The system demonstrated high feasibility across its test cases:
- Infinitude of Primes: Successfully aligned all 4 major components with high confidence ($c \ge 0.75$), despite the proofs being generated by different models (Claude vs. Gemini).
- Mean Value Theorem: Handled a complex many-to-one alignment, linking secant-slope notation and auxiliary function parts in the original to a single "reduction" step in the concise version.
- Uncountability of $\mathbb{R}$: Corrected for different formal labels, matching an "original corollary" to a "final theorem" because they shared the same argumentative role.
Limitations and Future Work
The authors note that this is an early-stage implementation. Key limitations include:
- Small Scale: The evaluation was limited to short, AI-generated proofs. Performance on long, human-written mathematical papers remains to be seen.
- LLM Dependency: The accuracy of the argument maps depends on the underlying LLM's ability to understand mathematical roles. Errors in node extraction can cascade into the alignment phase.
- No Correctness Check: ProofDiff compares arguments, not truth. It can tell you if two proofs are the same, but not if both are logically flawed.
- Future Directions: The team aims to incorporate "structural agreement" (graph topology) into the alignment scoring and to test the system on a wider variety of human-authored mathematical literature.
Summary updated