TL;DR
AI agents don't approach mathematical proofs the way humans do; instead of tackling the conceptual core, they act as "autofillers" attracted to explicit gaps and peripheral tasks. By replaying the history of major Lean formalization projects, researchers found that AI agents systematically avoid the complex intermediate lemmas where human mathematical synthesis happens, tending instead to "regularize" unusual proofs into more generic forms.
What Problem Does This Paper Address?
The rise of AI in mathematics has led to a "proof abundance," but finding a proof is not the same as understanding it. In formal verification systems like Lean, humans decompose complex theorems into subproblems. While AI agents are increasingly capable of solving these subproblems, we know very little about their "preferences"—what they choose to work on when given the freedom to "take the next step."
If AI agents and human mathematicians have fundamentally different priorities, collaboration could become a source of friction. AI might produce verbose, "monolithic" code that is hard for humans to audit, or it might steer human effort toward "AI-friendly" styles, potentially stifling mathematical innovation and diversity.
Methods
The authors introduce a methodology called Simulated Collaboration. They took the Git histories of nine major human-led Lean projects. For every human commit ($H_{t-1}$), they prompted three state-of-the-art AI agents (Claude Opus 4.6, GPT-5.2-Codex, and Gemini 3.1 Pro) to "take the next step" in the project.
To analyze the differences, they extracted several features from the resulting code:
- Syntactic: Presence of
sorrymarkers (placeholders for unproven steps) and definition length. - Semantic: The definition's position in the dependency graph (distance to "leaves" like axioms or "roots" like final theorems).
- Information-Theoretic: "Surprise" (per-token log-probability), which measures how unusual a proof is relative to the model's training distribution.
Datasets
The study analyzed nine significant Lean 4 formalization repositories, totaling 4.7 million definitions across more than 6,000 commits. These include high-profile projects like:
- flt: Fermat’s Last Theorem project.
- pfr: The Polynomial Freiman–Ruzsa conjecture.
- sphere: Sphere packing in 8 dimensions.
- zeta: The irrationality of $\zeta(3)$.
Key Insights and Ideas
- The "Autofill" Instinct: AI agents treat the
sorrykeyword as a literal command to fill a gap. While humans usesorryas a planning device to defer laborious but routine steps, AI agents are overwhelmingly drawn to them, often ignoring the surrounding context that a human would consider essential for a coherent argument. - Avoiding the "Marrow": Humans spend most of their time in the "thick of the action"—the intermediate lemmas that connect basic axioms to high-level theorems. AI agents show a "peripheral bias," preferring to work either at the very bottom (axioms) or the very top (final theorem boundaries), leaving the difficult conceptual synthesis in the middle to humans.
- Proof Homogenization: AI agents are attracted to "surprising" or idiosyncratic proof bodies but modify them to reduce that surprise. They act as "proof regularizers," smoothing out unusual mathematical styles toward a more generic, "LLM-standard" baseline.
- Monolithic vs. Granular: Human commits tend to be small, targeted, and interconnected. AI edits are significantly larger and more isolated, suggesting they treat the codebase as a fixed scaffold to be completed rather than a work-in-progress to be refined.
Key Results
- Syntactic Focus: Definitions containing a
sorrymarker accounted for 59–92% of AI modifications, compared to only 19% for humans. - Edit Size: The median number of characters modified by humans was 20, while AI agents modified between 84 and 150 characters per edit.
- Distance to Gaps: As shown in the figure below, the probability of an AI agent modifying a definition drops sharply as the distance from a
sorrymarker increases, a much stronger decline than seen in humans.
- Surprise Reduction: AI agents preferentially targeted high-surprise proof bodies and significantly lowered their surprise levels, effectively "standardizing" the code (see Figure 4 in the paper).
Limitations and Future Work
- Heuristic Parsing: The researchers used a custom lexical parser rather than the Lean compiler itself because much of the work-in-progress code (and AI-generated code) does not yet compile. While validated, this parser is an approximation of the true dependency graph.
- Data Leakage: Since the repositories are public, they likely appeared in the models' training data. However, the authors argue this would likely make AI behavior more like the human ground truth, meaning the observed divergences are likely a conservative lower bound.
- Causal vs. Correlational: The regression analysis is correlational. Future work could involve direct interventions (e.g., adding/removing
sorrymarkers) to establish causal links in agent behavior. - Human Agency: The study treats "the human" as a single average actor, though the repositories had up to 61 contributors with diverse styles.
Summary updated