Technical Report for AI4Math-2026 Track 1: Automated Semantic Alignment Verification and Error Categorization of Lean 4 Formalizations via Decomposition-Guided Auditing

Huan Vu, cuong tien nguyen, Thien Van LuongAI4Math @ ICML 2026 · OpenReview

TL;DR

Researchers have developed a "decomposition-guided" pipeline to verify if AI-generated formal math (Lean 4) actually matches the original natural language problem. By breaking down informal statements into structured logic and using a hybrid of symbolic rules and Large Language Models (LLMs), the system can not only detect errors but also fix them and categorize exactly what went wrong.

What Problem Does This Paper Address?

Autoformalization—the process of translating natural language mathematics into machine-verifiable code like Lean 4—is a "holy grail" for reliable AI reasoning. However, even if an AI generates code that compiles, that doesn't mean the code is correct. It might have swapped a "less than" for a "greater than," missed a constraint, or misinterpreted a quantifier.

Current evaluation methods usually provide a simple "pass/fail" or a similarity score, which offers zero diagnostic value. If the AI is wrong, we need to know how it's wrong. This paper addresses the challenge of Semantic Alignment Evaluation: ensuring the formal code faithfully preserves the meaning of the original math, and providing a fine-grained "medical report" (using the 28-category SCI taxonomy) when it doesn't.

Methods

The authors propose a four-stage pipeline powered by the Qwen3-235B model:

  1. Informal Statement Decomposition: Instead of comparing raw text to code, the system first translates the informal math into a structured intermediate representation. It extracts "Conditions" (premises/hypotheses) and a "Conclusion" (the goal). This bridges the gap between messy human language and rigid code.
  2. Verdict Prediction: The LLM compares the structured decomposition against the candidate Lean 4 code to issue a binary "aligned" or "misaligned" verdict, accompanied by a natural language explanation.
  3. Correction Generation & Segmentation: If misaligned, the model generates a "minimally modified" version of the code that fixes the error. By "diffing" the original and corrected code, the system automatically localizes the exact segment where the error occurred.
  4. Hybrid SCI Error Classification: To categorize the error (e.g., "Quantifier Strengthening" or "Function Confusion"), the system uses a two-pronged approach:
  5. Rule-Based: A deterministic "fast-path" that catches obvious symbolic swaps (like $\forall$ vs $\exists$ or sin vs cos) using a lookup table.
  6. Model-Based: For complex semantic errors, the LLM analyzes the reasoning and the code diff to pick the right category from the 28-option taxonomy.

Datasets

The system was evaluated on the FormalRx benchmark, which was released as part of the AI4Math @ ICML 2026 Track 1 Challenge. This benchmark is specifically designed for diagnostic evaluation of autoformalization, requiring participants to provide verdicts, corrections, error localizations, and classifications.

Key Insights and Ideas

  • The "Coupling" Paradox: Initially, one might think that breaking a task into many small, independent steps (heavy decomposition) is better. However, the authors found that tightly coupled reasoning—where the model predicts the verdict and generates the correction in a more integrated flow—performed significantly better. Why? Because in multi-stage pipelines, an error in the first step cascades and ruins every subsequent step.
  • Decomposition as a Diagnostic Tool: Using decomposition wasn't just about making the translation easier; it served as a "semantic reference." By forcing the LLM to write out the logic in plain terms first, it became much better at spotting subtle discrepancies in the formal code.
  • Symbolic Heuristics are Still King: Despite the power of 235B-parameter models, simple "if-then" rules for token swaps (like changing Nat.gcd to Nat.lcm) provided a reliable boost in classification accuracy. LLMs can sometimes overthink simple symbolic errors; the rule-based system keeps them grounded.

Key Results

The system was tested on the official Codabench evaluation server for the AI4Math challenge:

  • Overall Score: 0.3397 (The "Single-stage + rule-based" configuration was the top performer).
  • Verdict F1: 0.7327, indicating high reliability in simply identifying if a formalization is right or wrong.
  • Correction Accuracy: 0.7671, showing the model is very capable of fixing errors once they are identified.
  • The "Multi-stage" Penalty: Moving from a single-stage to a multi-stage pipeline caused the overall score to plummet from ~0.34 to ~0.09, highlighting the severity of error propagation in complex reasoning tasks.

Limitations and Future Work

While the system is excellent at fixing code and issuing verdicts, fine-grained categorization remains the hardest part of the puzzle (with a Categorization F1 of only 0.22). The 28-category SCI taxonomy is highly nuanced, and even expert humans can struggle to distinguish between similar error types.

The authors suggest that future work should involve "verifier-guided" correction. Currently, the system doesn't actually run the Lean 4 compiler to see if its "fix" works. Integrating the Lean prover directly into the feedback loop could allow the system to iterate on its corrections until they are mathematically sound.

Summary updated

Abstract

We present a semantic alignment verification pipeline for Lean 4 autoformalization developed for the AI4Math @ ICML 2026 FormalRx Challenge. Given an informal mathematical statement and a candidate Lean formalization, the system predicts semantic alignment, generates corrected formal statements, localizes erroneous segments, and classifies errors using the 28-category SCI taxonomy. Our approach combines structured decomposition of informal mathematics, LLM-based semantic auditing, correction generation, and a hybrid SCI classifier integrating deterministic token substitution rules with model-based classification. Experiments on the official FormalRx benchmark show that our best system achieves an Overall score of $0.3397$, with strong Verdict F1 ($0.7327$) and Correction accuracy ($0.7671$). Results indicate that tightly coupled semantic reasoning and correction generation outperform heavily decomposed multi-stage pipelines, while symbolic heuristics provide effective complementary signals for SCI categorization.
BibTeX
@inproceedings{vu2026technical,
  author = {Huan Vu and cuong tien nguyen and Thien Van Luong},
  title = {{Technical Report for AI4Math-2026 Track 1: Automated Semantic Alignment Verification and Error Categorization of Lean 4 Formalizations via Decomposition-Guided Auditing}},
  booktitle = {AI4Math @ ICML},
  year = {2026},
  url = {https://openreview.net/forum?id=FqYFz767mp},
}
Figure