TL;DR
Researchers have developed a "decomposition-guided" pipeline to verify if AI-generated formal math (Lean 4) actually matches the original natural language problem. By breaking down informal statements into structured logic and using a hybrid of symbolic rules and Large Language Models (LLMs), the system can not only detect errors but also fix them and categorize exactly what went wrong.
What Problem Does This Paper Address?
Autoformalization—the process of translating natural language mathematics into machine-verifiable code like Lean 4—is a "holy grail" for reliable AI reasoning. However, even if an AI generates code that compiles, that doesn't mean the code is correct. It might have swapped a "less than" for a "greater than," missed a constraint, or misinterpreted a quantifier.
Current evaluation methods usually provide a simple "pass/fail" or a similarity score, which offers zero diagnostic value. If the AI is wrong, we need to know how it's wrong. This paper addresses the challenge of Semantic Alignment Evaluation: ensuring the formal code faithfully preserves the meaning of the original math, and providing a fine-grained "medical report" (using the 28-category SCI taxonomy) when it doesn't.
Methods
The authors propose a four-stage pipeline powered by the Qwen3-235B model:
- Informal Statement Decomposition: Instead of comparing raw text to code, the system first translates the informal math into a structured intermediate representation. It extracts "Conditions" (premises/hypotheses) and a "Conclusion" (the goal). This bridges the gap between messy human language and rigid code.
- Verdict Prediction: The LLM compares the structured decomposition against the candidate Lean 4 code to issue a binary "aligned" or "misaligned" verdict, accompanied by a natural language explanation.
- Correction Generation & Segmentation: If misaligned, the model generates a "minimally modified" version of the code that fixes the error. By "diffing" the original and corrected code, the system automatically localizes the exact segment where the error occurred.
- Hybrid SCI Error Classification: To categorize the error (e.g., "Quantifier Strengthening" or "Function Confusion"), the system uses a two-pronged approach:
- Rule-Based: A deterministic "fast-path" that catches obvious symbolic swaps (like $\forall$ vs $\exists$ or
sinvscos) using a lookup table. - Model-Based: For complex semantic errors, the LLM analyzes the reasoning and the code diff to pick the right category from the 28-option taxonomy.
Datasets
The system was evaluated on the FormalRx benchmark, which was released as part of the AI4Math @ ICML 2026 Track 1 Challenge. This benchmark is specifically designed for diagnostic evaluation of autoformalization, requiring participants to provide verdicts, corrections, error localizations, and classifications.
Key Insights and Ideas
- The "Coupling" Paradox: Initially, one might think that breaking a task into many small, independent steps (heavy decomposition) is better. However, the authors found that tightly coupled reasoning—where the model predicts the verdict and generates the correction in a more integrated flow—performed significantly better. Why? Because in multi-stage pipelines, an error in the first step cascades and ruins every subsequent step.
- Decomposition as a Diagnostic Tool: Using decomposition wasn't just about making the translation easier; it served as a "semantic reference." By forcing the LLM to write out the logic in plain terms first, it became much better at spotting subtle discrepancies in the formal code.
- Symbolic Heuristics are Still King: Despite the power of 235B-parameter models, simple "if-then" rules for token swaps (like changing
Nat.gcdtoNat.lcm) provided a reliable boost in classification accuracy. LLMs can sometimes overthink simple symbolic errors; the rule-based system keeps them grounded.
Key Results
The system was tested on the official Codabench evaluation server for the AI4Math challenge:
- Overall Score: 0.3397 (The "Single-stage + rule-based" configuration was the top performer).
- Verdict F1: 0.7327, indicating high reliability in simply identifying if a formalization is right or wrong.
- Correction Accuracy: 0.7671, showing the model is very capable of fixing errors once they are identified.
- The "Multi-stage" Penalty: Moving from a single-stage to a multi-stage pipeline caused the overall score to plummet from ~0.34 to ~0.09, highlighting the severity of error propagation in complex reasoning tasks.
Limitations and Future Work
While the system is excellent at fixing code and issuing verdicts, fine-grained categorization remains the hardest part of the puzzle (with a Categorization F1 of only 0.22). The 28-category SCI taxonomy is highly nuanced, and even expert humans can struggle to distinguish between similar error types.
The authors suggest that future work should involve "verifier-guided" correction. Currently, the system doesn't actually run the Lean 4 compiler to see if its "fix" works. Integrating the Lean prover directly into the feedback loop could allow the system to iterate on its corrections until they are mathematically sound.
Summary updated