Abstract
Large Language Models (LLMs) demonstrate strong mathematical problem-solving abilities but frequently fail on problems that deviate syntactically from their training distribution. We identify a systematic failure mode, syntactic blind spots, in which models misapply familiar reasoning strategies to problems that are semantically straightforward but phrased in unfamiliar ways. These errors are not due to gaps in mathematical competence, but rather reflect a brittle coupling between surface form and internal representation. To test this, we rephrase incorrectly answered questions using syntactic templates drawn from correct examples. These rephrasings, which preserve semantics while reducing structural complexity, often lead to correct answers. We quantify syntactic complexity using a metric based on Dependency Locality Theory (DLT), and show that higher DLT scores are associated with increased failure rates across multiple datasets. Our findings suggest that many reasoning errors stem from structural misalignment rather than conceptual difficulty, and that syntax-aware interventions can reveal and mitigate these inductive failures.
Introduction: The Brittleness of LLM Reasoning
Despite the impressive performance of Large Language Models (LLMs) on mathematical benchmarks, they remain notoriously brittle. A minor adjustment in word order or the introduction of a more complex sentence structure can cause a model to fail a problem it previously solved with ease. This research investigates whether these failures stem from a genuine lack of mathematical competence or a "syntactic induction" failure—a tendency for models to use syntactic similarity as a proxy for problem logic.
Models often suffer from "syntactic blind spots," where their internal representation is too tightly coupled to the surface-level phrasing of a question. When a new problem's syntax mimics a known template, the model may force-fit a previously learned solution onto the new logic, even if it is inappropriate. This creates a gap between the model's latent mathematical ability and its structural fluency.
Methodology: Quantifying and Mitigating Structural Misalignment
The researchers developed a three-step framework to diagnose and mitigate these failures:
- Quantifying Complexity with DLT: They employed Dependency Locality Theory (DLT), a cognitive science metric, to measure the "processing cost" of a sentence. DLT calculates complexity based on the distance between dependent words (Integration cost), the number of unresolved syntactic expectations (Storage cost), and the introduction of new entities (Discourse cost).
- Syntactic Matching: For questions a model failed, the authors searched for a "match"—a syntactically similar question that the model answered correctly. These matches were identified by comparing dependency trees using Weisfeiler-Lehman Graph Kernels (WLK).
- Dependency-Guided Rephrasing: Finally, they used LLaMA-3 to rewrite the failed "incorrect" question to match the simpler syntactic structure of the "match" question while strictly preserving the original mathematical semantics.
Results and Key Insights
The study evaluated five open-source LLMs (LLaMA, Mistral, Qwen, Gemma, and Granite) across four benchmarks: GSM8K, SVAMP, MultiArith, and ASDiv.
Syntax as a Proxy for Logic
The core finding is the formalization of syntactic induction. Models don't just "reason"; they pattern-match. If a problem's syntax mimics a known template, the model will often misapply the associated logic. This suggests that as the "syntactic burden" increases, the model's ability to access its underlying mathematical logic breaks down.
DLT as a Failure Predictor
A statistically significant correlation was found between high DLT scores (complex syntax) and model failure. This was particularly evident in narrative-heavy datasets like GSM8K, where all five models showed significantly higher DLT scores for incorrect answers compared to correct ones.
Latent Competence and Recovery
The intervention of rephrasing led to substantial accuracy gains across the board. For instance, Gemma-7B recovered 210 previously failed answers on the ASDiv dataset, and Granite-7B saw a 15.48% accuracy improvement on MultiArith. This proves that models possess the "latent competence" to solve the logic but lack the "structural fluency" to navigate complex phrasing.
Conclusion and Future Directions
These findings suggest that many reasoning errors are systematic misapplications of valid rules rather than random hallucinations. The study highlights a critical need to move beyond surface-level pattern matching toward more robust structural alignment. Future work should focus on developing syntactic diagnostic tools to identify "at-risk" questions and creating failure-aware training curricula that expose models to high-complexity syntactic variations to encourage better generalization.
Summary updated