Standard language models treat mathematical equations and logical proofs as flat sequences of characters. However, mathematics is inherently hierarchical: tokens form terms, terms form expressions, and expressions form equations or propositions. A growing body of research suggests that for AI to truly "reason" rather than just match patterns, its internal architecture must mirror this tree-like structure.
The Structural Mismatch of Flat Sequences
When a model like a standard Transformer processes the equation $y = (a + b)^2$, it sees a linear string. In this view, the equals sign is just another token. But in a mathematical sense, the equals sign is often the "root" of a tree, governing the relationship between the entire left and right sides.
As Lei Wang and Felix Petersen argue in Translating Math Word Problem to Expression Tree and Towards Formula Translation using Recursive Neural Networks, treating math as a flat string ignores the logical grammar. This leads to "hallucinated" equations that might look correct locally but are syntactically or logically invalid globally. Without a structural inductive bias, models struggle to capture the long-range dependencies between an operator and its distant operands.
Encoding Formulas as Operator Trees
To solve this, researchers have moved toward representing formulas as Abstract Syntax Trees (ASTs) or Operator Trees. In these representations, nodes are operators (like $+$ or $\times$) and leaves are operands (numbers or variables).
Mingzhe Wang and colleagues introduced FormulaNet, which treats formulas as graphs. Unlike a simple list of tokens, this graph preserves the specific order of function arguments and the "topology" of the logic. Similarly, Aditya Paliwal demonstrated that Graph Neural Networks (GNNs) can outperform sequence models in higher-order logic by using message-passing to allow information to flow between related sub-expressions, regardless of how far apart they appear in a text string.
A key challenge identified by Paliwal is "upward context." In a standard tree-based model, a node knows about its children, but it may not know it is part of a larger subtraction or a denominator. Effective reasoning requires a bidirectional flow where the meaning of a sub-expression is influenced by the broader operators surrounding it.
Tree-Structured Decoding
If the input is a tree, the output should be one too. Traditional "left-to-right" generation is poorly suited for math because a single error early in the string can invalidate the entire hierarchical structure.
Qianying Liu proposed a tree-structured decoder that generates equations top-down. Instead of predicting the next character, the model predicts the next branch of the tree. This ensures that the generated output is always a valid mathematical expression. Wenqi Zhang 0001 extended this in An Expression Tree Decoding Strategy, showing that predicting independent solving steps in parallel—rather than sequentially—significantly improves accuracy on complex multi-step word problems.
Hierarchical Attention Transformers
The most recent evolution of this idea is the Hierarchical Attention Transformer (HAT). Rather than abandoning the Transformer architecture, HATs bake the hierarchy into the attention mechanism itself.
In HATSolver, Mohamed Malhou, Kristin E. Lauter, and Ludovic Perret apply this to computing Gröbner bases—a notoriously difficult algebraic problem. By nesting attention (tokens within terms, terms within equations), the model can focus on local algebraic simplifications while maintaining a "global" view of the entire polynomial system. This allowed them to solve systems with 13 variables, a feat where "flat" Transformers failed completely.
Similarly, Wenda Li used HATs in IsarStep to bridge logical gaps in formal proofs. By recognizing the hierarchy between individual tokens in a proposition and the broader relationships between propositions in a proof, the model can plan complex logical steps that linear models cannot.
Why It Works: Inductive Bias
The success of these methods stems from structural inductive bias. When we force a model to process math as a tree, we are giving it the "rules of the game" for free. It no longer has to spend its parameters learning that parentheses must balance or that addition is commutative; these properties are either enforced by the tree structure or made obvious by the graph topology.
As Zichao Wang demonstrates with FORTE, this structural awareness allows for better formula retrieval and generation because the model's internal "embedding space" is organized by logical similarity rather than just character-level overlap. By aligning the model's architecture with the data's natural geometry, we move closer to machines that can truly reason through the abstract structures of mathematics.