TL;DR
ELASTIC is a novel framework that treats numerical reasoning as a modular compilation task rather than a simple sequence-to-sequence problem. By decoupling the generation of mathematical operators from their operands and using a dedicated "Memory Register" for intermediate results, it achieves state-of-the-art performance on complex multi-step math word problems.
What Problem Does This Paper Address?
While Large Language Models (LLMs) have mastered basic reading comprehension, they often stumble when asked to perform "numerical reasoning"—the ability to understand a text, extract relevant numbers, and perform multi-step calculations to find an answer.
Previous approaches typically treated this as a translation task: converting a sentence into a mathematical string (like a sequence or a tree). However, these models suffer from two major flaws:
- Cascading Errors: In long, complex calculations, a single mistake in an early step ruins the entire sequence.
- Rigidity: Many models are hard-coded to handle only binary operations (operators with exactly two numbers), making them difficult to adapt to diverse mathematical domains where an operation might require one, three, or more operands.
Methods
The authors introduce ELASTIC (numEricaL reASoning with adapTive symbolIc Compiler). Instead of a single decoder, ELASTIC uses a "Compiler" architecture consisting of four specialized modules.
- Reasoning Manager: The "orchestrator" that decides whether the model should be picking an operator or an operand at any given step.
- Operator Generator: Predicts the next mathematical operation (e.g.,
add,divide,sqrt). - Operands Generator: Selects the specific numbers from the text or constants (like $\pi$) to plug into the operator. Crucially, it continues generating operands until it produces a
nonetoken, allowing for a dynamic number of inputs. - Memory Register: This acts as a "scratchpad." It saves the result of a previous calculation into a cache token (e.g.,
#0,#1). This allows the model to reference intermediate results in later steps of a complex problem.
The system uses RoBERTa as the underlying encoder to understand the context of the question and the provided text.
Datasets
The model was evaluated on two primary benchmarks:
- FinQA: A dataset of over 8,000 examples derived from annual financial reports. It requires reasoning over both unstructured text and structured tables.
- MathQA: A larger dataset (37,000+ problems) based on GRE and GMAT math word problems. These are significantly more complex and require more advanced reasoning steps.
- Extended FinQA: The authors also created a modified version of FinQA to test "elasticity"—specifically adding questions that require operators with more than two operands.
Key Insights and Ideas
- Separation of Concerns: The "aha moment" here is that generating a math program is fundamentally different from generating natural language. By separating the logic (operators) from the data (operands), the model avoids getting "confused" by the different roles these tokens play.
- The Memory Departing Distance (M-MDD): The authors introduced this metric to measure how "far back" a model has to look to find a previous result. They proved that as M-MDD increases, standard models fail, but ELASTIC’s Memory Register keeps performance stable.
- Domain Agnosticism: Because the compiler doesn't assume how many numbers follow an operator, it can be applied to financial data, physics problems, or general arithmetic without changing the architecture.
Key Results
ELASTIC outperformed previous state-of-the-art (SOTA) models across the board.
- FinQA: Achieved 68.96% execution accuracy, a significant jump over the previous best (FinQANet at 65.05%).
- MathQA: Achieved 83.00% program accuracy, outperforming the NeRd model (79.70%).
- Complexity Scaling: On problems requiring three or more steps, ELASTIC maintained much higher accuracy than baselines, which saw their performance "tumble" as the number of steps increased.
Limitations and Future Work
- The Human Gap: Despite the SOTA results, there is still a massive gap between ELASTIC (68.9%) and human experts (91.1%) on the FinQA dataset.
- Entity Grounding: The authors note that the model sometimes struggles to correctly link numbers to the specific entities they represent in the text (e.g., confusing "Revenue in 2021" with "Revenue in 2022").
- Annotation Dependency: The model requires training data that includes the intermediate "gold" reasoning programs. Future work could explore how to generate these programs automatically or learn from just the final answer (reinforcement learning).
Summary updated