Introduction
Chain-of-Thought (CoT) prompting is widely regarded as a standard technique for enhancing the reasoning capabilities of Large Language Models (LLMs). However, recent research into Table Question Answering (TableQA) reveals a significant scaling disparity. While step-by-step reasoning acts as a catalyst for models with 70B+ parameters, it frequently degrades the performance of 7B-class models. This "performance contrast" suggests that the benefits of explicit reasoning chains are not universal but are contingent on a model's underlying capacity to handle structured data.
The "Cascading Error" Trap
In TableQA, models must navigate complex grids to locate data, perform calculations, and draw inferences. When larger models like Llama-3-70B are prompted to think step-by-step, each intermediate "useful step" guides the model toward the correct answer. In contrast, smaller models (SLMs) like Llama-2-7B often stumble at the very first step—for example, by hallucinating coordinates or misidentifying a column. Because the model is then forced to follow its own flawed logic, these initial mistakes lead to a "cascading error" effect, resulting in a final answer that is often less accurate than a direct "vanilla" response.
Methodology: The Table-Logic Framework
To diagnose these failures, the researchers introduced Table-Logic, a sequential prompting framework designed to mirror human table processing. The flow involves five distinct stages:
- Table Reading Instruction: Establishing the table's dictionary-like structure.
- Identify Critical Columns: Selecting relevant headers.
- Identify Critical Rows: Filtering for relevant entries.
- Identify Operations: Determining the necessary logic (e.g., sum, average, retrieval).
- Final Prediction: Synthesizing all previous steps.
Structural Blindness in Smaller Models
The core issue for smaller models appears to be "structural blindness." While they can often identify the content of a table, they struggle significantly with its coordinates.
As shown in the sub-task analysis, the disparity is most pronounced in "Column Finding" and "Counting Columns." For instance, larger models achieved near-perfect scores (0.957) in column counting, whereas smaller models averaged only 0.340. This suggests that SLMs treat tables more like unstructured paragraphs than organized grids, making them unable to reliably map a question's intent to a specific index.
Key Results
The empirical results across benchmarks like TAT-QA, HybridQA, and WikiTableQuestions (WikiTQ) underscore this divide:
- Llama-3-70B saw a 7.8% accuracy boost on HybridQA using Table-Logic compared to vanilla inference.
- Llama-2-7B experienced an 11% performance decline under the same conditions.
Interestingly, when smaller models were provided with "scaffolding"—correct structural information or rows identified by a larger model—their performance improved significantly. This indicates that the bottleneck for SLMs is not necessarily logical reasoning, but the initial perception and navigation of the table structure.
Conclusion and Future Directions
The failure of step-by-step reasoning in smaller models highlights a critical need for specialized training. Rather than relying on general-purpose reasoning chains, future work should focus on fine-tuning SLMs specifically for "table structure awareness." Improving a model's ability to count, index, and locate data within a grid is a necessary prerequisite before it can successfully employ complex, multi-step logic in TableQA tasks.
Summary updated