Exploring Performance Contrasts in TableQA: Step-by-Step Reasoning Boosts Bigger Language Models, Limits Smaller Language Models

Haoyan Yang, Yixuan Wang, Keyue Tong, Hongjin Zhu, Yuanxin ZhangarXiv 2024 · DOI · arXiv

Introduction

Chain-of-Thought (CoT) prompting is widely regarded as a standard technique for enhancing the reasoning capabilities of Large Language Models (LLMs). However, recent research into Table Question Answering (TableQA) reveals a significant scaling disparity. While step-by-step reasoning acts as a catalyst for models with 70B+ parameters, it frequently degrades the performance of 7B-class models. This "performance contrast" suggests that the benefits of explicit reasoning chains are not universal but are contingent on a model's underlying capacity to handle structured data.

Figure 1: The performance contrast between large and small LMs when using step-by-step reasoning.

The "Cascading Error" Trap

In TableQA, models must navigate complex grids to locate data, perform calculations, and draw inferences. When larger models like Llama-3-70B are prompted to think step-by-step, each intermediate "useful step" guides the model toward the correct answer. In contrast, smaller models (SLMs) like Llama-2-7B often stumble at the very first step—for example, by hallucinating coordinates or misidentifying a column. Because the model is then forced to follow its own flawed logic, these initial mistakes lead to a "cascading error" effect, resulting in a final answer that is often less accurate than a direct "vanilla" response.

Methodology: The Table-Logic Framework

To diagnose these failures, the researchers introduced Table-Logic, a sequential prompting framework designed to mirror human table processing. The flow involves five distinct stages:

  1. Table Reading Instruction: Establishing the table's dictionary-like structure.
  2. Identify Critical Columns: Selecting relevant headers.
  3. Identify Critical Rows: Filtering for relevant entries.
  4. Identify Operations: Determining the necessary logic (e.g., sum, average, retrieval).
  5. Final Prediction: Synthesizing all previous steps.

Structural Blindness in Smaller Models

The core issue for smaller models appears to be "structural blindness." While they can often identify the content of a table, they struggle significantly with its coordinates.

Figure 5: Radar chart showing the performance gap between large and small models across granular table sub-tasks.

As shown in the sub-task analysis, the disparity is most pronounced in "Column Finding" and "Counting Columns." For instance, larger models achieved near-perfect scores (0.957) in column counting, whereas smaller models averaged only 0.340. This suggests that SLMs treat tables more like unstructured paragraphs than organized grids, making them unable to reliably map a question's intent to a specific index.

Key Results

The empirical results across benchmarks like TAT-QA, HybridQA, and WikiTableQuestions (WikiTQ) underscore this divide:

  • Llama-3-70B saw a 7.8% accuracy boost on HybridQA using Table-Logic compared to vanilla inference.
  • Llama-2-7B experienced an 11% performance decline under the same conditions.

Interestingly, when smaller models were provided with "scaffolding"—correct structural information or rows identified by a larger model—their performance improved significantly. This indicates that the bottleneck for SLMs is not necessarily logical reasoning, but the initial perception and navigation of the table structure.

Conclusion and Future Directions

The failure of step-by-step reasoning in smaller models highlights a critical need for specialized training. Rather than relying on general-purpose reasoning chains, future work should focus on fine-tuning SLMs specifically for "table structure awareness." Improving a model's ability to count, index, and locate data within a grid is a necessary prerequisite before it can successfully employ complex, multi-step logic in TableQA tasks.

Summary updated

Abstract

This paper proposes a detailed prompting flow, termed Table-Logic, to investigate the performance contrasts between bigger and smaller language models (LMs) utilizing step-by-step reasoning methods in the TableQA task. The method processes tasks by sequentially identifying critical columns and rows given question and table with its structure, determining necessary aggregations, calculations, or comparisons, and finally inferring the results to generate a precise prediction. By deploying this method, we observe a 7.8% accuracy improvement in bigger LMs like Llama-3-70B compared to the vanilla on HybridQA, while smaller LMs like Llama-2-7B shows an 11% performance decline. We empirically investigate the potential causes of performance contrasts by exploring the capabilities of bigger and smaller LMs from various dimensions in TableQA task. Our findings highlight the limitations of the step-by-step reasoning method in small models and provide potential insights for making improvements.
BibTeX
@misc{yang2024exploring,
  author = {Haoyan Yang and Yixuan Wang and Keyue Tong and Hongjin Zhu and Yuanxin Zhang},
  title = {{Exploring Performance Contrasts in TableQA: Step-by-Step Reasoning Boosts Bigger Language Models, Limits Smaller Language Models}},
  year = {2024},
  eprint = {2411.16002},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.16002},
}
Figure