Table Retrieval May Not Necessitate Table-specific Model Design

Zhiruo Wang, Zhengbao Jiang, Eric Nyberg, Graham Neubig

Abstract

Tables are an important form of structured data for both human and machine readers alike, providing answers to questions that cannot, or cannot easily, be found in texts. Recent work has designed special models and training paradigms for table-related tasks such as table-based question answering and table retrieval. Though effective, they add complexity in both modeling and data acquisition compared to generic text solutions and obscure which elements are truly beneficial. In this work, we focus on the task of table retrieval, and ask: "is table-specific model design necessary for table retrieval, or can a simpler text-based model be effectively used to achieve a similar result?" First, we perform an analysis on a table-based portion of the Natural Questions dataset (NQ-table), and find that structure plays a negligible role in more than 70% of the cases. Based on this, we experiment with a general Dense Passage Retriever (DPR) based on text and a specialized Dense Table Retriever (DTR) that uses table-specific model designs. We find that DPR performs well without any table-specific design and training, and even achieves superior results compared to DTR when fine-tuned on properly linearized tables. We then experiment with three modules to explicitly encode table structures, namely auxiliary row/column embeddings, hard attention masks, and soft relation-based attention biases. However, none of these yielded significant improvements, suggesting that table-specific model design may not be necessary for table retrieval.

Introduction

Tables are ubiquitous in digital documents, serving as a semi-structured bridge between raw text and formal databases. Because tables possess inherent structural properties—rows, columns, and headers—the prevailing sentiment in Natural Language Processing (NLP) has been that specialized architectures are required to "understand" them. Models like TAPAS or the Dense Table Retriever (DTR) often incorporate extra embeddings for row/column indices or customized attention mechanisms to capture these relationships.

However, these specialized models introduce significant complexity and often require expensive, table-specific pre-training on millions of examples. This paper investigates whether such complexity is truly necessary for the specific task of table retrieval: identifying the correct table from a large corpus (such as Wikipedia) to answer a given query. The authors hypothesize that for retrieval, semantic content matching may be far more critical than deep structural reasoning.

Methodology

The researchers compared two distinct approaches to table retrieval:

  1. Dense Passage Retriever (DPR): A standard, text-only bi-encoder based on the BERT architecture. To process tables, the researchers linearized them—converting the grid structure into a flat string of text using simple delimiters (e.g., periods to separate rows and vertical bars to separate cells).
  2. Dense Table Retriever (DTR): A specialized model designed for tables, featuring row/column embeddings and extensive pre-training on Wikipedia tables using tasks like Masked Language Modeling and the Inverse Cloze Task.
Comparison of training pipelines for DPR (text-based) and DTR (table-specific).

The study utilized the NQ-table dataset, a subset of the Natural Questions (NQ) corpus where answers are found within Wikipedia tables (approximately 12,000 questions). For training the baseline text retriever, the standard NQ-text dataset (71,000 questions) was used.

To further test the necessity of structural awareness, the authors attempted to "inject" structure into the standard DPR by adding three modules:

  • Auxiliary Embeddings: Adding row and column indices to each token's embedding.
  • Hard Attention Masks: Restricting tokens to only attend to others within the same row or column.
  • Soft Attention Biases: Adding scalars to attention scores based on the structural distance between cells.
Illustration of auxiliary row and column embeddings used to encode table structure.

Analysis and Results

A manual analysis of 100 random samples from the NQ-table dataset revealed that in 71% of cases, the correct table could be identified simply by matching keywords in the query to the table's title or headers. Only 29% of cases required an understanding of row/column alignment to distinguish the correct table from others.

Performance Comparison

When fine-tuned on the NQ-table dataset, the "simple" DPR-table achieved a Top-1 accuracy of 67.91%, notably outperforming the specialized DTR-large (63.98%). Even in zero-shot scenarios, the standard text-based DPR performed remarkably well, nearly matching the specialized DTR model.

The Role of Structure

The experiments with explicit structural features (row/column embeddings and specialized attention) yielded negligible improvements. In some instances, these additions even slightly degraded performance compared to the well-tuned, simple text model. This suggests that the semantic alignment—mapping a query term like "highest paid" to a header like "Salary"—is a task standard language models are already proficient at.

To confirm if the model was learning any structure implicitly, the authors conducted a "Shuffling Test." By randomizing the order of rows and columns in the linearized text, performance dropped significantly (from 67.91% to ~55-58%). This indicates that while specialized architectures may not be needed, the order of the linearized text provides sufficient structural cues for the Transformer to understand cell relationships.

Conclusion

The findings suggest that for the task of table retrieval, the "table-specific tax" of complex modeling and pre-training may not be justified. Linearization allows standard text models to capture enough structural information to outperform specialized counterparts.

However, the authors note a crucial distinction between retrieval and reasoning. While structure may be secondary for finding a table, it remains essential for Table Question Answering (Table QA), which requires operations like cross-cell summation, comparison, and precise extraction. Future work should explore how these findings translate to more complex, hierarchical tables found in specialized domains like finance or science, and how long-context models might handle tables that exceed standard token limits.

Summary updated

Abstract

Tables are an important form of structured data for both human and machine readers alike, providing answers to questions that cannot, or cannot easily, be found in texts. Recent work has designed special models and training paradigms for table-related tasks such as tablebased question answering and table retrieval. Though effective, they add complexity in both modeling and data acquisition compared to generic text solutions and obscure which elements are truly beneficial. In this work, we focus on the task of table retrieval, and ask: "is table-specific model design necessary for table retrieval, or can a simpler text-based model be effectively used to achieve a similar result?" First, we perform an analysis on a table-based portion of the Natural Questions dataset (NQtable), and find that structure plays a negligible role in more than 70% of the cases. Based on this, we experiment with a general Dense Passage Retriever (DPR) based on text and a specialized Dense Table Retriever (DTR) that uses table-specific model designs. We find that DPR performs well without any table-specific design and training, and even achieves superior results compared to DTR when fine-tuned on properly linearized tables. We then experiment with three modules to explicitly encode table structures, namely auxiliary row/column embeddings, hard attention masks, and soft relation-based attention biases. However, none of these yielded significant improvements, suggesting that table-specific model design may not be necessary for table retrieval. 1
BibTeX
@misc{wang2022table,
  author = {Zhiruo Wang and Zhengbao Jiang and Eric Nyberg and Graham Neubig},
  title = {{Table Retrieval May Not Necessitate Table-specific Model Design}},
  year = {2022},
  eprint = {2205.09843},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2205.09843},
}

Loading related content...

Figure