Abstract
Tabular data provide answers to a significant portion of search queries. However, reciting an entire result table is impractical in conversational search systems. We propose to generate natural language summaries as answers to describe the complex information contained in a table. Through crowdsourcing experiments, we build a new conversation-oriented, open-domain table summarization dataset. It includes annotated table summaries, which not only answer questions but also help people explore other information in the table. We utilize this dataset to develop automatic table summarization systems as SOTA baselines. Based on the experimental results, we identify challenges and point out future research directions that this resource will support.
Introduction
While approximately 10% of search engine "Question-Answering" queries are addressed by tabular data, presenting this information via conversational interfaces like Siri or Alexa remains a significant challenge. Simply reading a large table aloud leads to cognitive overload and a poor user experience. Existing summarization methods often rely on rigid schemas and fail to account for conversational context—the specific question asked—or provide "hooks" for further exploration. This research defines a high-quality conversational table summary as one that answers the immediate query, provides a high-level overview of the table's scope, and encourages follow-up questions.
Methodology and Dataset
To address this, the authors developed a new dataset based on WikiTableQuestions, selecting 200 Wikipedia tables across diverse domains like Sports and Music. Through a multi-stage crowdsourcing process, they collected 1,000 human-written summaries. These summaries were further annotated at the sentence level for relevance and ranked to establish a gold-standard ground truth.
For the modeling task, tables were "flattened" into a linear key:value format to be compatible with transformer-based architectures. The study evaluated three models:
- CopyNet: An LSTM-based model with a copy mechanism for technical terms.
- GPT-2: A pre-trained transformer decoder.
- T5 (Text-to-Text Transfer Transformer): A powerful encoder-decoder model.
Results and Discussion
Quantitative evaluation shows that T5 is the superior architecture for this task, significantly outperforming both GPT-2 and CopyNet across standard metrics.
Despite T5's high fluency and superior BLEU scores (10.43 vs. GPT-2's 5.35), qualitative analysis revealed a persistent "Math Gap." Models frequently struggled with basic mathematical reasoning and factual grounding. Error analysis showed that 84% of generated summaries contained mathematical errors (e.g., incorrect counts), and 38% referred to the wrong table attributes. This suggests that while models can mimic the style of a summary, they lack a logical understanding of the underlying data structure.
Conclusion
This work highlights the necessity of moving beyond simple text generation toward grounded reasoning in conversational search. Future research should explore combining semantic parsing—converting queries into executable commands like SQL—with natural language generation to ensure that conversational summaries are both fluent and factually accurate.
Summary updated