Summarizing and Exploring Tabular Data in Conversational Search

Shuo Zhang, Zhuyun Dai, Krisztian Balog, Jamie CallanSIGIR 2020 · DOI · OpenReview

Abstract

Tabular data provide answers to a significant portion of search queries. However, reciting an entire result table is impractical in conversational search systems. We propose to generate natural language summaries as answers to describe the complex information contained in a table. Through crowdsourcing experiments, we build a new conversation-oriented, open-domain table summarization dataset. It includes annotated table summaries, which not only answer questions but also help people explore other information in the table. We utilize this dataset to develop automatic table summarization systems as SOTA baselines. Based on the experimental results, we identify challenges and point out future research directions that this resource will support.

Introduction

While approximately 10% of search engine "Question-Answering" queries are addressed by tabular data, presenting this information via conversational interfaces like Siri or Alexa remains a significant challenge. Simply reading a large table aloud leads to cognitive overload and a poor user experience. Existing summarization methods often rely on rigid schemas and fail to account for conversational context—the specific question asked—or provide "hooks" for further exploration. This research defines a high-quality conversational table summary as one that answers the immediate query, provides a high-level overview of the table's scope, and encourages follow-up questions.

Figure 2: Example of a conversational agent providing a natural language summary of a sports stadium table.

Methodology and Dataset

To address this, the authors developed a new dataset based on WikiTableQuestions, selecting 200 Wikipedia tables across diverse domains like Sports and Music. Through a multi-stage crowdsourcing process, they collected 1,000 human-written summaries. These summaries were further annotated at the sentence level for relevance and ranked to establish a gold-standard ground truth.

For the modeling task, tables were "flattened" into a linear key:value format to be compatible with transformer-based architectures. The study evaluated three models:

  • CopyNet: An LSTM-based model with a copy mechanism for technical terms.
  • GPT-2: A pre-trained transformer decoder.
  • T5 (Text-to-Text Transfer Transformer): A powerful encoder-decoder model.

Results and Discussion

Quantitative evaluation shows that T5 is the superior architecture for this task, significantly outperforming both GPT-2 and CopyNet across standard metrics.

Figure 5: Performance comparison of CopyNet, GPT-2, and T5 models using ROUGE and BLEU metrics.

Despite T5's high fluency and superior BLEU scores (10.43 vs. GPT-2's 5.35), qualitative analysis revealed a persistent "Math Gap." Models frequently struggled with basic mathematical reasoning and factual grounding. Error analysis showed that 84% of generated summaries contained mathematical errors (e.g., incorrect counts), and 38% referred to the wrong table attributes. This suggests that while models can mimic the style of a summary, they lack a logical understanding of the underlying data structure.

Conclusion

This work highlights the necessity of moving beyond simple text generation toward grounded reasoning in conversational search. Future research should explore combining semantic parsing—converting queries into executable commands like SQL—with natural language generation to ensure that conversational summaries are both fluent and factually accurate.

Summary updated

Abstract

Tabular data provide answers to a significant portion of search queries. However, reciting an entire result table is impractical in conversational search systems. We propose to generate natural language summaries as answers to describe the complex information contained in a table. Through crowdsourcing experiments, we build a new conversation-oriented, open-domain table summarization dataset. It includes annotated table summaries, which not only answer questions but also help people explore other information in the table. We utilize this dataset to develop automatic table summarization systems as SOTA baselines. Based on the experimental results, we identify challenges and point out future research directions that this resource will support.
BibTeX
@inproceedings{zhang2020summarizing,
  author = {Shuo Zhang and Zhuyun Dai and Krisztian Balog and Jamie Callan},
  title = {{Summarizing and Exploring Tabular Data in Conversational Search}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2020},
  doi = {10.1145/3397271.3401205},
}
Figure