MuGER2: Multi-Granularity Evidence Retrieval and Reasoning for Hybrid Question Answering

Yingyao Wang, Junwei Bao, Chaoqun Duan, Youzheng Wu, Xiaodong He, Tiejun Zhao

TL;DR

This paper introduces MuGER$^2$, a novel approach for Hybrid Question Answering (HQA) that tackles the challenge of leveraging heterogeneous data (tables and linked passages) by effectively combining evidence of different granularities. MuGER$^2$ significantly improves HQA performance by using a unified retriever for multi-granularity evidence and an evidence selector to guide a reader model, outperforming existing methods.

What Problem Does This Paper Address?

Traditional question answering systems often focus on homogeneous data sources like knowledge graphs, tables, or text passages. However, real-world scenarios frequently involve answering questions that require reasoning over a mix of these data types. The Hybrid Question Answering (HQA) task specifically addresses this by requiring models to answer questions using information from both tables and passages linked to table cells. The challenge lies in effectively utilizing the diverse "granularity" of evidence available: coarse-grained information like entire columns or rows, and fine-grained details like individual cells or links. Existing HQA models tend to focus on either coarse-grained or fine-grained evidence, leading to a trade-off: coarse-grained evidence is easier to retrieve but less informative for reasoning, while fine-grained evidence is more informative but harder to retrieve accurately. This paper aims to bridge this gap by developing a method that can harness the strengths of both coarse and fine-grained evidence.

Methods

MuGER$^2$ is a two-stage approach:

  • Stage 1: Multi-Granularity Evidence Retrieval: Instead of separate retrievers for each granularity (column, row, cell, link), MuGER$^2$ employs a unified retriever. This retriever is trained jointly to learn representations for all four types of evidence simultaneously. This joint training allows the model to capture inter-dependencies between different granularities. To further enhance this, the paper incorporates contrastive learning within each granularity. This helps the model learn better representations by pushing apart dissimilar evidence instances and pulling together similar ones within the same granularity.
  • Stage 2: Answer Reasoning with Multi-Granularity Evidence: A key innovation here is the evidence selector (E-SEL). While the retriever provides scores for all granularities, the E-SEL intelligently navigates this multi-granularity information. It first calculates scores for whether the answer is likely within a table cell or a linked passage, considering all granularities. Based on these scores, it decides whether to focus on table-related evidence (In-Table) or passage-related evidence (In-Passage). This allows the model to select the most relevant fine-grained evidence (a specific cell or link) for the final answer reader. The answer reader itself is a standard span-based reading comprehension model (BERT-based).

Datasets

The primary dataset used for training and evaluation is HybridQA. This dataset is specifically designed for the HQA task and consists of questions aligned with tables and passages linked to table cells. The paper notes that the dataset includes "In-Table" (answer is a table cell value) and "In-Passage" (answer is in a linked passage) categories, and also a "Compute" category for numerical reasoning, though their focus is on the first two.

Key Insights and Ideas

  • Granularity Trade-off is Real: The paper empirically demonstrates that coarse-grained evidence (like columns) has higher retrieval recall but lower reasoning performance, while fine-grained evidence (cells, links) has lower retrieval recall but better reasoning. This confirms the core problem they set out to solve.
  • Unified Retrieval is Better: Training a single retriever for all evidence granularities is more effective than using separate models for each. This suggests that different granularities provide complementary information that can be learned together.
  • Evidence Selector is Crucial: Simply flattening all retrieved evidence can overwhelm the reader model. The E-SEL acts as a smart navigator, directing the reader to the most promising fine-grained evidence, significantly boosting performance.
  • Contrastive Learning Enhances Representations: Applying contrastive learning within each granularity helps the retriever learn more robust and discriminative representations of evidence.

Key Results

  • MuGER$^2$-base outperforms strong baselines on the HybridQA dataset, achieving significant improvements in both Exact Match (EM) and F1 scores. Specifically, it surpasses the Hybrider-large model despite having a smaller parameter scale.
  • Ablation studies show that the unified retrieval approach brings substantial gains (1.6% EM, 1.3% F1).
  • The evidence selector (E-SEL) contributes significantly, improving results by 7.8% EM and 7.9% F1 compared to a baseline that simply flattens all retrieved evidence.
  • Contrastive learning adds a further boost of 0.8% EM and 1.1% F1.
  • The paper also analyzes error types, finding that while evidence retrieval errors are minimized by their multi-granularity approach, errors in the evidence selector and reading comprehension module remain significant challenges.

Limitations and Future Work

The authors acknowledge that MuGER$^2$ is currently limited to questions where answers can be extracted from cell values or linked passages using reading comprehension. It cannot answer questions requiring numerical operations (e.g., count, compare). Future work will focus on developing numerical reasoning capabilities for the model to handle these more complex question types.

Summary updated

Abstract

Hybrid question answering (HQA) aims to answer questions over heterogeneous data, including tables and passages linked to table cells. The heterogeneous data can provide different granularity evidence to HQA models, e.t., column, row, cell, and link. Conventional HQA models usually retrieve coarse- or fine-grained evidence to reason the answer. Through comparison, we find that coarse-grained evidence is easier to retrieve but contributes less to the reasoner, while fine-grained evidence is the opposite. To preserve the advantage and eliminate the disadvantage of different granularity evidence, we propose MuGER$^2$, a Multi-Granularity Evidence Retrieval and Reasoning approach. In evidence retrieval, a unified retriever is designed to learn the multi-granularity evidence from the heterogeneous data. In answer reasoning, an evidence selector is proposed to navigate the fine-grained evidence for the answer reader based on the learned multi-granularity evidence. Experiment results on the HybridQA dataset show that MuGER$^2$ significantly boosts the HQA performance. Further ablation analysis verifies the effectiveness of both the retrieval and reasoning designs.
BibTeX
@inproceedings{wang2022muger,
  author = {Yingyao Wang and Junwei Bao and Chaoqun Duan and Youzheng Wu and Xiaodong He and Tiejun Zhao},
  title = {{MuGER2: Multi-Granularity Evidence Retrieval and Reasoning for Hybrid Question Answering}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=r_KNPNgL6q},
}
Figure