Towards Collaborative Question Answering: A Preliminary Study

Xiangkun Hu, Hang Yan, Qipeng Guo, Xipeng Qiu, Weinan Zhang, Zheng ZhangarXiv 2022 · arXiv

Abstract

Modern Question Answering (QA) systems typically assume a monolithic knowledge source. However, real-world information is often siloed across different departments or organizations. This paper introduces CollabQA, a framework for collaborative QA where a moderator agent must coordinate with multiple specialized "panelists" to answer complex, multi-hop questions. Our study highlights the challenges of learning effective collaboration policies, particularly the "Right Answer, Wrong Reason" trap caused by data bias and the emergence of "group bias" in agent partnerships.

Introduction: The Challenge of Siloed Knowledge

Most current QA research focuses on agents with access to a single, massive database. In practice, knowledge is fragmented. For instance, a customer service bot may access order history but lack technical specifications stored elsewhere. This work shifts the focus toward Collaborative Question Answering, where a front-facing moderator decomposes complex queries into sub-questions, polls specialized experts, and synthesizes their responses. The primary hurdle is developing a policy that determines what to ask, whom to ask, and when a sufficient answer has been reached.

Methods: A Two-Tier Architecture

The proposed system utilizes a hierarchical structure:

  1. The Panelists (Experts): Each panelist manages a private Knowledge Graph (KG). They employ a modified Relational Graph Convolutional Network (R-GCN) with trainable vectors instead of standard weight matrices to reduce GPU memory overhead.
  2. The Moderator (Receptionist): This agent lacks internal knowledge, instead using a BiLSTM to track dialogue history. A Reinforcement Learning (RL) policy decides whether to issue a sub-query (via pre-defined templates) or return a final answer.

Training occurs in two stages: pre-training panelists on simple one-hop questions, followed by training the moderator via Policy Gradient to manage the multi-turn collaboration.

The CollabQA Dataset

To evaluate this framework, the authors developed the CollabQA dataset, a synthetic benchmark featuring three interconnected KGs (Person, Company, and City).

  • Scale: 66,800 training and 8,350 test pairs.
  • Complexity: Questions require cross-graph reasoning. For example, answering "When was the largest company in the city where Person#1 was born established?" requires jumping from the Person KG to the City KG, and finally to the Company KG.
Multi-turn collaborative reasoning process on the CollabQA dataset.

Key Insights: Reasoning vs. Accuracy

A critical finding of this study is the "Right Answer, Wrong Reason" trap. There is a significant discrepancy between Exact Match Accuracy (EMA) of the final answer and Exact Match of the reasoning Path (EMP).

Impact of data overlap on answer accuracy (EMA) vs. reasoning path accuracy (EMP).

As shown in Figure 6, as data overlap increases (e.g., most people living in their birth city), the moderator often achieves the correct answer despite following an incorrect reasoning path. This suggests that high QA accuracy can mask logical failures.

Furthermore, the study identified Group Bias: moderators become specialized to their specific panel. Performance drops when a moderator is paired with new, equally accurate experts, indicating that collaboration involves learning specific partner "quirks" rather than just general logic. Finally, the authors found that inductive priors (penalties for redundant or failed queries) were essential for policy convergence.

Key Results

  • Baseline RL: Achieved 68.8% EMA.
  • Enhanced Reward: With inductive biases, performance reached 80.1% EMA.
  • Reasoning Accuracy: EMP hovered around 52.6%, reflecting the difficulty in distinguishing semantically similar relations (e.g., "born in" vs. "lives in") without common-sense grounding.

Limitations and Future Directions

This preliminary study identifies several areas for growth:

  • Proactive Collaboration: Moving beyond passive panelists to agents that can discuss or share information autonomously.
  • Natural Language Generation: Replacing fixed templates with NLG to increase flexibility, despite the larger action space.
  • Common Sense Integration: Helping agents distinguish between logically distinct but statistically correlated relations.
  • Real-World Scaling: Transitioning from synthetic "toy" tasks to noisy, large-scale disjoint datasets.

Summary updated

Abstract

Knowledge and expertise in the real-world can be disjointedly owned. To solve a complex question, collaboration among experts is often called for. In this paper, we propose CollabQA, a novel QA task in which several expert agents coordinated by a moderator work together to answer questions that cannot be answered with any single agent alone. We make a synthetic dataset of a large knowledge graph that can be distributed to experts. We define the process to form a complex question from ground truth reasoning path, neural network agent models that can learn to solve the task, and evaluation metrics to check the performance. We show that the problem can be challenging without introducing prior of the collaboration structure, unless experts are perfect and uniform. Based on this experience, we elaborate extensions needed to approach collaboration tasks in real-world settings.
BibTeX
@misc{hu2022towards,
  author = {Xiangkun Hu and Hang Yan and Qipeng Guo and Xipeng Qiu and Weinan Zhang and Zheng Zhang},
  title = {{Towards Collaborative Question Answering: A Preliminary Study}},
  year = {2022},
  eprint = {2201.09708},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2201.09708},
}
Figure