Powering COVID-19 community Q&A with Curated Side Information

Manisha Verma, Kapil Thadani, Shaunak MishraarXiv 2021 · arXiv

TL;DR

This paper introduces a neural ranking framework that improves the quality of community-driven COVID-19 Q&A by grounding user-generated answers in curated "side information" from trusted sources like the WHO and CDC. By using a novel temperature-regulated attention mechanism, the model selectively weighs external technical documents to identify the most accurate and relevant answers, significantly outperforming standard similarity-based and BERT-based baselines.

What Problem Does This Paper Address?

Community Question Answering (cQ&A) platforms like Reddit, Quora, and Yahoo! Answers are popular for their flexibility, but they pose risks when the topic is a public health crisis like COVID-19. User-generated answers can be speculative, outdated, or flat-out wrong.

Traditional ranking systems often rely on simple question-answer similarity or user reputation. While some advanced systems use Knowledge Bases (KBs), these are poorly suited for rapidly evolving events; KBs are hard to update in real-time and struggle with unstructured data. The authors tackle the challenge of how to effectively "inject" reliable, free-text external information (like scientific papers or official health guidelines) into the ranking process to ensure that the most medically sound answers rise to the top.

Methods

The authors propose a neural architecture called $\tau$-att (temperature-regulated attention). The system works in three main stages:

  1. Encoding: User questions and answers are encoded into vectors using Long Short-Term Memory (LSTM) networks.
  2. External Source Augmentation: The model encodes segments of external data (e.g., a WHO FAQ pair or a research paper abstract).
  3. Temperature-Regulated Attention: This is the core innovation. The model uses the user's question as a "query" to look across the external "keys" (the side information). The "temperature" parameter ($\tau$) controls the sharpness of the attention distribution. A lower temperature makes the model focus intensely on the most relevant external snippets, while a higher temperature spreads attention more uniformly.
Figure 2: External source augmentation model

Finally, the model concatenates the user Q&A representation with the weighted average of the relevant side information to predict a relevance score.

Datasets

The study utilizes two primary cQ&A datasets and two external "side information" sources:

  • Primary (User) Datasets:
    • Yahoo! Answers: A custom-crawled dataset of 1,880 COVID-19 questions and 11,500 answers.
  • Infobot Dataset: A manually labeled dataset from Johns Hopkins University (JHU) containing 8,000+ question-answer pairs.
  • External (Curated) Sources:
    • TREC-COVID: A collection of 5,000+ scientific documents from the CORD-19 dataset.
  • WHO FAQ: 147 curated question-answer pairs from the World Health Organization website.

Key Insights and Ideas

  • Similarity is Not Enough: The authors found that simply matching the text of a question to an answer (the qasim baseline) is insufficient for sensitive health topics.
  • The Power of Temperature: Tuning the temperature ($\tau$) is critical. The authors found that an optimal range exists (typically between 0.1 and 1.0). If $\tau$ is too high, the model is overwhelmed by "noise" from irrelevant external documents; if it's too low, it might ignore useful context.
  • Category Sensitivity: The model performed exceptionally well in the "Family" category (71% improvement over baselines), suggesting that external curated info is particularly helpful for practical, life-oriented questions.
  • Explainability: Because the model uses attention, researchers can "peek" under the hood to see which external documents the model used to validate a user's answer. As seen in Figures 5 and 6, the model correctly assigns higher weights to external queries that are semantically related to the user's specific concern (e.g., masks or virus transmission).
Attention weights showing how the model focuses on specific external TREC queries when ranking a user's question.

Key Results

The $\tau$-att model consistently outperformed baselines across multiple metrics, including Precision@1 (P@1) and Mean Reciprocal Rank (MRR).

  • Against BERT: On the TREC-COVID data, the $\tau$-att model beat fine-tuned BERT models by approximately 30% in precision and 18% in recall.
  • Against Simple Attention: The temperature-regulated version outperformed standard linear attention, proving that selective focus is better than broad aggregation when dealing with large external corpora.
  • Parameter Impact: Figure 4 illustrates that the $\tau$-att model (blue line) maintains higher precision across various settings compared to a simpler linear combination ($\lambda$-sim) of scores.
Comparison of P@1 performance across different values of temperature ($\tau$) and linear combination ($\lambda$).

Limitations and Future Work

The authors acknowledge a few key hurdles:

  • Scaling: While effective for thousands of documents, the current attention mechanism might need more efficient, scalable implementations to handle millions of external entries in real-time.
  • Source Availability: The system relies on the existence of high-quality, curated external data. For very niche or emerging topics where no "expert" source exists yet, the model's advantage would diminish.
  • Future Directions: The researchers plan to expand this work to other sensitive domains like disaster relief (e.g., wildfires or earthquakes) and explore ways to integrate multiple different external sources simultaneously into a single ranking model.

Summary updated

Abstract

Community question answering and discussion platforms such as Reddit, Yahoo! answers or Quora provide users the flexibility of asking open ended questions to a large audience, and replies to such questions maybe useful both to the user and the community on certain topics such as health, sports or finance. Given the recent events around COVID-19, some of these platforms have attracted 2000+ questions from users about several aspects associated with the disease. Given the impact of this disease on general public, in this work we investigate ways to improve the ranking of user generated answers on COVID-19. We specifically explore the utility of external technical sources of side information (such as CDC guidelines or WHO FAQs) in improving answer ranking on such platforms. We found that ranking user answers based on question-answer similarity is not sufficient, and existing models cannot effectively exploit external (side) information. In this work, we demonstrate the effectiveness of different attention based neural models that can directly exploit side information available in technical documents or verified forums (e.g., research publications on COVID-19 or WHO website). Augmented with a temperature mechanism, the attention based neural models can selectively determine the relevance of side information for a given user question, while ranking answers.
BibTeX
@misc{verma2021powering,
  author = {Manisha Verma and Kapil Thadani and Shaunak Mishra},
  title = {{Powering COVID-19 community Q\&A with Curated Side Information}},
  year = {2021},
  eprint = {2101.11556},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2101.11556},
}
Figure