Relation Extraction from Community Generated Question-Answer Pairs

Denis Savenkov, Wei-Lwun Lu, Jeff Dalton, Eugene AgichteinNAACL 2015

TL;DR

This paper introduces a novel model for relation extraction from community-generated question-answer (CQA) pairs. By leveraging the discourse between questions and answers, their model extracts significantly more factual triples than existing sentence-based methods, improving knowledge base population efforts.

What Problem Does This Paper Address?

Knowledge bases (KBs) are increasingly important for providing structured answers to user queries. However, existing KBs are far from complete, and a vast amount of information is hidden within unstructured text. Community Question Answering (CQA) websites, like Yahoo! Answers and Answers.com, are rich sources of this unstructured data, containing millions of user-generated question-answer pairs. Traditional relation extraction methods typically process sentences in isolation, which is problematic for CQA data. Often, an answer is only understandable in the context of its question, or it might omit explicit mentions of entities or relations. This paper aims to bridge this gap by developing a model that can extract factual relations by considering the question and answer as a cohesive unit.

Methods

The core of the proposed approach is a QnA-based relation extraction model that incorporates information from both the question and the answer. This is framed as a graphical model where each mention of an entity pair (an entity from the question and an entity from the answer) is represented by a set of features. These features include mention-based features (derived from the text surrounding the entities) and question-based features (capturing aspects of the question's structure and topic). A latent variable predicts the relation expressed in each mention. These individual mention predictions are then combined to infer the set of relations between the entity pair.

The authors also present a sentence-based baseline model, which is a standard approach that processes each sentence independently. They then introduce a variation of this baseline that incorporates question features, aiming to improve performance by giving the model more context. The main contribution is the QnA-based model, which explicitly models the relationship between question and answer entities. The models are trained using a distant supervision approach, where entity pairs known to be related in Freebase are used as positive examples.

Datasets

The experiments were conducted on two publicly available CQA datasets:

  • Yahoo! Answers Comprehensive Questions and Answers: Contains over 4.4 million questions.
  • WikiAnswers: A crawl of WikiAnswers, resulting in over 19.6 million question-answer pairs after processing.

These datasets are characterized by their large scale and the often informal and conversational nature of the text.

Key Insights and Ideas

  • Discourse Matters: The most significant insight is that treating question-answer pairs as a single discourse unit, rather than independent sentences, is crucial for effective relation extraction from CQA data. Answers often rely on the question for context, and ignoring this link leads to missed information.
  • Beyond Explicit Statements: The model can extract relations even when the answer doesn't explicitly state the predicate or fully name the entities, as long as the context of the question and answer together provides enough clues.
  • Complementary Strengths: The QnA-based model extracts a substantial number of relation triples that sentence-based models miss, suggesting that combining both approaches can lead to more comprehensive knowledge base population.

Key Results

  • The QnA-based model successfully extracts an additional ~20% to ~40% of relation triples that are not captured by existing sentence-based models.
  • Combining the QnA-based model with the sentence-based baseline leads to higher precision and recall compared to either model alone.
  • Interestingly, adding question features to the sentence-based model did not show a significant improvement in performance, highlighting the importance of the QnA-pair discourse modeling itself.
  • The authors report precision-recall curves demonstrating the superiority of their QnA-based model and the combined approach across both datasets.

Limitations and Future Work

  • Entity Linking Errors: A significant source of errors is entity linking, where entities are incorrectly linked to existing KB entries, leading to false positives.
  • Discourse Model Simplicity: The current discourse model might be too simplistic for answers containing extensive additional information or complex sentence structures.
  • Distant Supervision Artifacts: The distant supervision approach can lead to errors, for example, extracting relations for composers when the text mentions singers who are also composers.
  • Freebase Limitations: A notable portion of extracted relations are not found in Freebase, either because they are missing (33%) or contradict existing entries (8%). This points to potential limitations in the Freebase schema itself or issues with data quality.
  • Trustworthiness of CQA Data: The paper raises questions about the trustworthiness of information in CQA sites, especially for medical or factual queries where user expertise might vary.
  • Automatic Predicate Extraction: A promising future direction is the automatic extraction of new predicates that users are interested in, going beyond existing KB schemas.

Summary updated

BibTeX
@inproceedings{savenkov2015relation,
  author = {Denis Savenkov and Wei-Lwun Lu and Jeff Dalton and Eugene Agichtein},
  title = {{Relation Extraction from Community Generated Question-Answer Pairs}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2015},
}
Figure