Using dependency parsing and machine learning for factoid question answering on spoken documents

Pere R. Comas, Jordi Turmo, Lluís MàrquezInterspeech 2010

TL;DR

This paper tackles the challenge of extracting precise answers from spoken document transcripts, a task complicated by speech disfluencies and Automatic Speech Recognition (ASR) errors. They improve upon a heuristic-based answer extraction system by introducing a machine learning reranker, and further enhance it by incorporating syntactic information derived from dependency parsing, achieving significant gains in precision and coverage.

What Problem Does This Paper Address?

Question Answering (QA) systems aim to provide direct answers to natural language questions, rather than just returning relevant documents. Factoid QA, a sub-task, focuses on extracting specific entities like names, dates, or locations. Applying QA to spoken documents presents unique challenges: speech transcripts are often noisy due to disfluencies, repetitions, and ASR errors, making them less amenable to traditional NLP techniques that rely on clean, written text. Existing QA systems for spoken documents often use shallow linguistic processing due to robustness concerns. This paper seeks to bridge this gap by leveraging more sophisticated NLP techniques, specifically dependency parsing, within a machine learning framework to improve answer extraction from these challenging transcripts.

Methods

The authors propose a two-stage approach to improve answer extraction:

  1. Heuristic Answer Extractor (Baseline): This initial module ranks candidate answers (identified Named Entities) based on seven hand-crafted heuristics. These heuristics measure factors like keyword proximity, sentence co-occurrence, and punctuation. The weights for these heuristics were manually optimized.
  2. Machine Learning Reranker: This layer takes the ranked candidates from the heuristic extractor and reranks them using a machine learning model.
  3. Heuristic Reranker: This version uses the seven heuristic scores, along with features like candidate repetition, length, and type, to train a binary Support Vector Machine (SVM). This approach automates the tuning of heuristic weights and can incorporate more information.
  4. Syntactic Reranker: This advanced version further enriches the feature set by incorporating syntactic information derived from dependency parsing. They parse both the document transcripts and the questions using an in-house dependency parser trained on written text. The core idea is to compare the syntactic paths between keywords and candidate answers in the text with the syntactic paths between keywords and the question's "focus word" (e.g., "where," "who"). This comparison helps identify candidates that are not just lexically close but also syntactically related in a way that mirrors the question's structure. Features derived from this comparison include the number of syntactically related keywords, the distance in syntactic relations, and the similarity of parsed paths.

Datasets

  • EPPS English Corpus: A collection of manual transcripts from 3 hours of European Parliament Plenary Sessions (approximately 35,000 words). This corpus is used for evaluating the QA system.
  • QAst (Question Answering on Speech Transcripts) Corpus: This corpus provides the questions used for evaluation. The authors specifically mention using the DEV set (50 questions) for training and the TEST set (100 questions) for evaluation. These questions are spontaneous oral questions that have been manually transcribed.

Key Insights and Ideas

  • Syntactic Structure Matters, Even with Noisy Text: The paper demonstrates that dependency parsing, despite being trained on clean text, can still provide valuable signals for improving answer extraction from noisy speech transcripts. The syntactic similarity between question and answer contexts proves to be a powerful feature.
  • Machine Learning Automates and Enhances Ranking: Moving from manually tuned heuristics to a machine learning reranker significantly improves performance by automating weight optimization and allowing for the incorporation of a richer set of features.
  • Syntactic Paths as a Robust Signal: Comparing dependency parse paths between question keywords and candidate answers in the text acts as a robust way to filter out spurious candidates that might be lexically close but semantically or syntactically disconnected.
  • Small Training Sets Can Yield Significant Gains: The experiments show that even a relatively small development set (50 questions) is sufficient to achieve a notable impact with the machine learning rerankers.

Key Results

  • The Heuristic Reranker improved MRR (Mean Reciprocal Rank) and Accuracy over the baseline heuristic extractor.
  • The Syntactic Reranker further improved MRR and Accuracy over the Heuristic Reranker, with both T1 (answer ranked first) and T5 (answer ranked anywhere in top 5) metrics showing improvement, indicating better precision and coverage.
  • The Syntactic Reranker achieved an accuracy comparable to the best system (LIMSI) in the QAst 2009 evaluation, despite being free of hand-crafted, language-dependent rules.
  • When using expanded training sets (combining DEV and TEST sets, excluding the question itself), both rerankers showed substantial improvements, with the Syntactic Reranker reaching 89% of the theoretical upper bound in coverage.

Limitations and Future Work

  • Parser Robustness: The authors acknowledge that the dependency parser, trained on written text, might have a high error ratio when applied to speech transcripts. They state that adapting the parser to speech is beyond the scope of this work, focusing instead on extracting robust features from its output.
  • White-box Evaluation: The paper notes that it's not possible to isolate the impact of the answer extraction module solely from the upstream modules (question processing and information retrieval) without a white-box evaluation, which is beyond the scope of their study.
  • Future Work: The authors suggest that their approach, which leverages machine learning and syntactic information, is easily adaptable to different tasks. The significant improvements achieved with relatively small training sets also point towards the potential for further gains with larger datasets.

Summary updated

BibTeX
@inproceedings{comas2010using,
  author = {Pere R. Comas and Jordi Turmo and Lluís Màrquez},
  title = {{Using dependency parsing and machine learning for factoid question answering on spoken documents}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure