Semi-Distantly Supervised Neural Model for Generating Compact Answers to Open-Domain Why Questions

Ryo Ishida, Kentaro Torisawa, Jong-Hoon Oh, Ryu Iida, Canasai Kruengkrai, Julien KloetzerAAAI 2018 · DOI

Modern Question Answering (QA) systems often struggle with "why" questions, typically retrieving long, redundant paragraphs rather than the concise, single-sentence answers required for voice assistants like Amazon Echo or Google Home. This paper, presented at AAAI-18, proposes a neural model designed to generate these "compact answers" by leveraging a novel architecture and a massive semi-distantly supervised dataset.

The Challenge of Compact Why-QA

Most open-domain QA systems rely on retrieval, returning relevant passages from a database. While effective for human readers, these passages often contain redundant context. For instance, in answering "Why did Yubari City go bankrupt?", a retrieved paragraph might repeat the question's premise (the "effect") multiple times.

Existing summarization methods often fail in this context because "why" answers are unique: the words in the question are usually redundant in the answer, yet the model must use the question to identify the specific "cause" within a multi-causal passage. Furthermore, standard span-based models like DrQA are insufficient because optimal answers often require generative capabilities to stitch together disparate words or adjust verb conjugations.

Methodology

The authors extend the Pointer-Generator Network—an architecture capable of both generating new vocabulary and copying words from the source—with two key innovations:

1. Question Word Marking (QWM)

The encoder is modified to "mark" words in the answer passage that also appear in the question. By feeding this overlap information directly into the LSTM, the model learns to distinguish between the "effect" (already known from the query) and the "cause" (the target answer). In Why-QA, question words often act as "negative signals" that should be excluded from the final compact answer to avoid redundancy.

2. Semi-Distantly Supervised Learning

To overcome the scarcity of human-labeled data, the researchers utilized a supervised causality recognizer to scan a 4-billion-page web archive. They extracted 101 million causality expressions to create a massive pre-training signal.

Overview of the proposed neural model architecture and the semi-distantly supervised training pipeline.

To teach the model to filter irrelevant information, they employed a "Virtual Passage" strategy. By merging three random causal relations into a single block of text, they forced the model to use the question as a filter to ignore the two irrelevant causes and extract only the correct one.

Experimental Results

The model was evaluated using a manual dataset of 19,639 triples and the auto-generated millions. The results demonstrated that the proposed method significantly outperforms both non-neural and neural baselines.

Performance comparison of the proposed method against various baselines using ROUGE metrics.

As shown in the table above, the "Proposed+250K" model (utilizing QWM and 250,000 auto-generated samples) achieved a ROUGE-L of 44.4, a substantial improvement over the base Pointer-Generator (36.0) and DrQA (23.4). Human evaluations further supported these findings, with the proposed model receiving a "Good" fluency rating 65.7% of the time, compared to 52.7% for the baseline.

Conclusion and Future Work

The study proves that semi-distant supervision is a viable substitute for expensive manual labeling in the Why-QA domain. While the model excels at generating fluent, compact answers, a remaining challenge is "informativeness." Human evaluators noted that without the original passage for context, many compact answers felt incomplete (only 44.1% were rated "Good" for informativeness in isolation). Future work may focus on improving the semantic completeness of these generated summaries and expanding the approach beyond the Japanese language.

Summary updated

BibTeX
@inproceedings{ishida2018semi,
  author = {Ryo Ishida and Kentaro Torisawa and Jong-Hoon Oh and Ryu Iida and Canasai Kruengkrai and Julien Kloetzer},
  title = {{Semi-Distantly Supervised Neural Model for Generating Compact Answers to Open-Domain Why Questions}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2018},
  doi = {10.1609/aaai.v32i1.12064},
}
Figure