Semi-Supervised Answer Extraction from Discussion Forums

Rose Catherine, Rashmi Gangadharaiah, Karthik Visweswariah, Dinesh RaghuIJCNLP 2013 · OpenReview

Abstract

Rose Catherine, Rashmi Gangadharaiah, Karthik Visweswariah, Dinesh Raghu. Proceedings of the Sixth International Joint Conference on Natural Language Processing.

2013.

Introduction

Online discussion forums represent a vast repository of human knowledge, particularly for technical troubleshooting. However, extracting specific solutions from these threads is challenging due to their multi-party nature, which includes digressions, clarifications, and multiple suggested answers. Traditional supervised methods require extensive manual labeling for every new domain, while unsupervised methods often fail when a correct answer is unique rather than repeated. This paper proposes a semi-supervised framework that leverages a small amount of labeled data and a large pool of unlabeled data to identify answers effectively.

Example of a forum thread with labeled components including questions, answer suggestions, and acknowledgments.

Methodology

The proposed system utilizes a Co-Training framework, which trains two models in parallel using independent feature sets ("views") that reinforce each other over multiple iterations.

Feature Views

  1. STRUCT View: Utilizes metadata and thread structure, including author reputation, post position, and community ratings.
  2. PATTERN View: Focuses on linguistic structures. Using the PrefixSpan sequential pattern mining algorithm, the model identifies common answer-oriented phrasing (e.g., "Try [verb] your [noun]") based on Part-of-Speech (POS) tags. This allows the model to generalize across different technical topics.

Parallel Co-Training (ANS-ACK PCT)

The primary innovation is the introduction of a secondary task: Acknowledgment Extraction. The authors observed that user acknowledgments (e.g., "Thanks, that worked!") provide a strong signal for identifying the preceding answer. The framework runs two co-training instances—one for answers and one for acknowledgments—in parallel. The acknowledgment classifier provides features to the answer classifier, creating a mutual reinforcement loop that bootstraps performance from minimal initial labels.

The Parallel Co-training (ANS-ACK PCT) framework showing the interaction between Answer and Acknowledgment co-training instances.

Experimental Setup and Results

The model was evaluated using a dataset of 140,000 threads crawled from Apple Discussions. A subset of 303 threads was manually annotated for evaluation. To test the efficiency of the semi-supervised approach, the system was trained using only three labeled threads, with the remaining 300 serving as the test set.

The ANS-ACK PCT model significantly outperformed all baselines:

Method F-Score
Supervised SVM (3 threads) 28.6%
Unsupervised (CONG et al.) 38.7%
ANS CT (Basic Co-training) 55.6%
ANS-ACK PCT (Parallel Co-training) 67.8%

The parallel co-training method achieved a 137% improvement over the supervised baseline and a 75% improvement over the state-of-the-art unsupervised method. Most of the performance gains were realized within the first 10 iterations.

Conclusion

This research demonstrates that identifying the reaction to a post is a powerful proxy for identifying the answer itself. By integrating acknowledgment detection into a semi-supervised co-training framework, high accuracy can be achieved with negligible manual labeling. Future work includes refining extraction to the sentence level and exploring the generalizability of these patterns to non-technical domains.

Summary updated

BibTeX
@inproceedings{catherine2013semi,
  author = {Rose Catherine and Rashmi Gangadharaiah and Karthik Visweswariah and Dinesh Raghu},
  title = {{Semi-Supervised Answer Extraction from Discussion Forums}},
  booktitle = {IJCNLP},
  year = {2013},
  url = {https://openreview.net/forum?id=il-xZ2Az1L},
}
Figure