Abstract
Rose Catherine, Rashmi Gangadharaiah, Karthik Visweswariah, Dinesh Raghu. Proceedings of the Sixth International Joint Conference on Natural Language Processing.
2013.
Introduction
Online discussion forums represent a vast repository of human knowledge, particularly for technical troubleshooting. However, extracting specific solutions from these threads is challenging due to their multi-party nature, which includes digressions, clarifications, and multiple suggested answers. Traditional supervised methods require extensive manual labeling for every new domain, while unsupervised methods often fail when a correct answer is unique rather than repeated. This paper proposes a semi-supervised framework that leverages a small amount of labeled data and a large pool of unlabeled data to identify answers effectively.
Methodology
The proposed system utilizes a Co-Training framework, which trains two models in parallel using independent feature sets ("views") that reinforce each other over multiple iterations.
Feature Views
- STRUCT View: Utilizes metadata and thread structure, including author reputation, post position, and community ratings.
- PATTERN View: Focuses on linguistic structures. Using the PrefixSpan sequential pattern mining algorithm, the model identifies common answer-oriented phrasing (e.g., "Try [verb] your [noun]") based on Part-of-Speech (POS) tags. This allows the model to generalize across different technical topics.
Parallel Co-Training (ANS-ACK PCT)
The primary innovation is the introduction of a secondary task: Acknowledgment Extraction. The authors observed that user acknowledgments (e.g., "Thanks, that worked!") provide a strong signal for identifying the preceding answer. The framework runs two co-training instances—one for answers and one for acknowledgments—in parallel. The acknowledgment classifier provides features to the answer classifier, creating a mutual reinforcement loop that bootstraps performance from minimal initial labels.
Experimental Setup and Results
The model was evaluated using a dataset of 140,000 threads crawled from Apple Discussions. A subset of 303 threads was manually annotated for evaluation. To test the efficiency of the semi-supervised approach, the system was trained using only three labeled threads, with the remaining 300 serving as the test set.
The ANS-ACK PCT model significantly outperformed all baselines:
| Method | F-Score |
|---|---|
| Supervised SVM (3 threads) | 28.6% |
| Unsupervised (CONG et al.) | 38.7% |
| ANS CT (Basic Co-training) | 55.6% |
| ANS-ACK PCT (Parallel Co-training) | 67.8% |
The parallel co-training method achieved a 137% improvement over the supervised baseline and a 75% improvement over the state-of-the-art unsupervised method. Most of the performance gains were realized within the first 10 iterations.
Conclusion
This research demonstrates that identifying the reaction to a post is a powerful proxy for identifying the answer itself. By integrating acknowledgment detection into a semi-supervised co-training framework, high accuracy can be achieved with negligible manual labeling. Future work includes refining extraction to the sentence level and exploring the generalizability of these patterns to non-technical domains.
Summary updated