Finding Problem Solving Threads in Online Forum

Zhonghua Qu, Yang LiuIJCNLP 2011 · OpenReview

Abstract: Zhonghua Qu, Yang Liu. Proceedings of 5th International Joint Conference on Natural Language Processing.

  1. Searching for answers in online forums is often a gamble; many "relevant" threads are long-winded dead ends. This paper proposes a two-step machine learning approach that first identifies the function of individual posts (e.g., problem, solution, feedback) and then uses that conversational structure to predict if a thread actually solves a problem, achieving an F1 score of approximately 85%.

Introduction

Technical forums are a primary resource for troubleshooting, yet standard keyword-based search engines often fail to distinguish between a thread that contains a working solution and one that is merely a collection of unresolved complaints. This paper addresses the lack of "informativeness" filtering by analyzing the conversational dynamics of forum threads to determine if a problem was successfully resolved. The authors argue that a thread's value is determined by its outcome—whether the user's problem was actually solved—which requires understanding the structural roles of individual posts within the conversation.

Methodology

The researchers developed a two-step classification pipeline to evaluate thread usefulness.

Post-Level Classification

Every post in a thread is categorized into one of four types: Problem, Solution, Good Feedback, or Bad Feedback. The authors compared a standard Naive Bayes (NB) classifier against a Hidden Markov Model (HMM). The HMM is particularly effective because it models the sequential dependencies of forum posts; for instance, a "Solution" is statistically more likely to follow a "Problem" than a "Good Feedback" post.

HMM structure for modeling post sequences

Thread-Level Classification

Once posts are labeled, a Maximum Entropy (MaxEnt) model decides if the thread is "useful." Key features include the number of solution posts, the frequency of positive feedback, and the "Author Post-End" metric, which measures how close the original author's last post is to the end of the thread. A resolved problem often ends shortly after the author acknowledges a solution.

Dataset and Evaluation

The study utilized a dataset of 20,000 threads crawled from the Oracle database support forum. A subset of 200 threads was manually annotated by two computer science students on a helpfulness scale of 1 to 5. Interestingly, the data showed a "U-shaped" distribution: threads were typically either very helpful (score of 5) or not helpful at all (score of 1), with very few intermediate cases.

Key Results

The HMM approach for post classification achieved an 83.17% F1 score, clearly outperforming the Naive Bayes baseline of 77.57%. This confirms that context and sequence are vital for understanding forum roles.

In the final thread classification task, the full system achieved an F1 score of 84.97% when using HMM-generated labels, significantly better than the rule-based baseline of 80.35%. When using "perfect" manual labels (the "Ref" classifier), the F1 score rose to 88.89%, suggesting that improvements in post-level AI will directly translate to more accurate thread filtering.

Performance comparison of thread classification methods

Conclusion and Future Work

This research demonstrates that modeling forum threads as structured dialogues provides a powerful signal for identifying helpful content. The primary limitation of the current study is the small annotated sample size, which constrained the complexity of the models. Future work will focus on joint optimization of the classification pipeline and exploring unsupervised methods to automatically derive post categories, reducing the need for manual annotation.

Summary updated

BibTeX
@inproceedings{qu2011finding,
  author = {Zhonghua Qu and Yang Liu},
  title = {{Finding Problem Solving Threads in Online Forum}},
  booktitle = {IJCNLP},
  year = {2011},
  url = {https://openreview.net/forum?id=YoTQpey79Mf},
}
Figure