Introduction
Web forums serve as vast repositories of human knowledge, yet their unstructured nature often obscures the most valuable information. A single discussion thread may contain dozens of posts, but only a few typically hold the "nugget" of information a user seeks. Standard search engines frequently treat an entire thread as a single document, making it difficult to pinpoint specific answers or filter out "junk" posts like simple expressions of gratitude or "me too" comments.
This research addresses the challenge of automated post classification: identifying the functional role of every message within a thread. By distinguishing between questions, solutions, and feedback, systems can significantly improve search ranking, automated summarization, and question-answering capabilities.
Methodology
The researchers approached post classification as a supervised learning task, evaluating several algorithms including Support Vector Machines (SVM), Naive Bayes, and Decision Trees. Ultimately, a Logit Model (Logistic Regression) yielded the best performance.
The core of the approach involves a multi-dimensional feature set categorized into four primary groups:
- Content-Based Features: These measure the similarity between a post and the thread's title or initial post, and check for "5W1H" words (Who, What, Where, When, Why, How).
- Structural Features: These analyze the absolute and relative position of a post within the thread timeline, as well as its length.
- User Features: These track the "authority" of the poster, their total post history, and whether they are the original thread starter.
- Sentiment Features: Using the SentiStrength algorithm, the model extracts emotional tone and looks for specific keywords like "thank" or "did not."
Dataset and Labeling
The study utilized a dataset of 100 threads (comprising 556 posts) sampled from the Ubuntu Forums. Three human evaluators manually categorized these posts into eight distinct classes: Question, Repeat Question, Clarification, Further Details, Solution, Positive Feedback, Negative Feedback, and Junk.
To ensure high data quality, the authors only retained posts where at least two evaluators agreed on the label, resulting in a gold-standard dataset of 529 posts. A notable innovation in this work was the inclusion of the "Junk" category, acknowledging that a significant portion of forum data contains no actionable information.
Results and Discussion
The system achieved an overall classification accuracy of 72.02%. The model was particularly effective at identifying the most critical components for information retrieval: Questions (F1-score: 0.856) and Solutions (F1-score: 0.810).
Key insights from the feature analysis include:
- Content Dominance: Content-based features were the strongest predictors. Specifically, similarity to the initial post (InitPostSim) was the most influential feature.
- Positional Context: Structural features proved highly effective, as questions typically appear at the start of a thread while solutions and feedback gravitate toward the end.
- Sentiment Limitations: Sentiment features were the least effective, largely due to data imbalance; for instance, there were only 11 instances of "Negative Feedback," making it difficult for the model to learn those nuances.
Conclusion
While the model demonstrates strong performance in identifying questions and solutions, limitations remain regarding dataset size and domain specificity. Future work will focus on expanding the feature set to include complex linguistic patterns and testing the model on more diverse datasets, such as general-interest boards or travel forums, to improve the detection of rarer classes like clarifications and negative feedback.
Summary updated