Introduction
Community Question Answering (cQA) platforms, such as Qatar Living, provide a wealth of information but often suffer from a high noise-to-signal ratio. Users frequently encounter long threads containing off-topic comments, insults, or simple acknowledgments that do not answer the original question. The PMI-cool system was developed for SemEval-2016 Task 3 (Subtask A) to address this by automatically reranking comments to prioritize high-quality answers.
Methodology
The core of the PMI-cool approach is the creation of Goodness Polarity Lexicons. Unlike traditional sentiment analysis, which categorizes words as "positive" or "negative," these lexicons identify words associated with "good" (informative) versus "bad" (useless) answers.
Feature Extraction
- Pointwise Mutual Information (PMI): The team calculated the Semantic Orientation (SO) of words based on their frequency in annotated "good" vs. "bad" answers.
- Bootstrapping: A semi-supervised method was used to expand the initial seed words into a larger, unannotated corpus of forum threads.
- Additional Features: The model incorporated metadata (e.g., chronological rank), general sentiment (using the Sentiment140 lexicon), and "Big Five" personality traits.
Learning Model
These features were integrated into a Support Vector Machine (SVM) regression model to predict a quality score for each answer, facilitating the final ranking.
Results and Analysis
The system's performance was evaluated using Mean Average Precision (MAP). An ablation study revealed the relative importance of each feature set.
As shown in the ablation study, removing the Word PMI features resulted in a substantial 6.6-point drop in MAP, confirming that statistical word associations are the primary driver of performance. Metadata also proved significant, reflecting the tendency for earlier answers in a thread to be more relevant. Conversely, personality traits did not contribute to model accuracy.
Competition Performance
In the official SemEval-2016 competition, PMI-cool placed 10th among primary runs. However, a post-submission "improved" version, which corrected a bootstrapping bug and refined the training process, achieved a MAP of 74.67.
Key Insights
A critical finding was that "goodness" is distinct from "sentiment." For instance, polite phrases like "thanks" or "thanx" often appear in comments that do not provide new information, leading the model to assign them negative SO scores despite their positive sentiment. This highlights the necessity of domain-specific lexicons over general-purpose sentiment tools.
Conclusion
The PMI-cool experiments demonstrate that specialized goodness lexicons derived via PMI are highly effective for ranking cQA content. While metadata provides a strong baseline, the inclusion of statistical word-level features is essential for capturing the nuances of answer quality. Future work will explore thread-level context and deep learning architectures to further improve ranking precision.
Summary updated