PMI-cool at SemEval-2016 Task 3: Experiments with PMI and Goodness Polarity Lexicons for Community Question Answering

Daniel Balchev, Yasen Kiprov, Ivan Koychev, Preslav NakovSemEval @ NAACL 2016 · OpenReview

Introduction

Community Question Answering (cQA) platforms, such as Qatar Living, provide a wealth of information but often suffer from a high noise-to-signal ratio. Users frequently encounter long threads containing off-topic comments, insults, or simple acknowledgments that do not answer the original question. The PMI-cool system was developed for SemEval-2016 Task 3 (Subtask A) to address this by automatically reranking comments to prioritize high-quality answers.

Methodology

The core of the PMI-cool approach is the creation of Goodness Polarity Lexicons. Unlike traditional sentiment analysis, which categorizes words as "positive" or "negative," these lexicons identify words associated with "good" (informative) versus "bad" (useless) answers.

Feature Extraction

  1. Pointwise Mutual Information (PMI): The team calculated the Semantic Orientation (SO) of words based on their frequency in annotated "good" vs. "bad" answers.
  2. Bootstrapping: A semi-supervised method was used to expand the initial seed words into a larger, unannotated corpus of forum threads.
  3. Additional Features: The model incorporated metadata (e.g., chronological rank), general sentiment (using the Sentiment140 lexicon), and "Big Five" personality traits.

Learning Model

These features were integrated into a Support Vector Machine (SVM) regression model to predict a quality score for each answer, facilitating the final ranking.

Results and Analysis

The system's performance was evaluated using Mean Average Precision (MAP). An ablation study revealed the relative importance of each feature set.

Ablation study showing the impact of different feature sets on MAP.

As shown in the ablation study, removing the Word PMI features resulted in a substantial 6.6-point drop in MAP, confirming that statistical word associations are the primary driver of performance. Metadata also proved significant, reflecting the tendency for earlier answers in a thread to be more relevant. Conversely, personality traits did not contribute to model accuracy.

Competition Performance

In the official SemEval-2016 competition, PMI-cool placed 10th among primary runs. However, a post-submission "improved" version, which corrected a bootstrapping bug and refined the training process, achieved a MAP of 74.67.

Official SemEval-2016 Task 3 results and the PMI-cool improved run.

Key Insights

A critical finding was that "goodness" is distinct from "sentiment." For instance, polite phrases like "thanks" or "thanx" often appear in comments that do not provide new information, leading the model to assign them negative SO scores despite their positive sentiment. This highlights the necessity of domain-specific lexicons over general-purpose sentiment tools.

Conclusion

The PMI-cool experiments demonstrate that specialized goodness lexicons derived via PMI are highly effective for ranking cQA content. While metadata provides a strong baseline, the inclusion of statistical word-level features is essential for capturing the nuances of answer quality. Future work will explore thread-level context and deep learning architectures to further improve ranking precision.

Summary updated

BibTeX
@inproceedings{balchev2016pmi,
  author = {Daniel Balchev and Yasen Kiprov and Ivan Koychev and Preslav Nakov},
  title = {{PMI-cool at SemEval-2016 Task 3: Experiments with PMI and Goodness Polarity Lexicons for Community Question Answering}},
  booktitle = {SemEval @ NAACL},
  year = {2016},
  url = {https://openreview.net/forum?id=vUb73Qg69S},
}
Figure