Abstract
Community Question Answering (CQA) platforms often suffer from a high volume of low-quality or irrelevant responses. This paper presents a ranking system based on Paul Grice’s Maxims of Conversation, designed for the SemEval-2017 Task 3 competition. By operationalizing the maxims of Quantity, Relation, and Manner through features such as named entity density, semantic similarity, and sentiment analysis, the system effectively identifies high-quality answers. Our approach significantly outperforms standard IR baselines, achieving a Mean Average Precision (MAP) of 0.7151 on the 2016 benchmark and 78.56 in the 2017 competition.
Introduction
In CQA forums like Qatar Living, users often encounter a mix of helpful answers, spam, and conversational filler. The task of relevancy ranking aims to automatically prioritize the most useful responses. While many contemporary systems utilize deep learning, this work explores the utility of classical linguistic theory—specifically Gricean pragmatics—as a framework for feature engineering in CQA ranking.
Methodology: Applying Grice’s Maxims
The core of the system is built upon three of the four Gricean Maxims, which describe the principles of cooperative conversation:
- Quantity (Be Informative): High-quality answers are expected to contain specific information. The system rewards the presence of Named Entities (e.g., locations, organizations), URLs, email addresses, and numerical values.
- Relation (Be Relevant): Relevance is captured through semantic overlap between question and answer using Word2Vec embeddings and Brown clusters. Additionally, the system identifies instructional cues (imperatives) and domain-specific terminology.
- Manner (Be Clear and Polite): This maxim is interpreted through sentiment. Cooperative answers are typically polite and professional, while unhelpful ones may be ironic or insulting. Sentiment analysis is used to reward positive tones and penalize negative ones.
The system also employs a heuristic to penalize responses from the original poster (OP), as these are typically clarifications rather than answers. The final score is a weighted combination of these rewards and penalties.
Datasets and Evaluation
The system was evaluated using the SemEval-2017 Task 3 (Subtask A) dataset, derived from the Qatar Living forum. The SemEval-2016 dataset served as the primary benchmark for internal tuning and experimentation. To enhance performance, the team developed a custom Named Entity Recognition (NER) model specifically for the Qatar Living domain, achieving an 87% F1-measure.
Key Results
Internal benchmarking on the 2016 test set demonstrated the effectiveness of the Gricean features. As shown in Figure 1, the final system ("Our System") achieved a MAP of 0.7151, representing a substantial improvement over the 0.5280 baseline.
In the official SemEval-2017 competition, the system reached a MAP of 78.56. While it outperformed the baseline IR system by approximately 6 points, it ranked 12th out of 13 participants. This suggests that while linguistic pragmatics provide a strong foundation, they may benefit from integration with more complex neural architectures to reach state-of-the-art performance.
Conclusion and Future Work
This research demonstrates that classical linguistic theories like Grice’s Maxims can be effectively operationalized for modern NLP tasks. The primary limitation remains the exclusion of the Quality maxim (truthfulness), which is critical for preventing the promotion of polite but incorrect information. Future work will focus on integrating automated fact-checking and expanding domain-specific lexicons to further refine the "Relation" maxim.
Summary updated