Introduction
In the field of Information Retrieval (IR), the task of determining document relevance is rarely a simple binary choice. Human annotators often encounter "gray area" documents—those that contain some pertinent information but do not fully resolve a query. To accommodate this ambiguity, many benchmark datasets, particularly those from the SemEval community question answering (CQA) tasks, have historically included a third label: potential.
While intended to capture nuance, the inclusion of an intermediate class raises a critical question for machine learning practitioners: Does this extra granularity actually improve the performance of ranking models, or does it merely introduce noise and increase the complexity of the annotation process? This paper investigates the utility of the "potential" class in the context of Learning to Rank (LTR) comments in community forums.
Methodology
The researchers evaluated the impact of annotation schemes using a state-of-the-art ranking architecture: a kernel-based Support Vector Machine (SVM) classifier. This model leverages a hybrid approach:
- Linear Kernels: These process traditional similarity features such as word embeddings, comment length, and thread-specific metadata (e.g., conversation flow).
- Tree Kernels: These capture relational patterns by processing the structural and syntactic representations of question-answer pairs.
To isolate the effect of the intermediate class, the authors conducted two primary types of experiments. First, they performed a re-annotation of the CQA-QL-2016 test set. Using the CrowdFlower platform, they collected 10 binary judgments per instance for 93% of the original dataset, creating a "clean" binary ground truth. Second, they compared various training strategies, including mapping "potential" instances to "bad," discarding them entirely, or downweighting them.
The Ambiguity of the "Potential" Class
A core finding of the study is that the "potential" label often acts as a trap for annotators rather than a distinct category of relevance. When crowd workers were forced to re-annotate "potential" items using a binary scheme, the results showed a significant split.
As seen in the confusion matrix above, instances originally labeled as "potential" were split roughly 58% to "good" and 42% to "bad." This distribution suggests that the class is truly ambiguous and does not consistently represent a "weakly bad" or "weakly good" category. Instead, it likely captures instances where annotators are simply uncertain.
Experimental Results
The experimental results demonstrate that the additional complexity of a three-class system does not translate into superior model performance. In fact, simpler approaches often yielded better or equivalent results.
Key observations from the performance data include:
- Performance Parity: The 2-class annotation achieved a Mean Average Precision (MAP) of 85.13, which is statistically comparable to the more complex 3-class models.
- Addition by Subtraction: When working with existing 3-class data, simply discarding the "potential" instances (MAP 85.01) proved more effective than attempting to incorporate them by mapping them to the "bad" class (MAP 84.33).
- Efficiency Gains: By discarding the intermediate class, the training set size is reduced by approximately 10-17%. This leads to faster training and prediction cycles without sacrificing ranking quality.
Conclusion
The findings of this research suggest a clear path forward for those designing crowdsourcing tasks for IR: simplicity wins. Sticking to a binary (good/bad) scheme simplifies the cognitive load for annotators, reduces costs, and speeds up model training without harming the final ranker's accuracy. While intermediate classes might have a place in "graded relevance" schemes, for Boolean relevance tasks in community question answering, the "potential" label appears to be largely unnecessary.
Summary updated