On the Use of an Intermediate Class in Boolean Crowdsourced Relevance Annotations for Learning to Rank Comments

Alberto Barrón-Cedeño, Giovanni Da San Martino, Simone Filice, Alessandro Moschitti

Introduction

In the field of Information Retrieval (IR), the task of determining document relevance is rarely a simple binary choice. Human annotators often encounter "gray area" documents—those that contain some pertinent information but do not fully resolve a query. To accommodate this ambiguity, many benchmark datasets, particularly those from the SemEval community question answering (CQA) tasks, have historically included a third label: potential.

While intended to capture nuance, the inclusion of an intermediate class raises a critical question for machine learning practitioners: Does this extra granularity actually improve the performance of ranking models, or does it merely introduce noise and increase the complexity of the annotation process? This paper investigates the utility of the "potential" class in the context of Learning to Rank (LTR) comments in community forums.

Methodology

The researchers evaluated the impact of annotation schemes using a state-of-the-art ranking architecture: a kernel-based Support Vector Machine (SVM) classifier. This model leverages a hybrid approach:

  • Linear Kernels: These process traditional similarity features such as word embeddings, comment length, and thread-specific metadata (e.g., conversation flow).
  • Tree Kernels: These capture relational patterns by processing the structural and syntactic representations of question-answer pairs.

To isolate the effect of the intermediate class, the authors conducted two primary types of experiments. First, they performed a re-annotation of the CQA-QL-2016 test set. Using the CrowdFlower platform, they collected 10 binary judgments per instance for 93% of the original dataset, creating a "clean" binary ground truth. Second, they compared various training strategies, including mapping "potential" instances to "bad," discarding them entirely, or downweighting them.

The Ambiguity of the "Potential" Class

A core finding of the study is that the "potential" label often acts as a trap for annotators rather than a distinct category of relevance. When crowd workers were forced to re-annotate "potential" items using a binary scheme, the results showed a significant split.

Confusion matrix showing how 3-class labels map to binary labels during re-annotation.

As seen in the confusion matrix above, instances originally labeled as "potential" were split roughly 58% to "good" and 42% to "bad." This distribution suggests that the class is truly ambiguous and does not consistently represent a "weakly bad" or "weakly good" category. Instead, it likely captures instances where annotators are simply uncertain.

Experimental Results

The experimental results demonstrate that the additional complexity of a three-class system does not translate into superior model performance. In fact, simpler approaches often yielded better or equivalent results.

Performance comparison (MAP and Accuracy) across different training label strategies.

Key observations from the performance data include:

  1. Performance Parity: The 2-class annotation achieved a Mean Average Precision (MAP) of 85.13, which is statistically comparable to the more complex 3-class models.
  2. Addition by Subtraction: When working with existing 3-class data, simply discarding the "potential" instances (MAP 85.01) proved more effective than attempting to incorporate them by mapping them to the "bad" class (MAP 84.33).
  3. Efficiency Gains: By discarding the intermediate class, the training set size is reduced by approximately 10-17%. This leads to faster training and prediction cycles without sacrificing ranking quality.

Conclusion

The findings of this research suggest a clear path forward for those designing crowdsourcing tasks for IR: simplicity wins. Sticking to a binary (good/bad) scheme simplifies the cognitive load for annotators, reduces costs, and speeds up model training without harming the final ranker's accuracy. While intermediate classes might have a place in "graded relevance" schemes, for Boolean relevance tasks in community question answering, the "potential" label appears to be largely unnecessary.

Summary updated

Abstract

In many Information Retrieval tasks, the boundary between classes is not well defined, and assigning a document to a specific class may be complicated, even for humans. For instance, a document which is not directly related to the user's query may still contain relevant information. In this scenario, an option is to define an intermediate class collecting ambiguous instances. Yet some natural questions arise. Is this annotation strategy convenient? how should the intermediate class be treated? To answer these questions, we explored two community question answering datasets whose comments were originally annotated with three classes. We re-annotated a subset of instances considering a binary good vs bad setting. Our main contribution is to show empirically that the inclusion of an intermediate class to assess Boolean relevance is not useful. Moreover, in case the data is already annotated with a 3-class strategy, the instances from the intermediate class can be safely removed at training time.
BibTeX
@inproceedings{barroncedeno2017use,
  author = {Alberto Barrón-Cedeño and Giovanni Da San Martino and Simone Filice and Alessandro Moschitti},
  title = {{On the Use of an Intermediate Class in Boolean Crowdsourced Relevance Annotations for Learning to Rank Comments}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2017},
  doi = {10.1145/3077136.3080763},
}
Figure