Annotating Data for Fine-Tuning a Neural Ranker? Current Active Learning Strategies are not Better than Random Selection

Sophia Althammer, Guido Zuccon, Sebastian Hofstätter, Suzan Verberne, Allan Hanbury

Abstract

This paper investigates the effectiveness of Active Learning (AL) for fine-tuning neural rankers such as MonoBERT, DPR, and ColBERT. While AL is often used to reduce annotation costs in other NLP tasks, this study finds that current AL strategies do not consistently outperform simple random selection in ranking tasks. Furthermore, when accounting for the increased human effort required to annotate "hard" queries selected by AL, these strategies can be significantly more expensive than random sampling for the same level of performance.

Introduction

Fine-tuning neural rankers typically requires thousands of labeled query-document pairs, a process that is particularly costly in specialized domains requiring subject matter expertise. Active Learning (AL) has long been viewed as a potential solution to this "cold-start" problem by algorithmically identifying the most informative training samples. However, most AL research has focused on classification rather than ranking. This study evaluates whether AL can effectively reduce the annotation budget needed for high-performing neural rankers in both general (Scratch) and specialized (Re-Train) scenarios.

Methodology

The authors evaluated three neural architectures—MonoBERT (cross-encoder), DPR (bi-encoder), and ColBERT (late interaction)—against four data selection strategies:

  • Random Selection (Baseline): Picking training subsets at random.
  • Uncertainty-based: Selecting samples where the model is least confident.
  • Query-by-Committee (QBC): Selecting queries where a committee of models disagrees most on the ranking.
  • Diversity-based: Selecting a diverse set of query representatives via clustering.

The study utilized the MS MARCO dataset for general web search and the TripClick dataset for health-oriented domain adaptation. A "Budget-Aware Evaluation" framework was introduced to calculate total costs, including human annotation time (the rank of the first relevant document) and computational overhead.

Results and Discussion

The High-Yield Potential and the Difficulty Trap

The study revealed a massive variance in performance when using different random subsets of the same size. For instance, MonoBERT’s nDCG@10 at 1,000 samples could range from 0.27 to 0.46 based on the subset chosen, as illustrated in Figure 1. This confirms that "optimal" subsets exist, though current AL methods fail to find them reliably.

Performance variance of neural rankers across different training data sizes in Scratch and Re-Train scenarios.

A major drawback of AL strategies like QBC and Uncertainty-based selection is their tendency to pick "hard" queries. In a ranking context, these queries often have relevant documents buried deep in the results, requiring annotators to screen many irrelevant documents. This "difficulty trap" makes these supposedly informative queries much more expensive to label than average ones.

Comparative Performance of AL Strategies

Across nearly all tests, AL strategies failed to significantly beat random selection in terms of ranking effectiveness (nDCG@10). As shown in Figure 5, when performance is plotted against the number of assessed query-passage pairs, random selection remains highly competitive and often superior.

Learning curves comparing Active Learning strategies against Random selection across different architectures and scenarios, showing nDCG@10 versus assessment cost.

Notably, DPR proved particularly resistant to AL, with random selection consistently performing best. In the ColBERT scenario, the marginal gains from using QBC were offset by an additional $200,000 in annotation effort compared to random selection for 50k samples.

Conclusion

The findings suggest that current AL strategies are poorly suited for neural ranking tasks. They fail to provide a clear performance advantage and often increase the total cost of annotation. Future work should focus on developing "cost-aware" selection strategies that balance potential information gain against the human effort required for labeling. The authors suggest that better uncertainty estimation, such as Monte-Carlo dropout, might be a necessary first step.

Summary updated

Abstract

Search methods based on Pretrained Language Models (PLM) have demonstrated great effectiveness gains compared to statistical and early neural ranking models. However, fine-tuning PLM-based rankers requires a great amount of annotated training data. Annotating data involves a large manual effort and thus is expensive, especially in domain specific tasks. In this paper we investigate fine-tuning PLM-based rankers under limited training data and budget. We investigate two scenarios: fine-tuning a ranker from scratch, and domain adaptation starting with a ranker already fine-tuned on general data, and continuing fine-tuning on a target dataset. We observe a great variability in effectiveness when fine-tuning on different randomly selected subsets of training data. This suggests that it is possible to achieve effectiveness gains by actively selecting a subset of the training data that has the most positive effect on the rankers. This way, it would be possible to fine-tune effective PLM rankers at a reduced annotation budget. To investigate this, we adapt existing Active Learning (AL) strategies to the task of fine-tuning PLM rankers and investigate their effectiveness, also considering annotation and computational costs. Our extensive analysis shows that AL strategies do not significantly outperform random selection of training subsets in terms of effectiveness. We further find that gains provided by AL strategies come at the expense of more assessments (thus higher annotation costs) and AL strategies underperform random selection when comparing effectiveness given a fixed annotation cost. Our results highlight that ``optimal'' subsets of training data that provide high effectiveness at low annotation cost do exist, but current mainstream AL strategies applied to PLM rankers are not capable of identifying them.
BibTeX
@inproceedings{althammer2023annotating,
  author = {Sophia Althammer and Guido Zuccon and Sebastian Hofstätter and Suzan Verberne and Allan Hanbury},
  title = {{Annotating Data for Fine-Tuning a Neural Ranker? Current Active Learning Strategies are not Better than Random Selection}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2023},
  doi = {10.1145/3624918.3625333},
}
Figure