Abstract
This paper investigates the effectiveness of Active Learning (AL) for fine-tuning neural rankers such as MonoBERT, DPR, and ColBERT. While AL is often used to reduce annotation costs in other NLP tasks, this study finds that current AL strategies do not consistently outperform simple random selection in ranking tasks. Furthermore, when accounting for the increased human effort required to annotate "hard" queries selected by AL, these strategies can be significantly more expensive than random sampling for the same level of performance.
Introduction
Fine-tuning neural rankers typically requires thousands of labeled query-document pairs, a process that is particularly costly in specialized domains requiring subject matter expertise. Active Learning (AL) has long been viewed as a potential solution to this "cold-start" problem by algorithmically identifying the most informative training samples. However, most AL research has focused on classification rather than ranking. This study evaluates whether AL can effectively reduce the annotation budget needed for high-performing neural rankers in both general (Scratch) and specialized (Re-Train) scenarios.
Methodology
The authors evaluated three neural architectures—MonoBERT (cross-encoder), DPR (bi-encoder), and ColBERT (late interaction)—against four data selection strategies:
- Random Selection (Baseline): Picking training subsets at random.
- Uncertainty-based: Selecting samples where the model is least confident.
- Query-by-Committee (QBC): Selecting queries where a committee of models disagrees most on the ranking.
- Diversity-based: Selecting a diverse set of query representatives via clustering.
The study utilized the MS MARCO dataset for general web search and the TripClick dataset for health-oriented domain adaptation. A "Budget-Aware Evaluation" framework was introduced to calculate total costs, including human annotation time (the rank of the first relevant document) and computational overhead.
Results and Discussion
The High-Yield Potential and the Difficulty Trap
The study revealed a massive variance in performance when using different random subsets of the same size. For instance, MonoBERT’s nDCG@10 at 1,000 samples could range from 0.27 to 0.46 based on the subset chosen, as illustrated in Figure 1. This confirms that "optimal" subsets exist, though current AL methods fail to find them reliably.
A major drawback of AL strategies like QBC and Uncertainty-based selection is their tendency to pick "hard" queries. In a ranking context, these queries often have relevant documents buried deep in the results, requiring annotators to screen many irrelevant documents. This "difficulty trap" makes these supposedly informative queries much more expensive to label than average ones.
Comparative Performance of AL Strategies
Across nearly all tests, AL strategies failed to significantly beat random selection in terms of ranking effectiveness (nDCG@10). As shown in Figure 5, when performance is plotted against the number of assessed query-passage pairs, random selection remains highly competitive and often superior.
Notably, DPR proved particularly resistant to AL, with random selection consistently performing best. In the ColBERT scenario, the marginal gains from using QBC were offset by an additional $200,000 in annotation effort compared to random selection for 50k samples.
Conclusion
The findings suggest that current AL strategies are poorly suited for neural ranking tasks. They fail to provide a clear performance advantage and often increase the total cost of annotation. Future work should focus on developing "cost-aware" selection strategies that balance potential information gain against the human effort required for labeling. The authors suggest that better uncertainty estimation, such as Monte-Carlo dropout, might be a necessary first step.
Summary updated