Abstract
In multi-label active learning, the primary focus has historically been on instance selection—identifying which data points are most informative for a human to label. This paper shifts the focus to the query type, arguing that the nature of the question asked is just as critical. We propose AURO (Active qUery on Relevance Ordering), a framework that asks annotators to rank the relative relevance of label pairs rather than providing absolute binary assignments. Our results across 20 benchmark datasets demonstrate that AURO significantly outperforms state-of-the-art methods, proving that relative ranking is a more efficient and informative query mechanism for multi-label tasks.
1. Introduction
Active learning aims to minimize labeling costs by strategically selecting the most informative examples for human annotation. In multi-label learning, where an instance (e.g., an image) can belong to multiple categories simultaneously, the labeling burden is particularly high. Traditional approaches require annotators to verify every possible label for a selected instance, which is both time-consuming and prone to error.
While existing research has optimized which images to show an expert, it has largely ignored what to ask the expert. Standard queries—asking for all labels or a single instance-label pair—often ignore the rich correlational structure between labels. This paper introduces the concept that the query type itself is a vital lever for improving model performance.
2. The AURO Framework
The proposed framework, AURO (Active qUery on Relevance Ordering), replaces binary "yes/no" questions with pairwise relevance ranking.
2.1 Query Mechanism
Instead of asking "Is this a lion?", AURO presents two labels, $y_1$ and $y_2$, for a given instance $x$ and asks the annotator to determine their relative relevance. The annotator has three choices:
- $y_1$ is more relevant than $y_2$.
- $y_2$ is more relevant than $y_1$.
- Both labels are irrelevant.
2.2 Ranking Model and Selection Strategy
AURO utilizes a label ranking model that learns to order relevant labels above irrelevant ones. It incorporates a "dummy label" ($y_0$) to serve as a threshold: labels ranked above $y_0$ are predicted as relevant, while those below are irrelevant.
To maximize information gain, AURO employs a specific selection strategy:
- Instance Selection: It picks the least-queried instance to ensure broad exploration of the data space.
- Label Selection: It selects $y_1$ (the label the model is most confident is relevant) and $y_2$ (the label the model is most uncertain about, i.e., closest to the $y_0$ threshold). This forces the model to resolve its greatest ambiguities regarding label boundaries.
3. Experimental Evaluation
The authors evaluated AURO against state-of-the-art active learning methods (QUIRE, AUDI, MMC) on 20 benchmark datasets. These datasets spanned diverse domains, including image classification (Corel5K), text categorization (Reuters), and biological data (Genebase). A key component of the evaluation was the MSRA dataset, which provides human-ranked ground truth for 1,868 images across 19 labels, allowing for a direct assessment of ranking accuracy.
4. Key Results and Insights
AURO consistently outperformed all baseline methods across all 20 datasets. Statistical t-tests at a 95% confidence level showed that AURO beat the MMC algorithm in 98% of test cases and QUIRE in 87% of cases.
4.1 Query Type Dominance
A striking finding was that even a version of AURO with random instance selection (AURO-r) often outperformed sophisticated instance selection methods using traditional queries. This reinforces the paper's core thesis: the query type is more important than the selection criterion.
4.2 Human Efficiency and Label Correlation
While a pairwise query takes slightly longer for a human to answer than a single-label query (1.43s vs 1.38s), the information density is significantly higher. By asking for relative order, the query inherently captures label correlations that independent binary queries miss. Furthermore, ranking is often cognitively easier for humans than absolute classification, especially in ambiguous or complex domains.
5. Conclusion
AURO demonstrates that pairwise relevance ordering is a powerful query type for multi-label active learning. By focusing on how we ask questions, we can build more accurate models with significantly less human effort. Future work will explore combining AURO with more advanced instance selection techniques and scaling pairwise queries to datasets with extremely large label sets.
Summary updated