Multi-Label Active Learning: Query Type Matters

Sheng-Jun Huang, Songcan Chen, Zhi-Hua Zhou

Abstract

In multi-label active learning, the primary focus has historically been on instance selection—identifying which data points are most informative for a human to label. This paper shifts the focus to the query type, arguing that the nature of the question asked is just as critical. We propose AURO (Active qUery on Relevance Ordering), a framework that asks annotators to rank the relative relevance of label pairs rather than providing absolute binary assignments. Our results across 20 benchmark datasets demonstrate that AURO significantly outperforms state-of-the-art methods, proving that relative ranking is a more efficient and informative query mechanism for multi-label tasks.

1. Introduction

Active learning aims to minimize labeling costs by strategically selecting the most informative examples for human annotation. In multi-label learning, where an instance (e.g., an image) can belong to multiple categories simultaneously, the labeling burden is particularly high. Traditional approaches require annotators to verify every possible label for a selected instance, which is both time-consuming and prone to error.

While existing research has optimized which images to show an expert, it has largely ignored what to ask the expert. Standard queries—asking for all labels or a single instance-label pair—often ignore the rich correlational structure between labels. This paper introduces the concept that the query type itself is a vital lever for improving model performance.

2. The AURO Framework

The proposed framework, AURO (Active qUery on Relevance Ordering), replaces binary "yes/no" questions with pairwise relevance ranking.

2.1 Query Mechanism

Instead of asking "Is this a lion?", AURO presents two labels, $y_1$ and $y_2$, for a given instance $x$ and asks the annotator to determine their relative relevance. The annotator has three choices:

  1. $y_1$ is more relevant than $y_2$.
  2. $y_2$ is more relevant than $y_1$.
  3. Both labels are irrelevant.
Comparison between traditional binary queries (a) and AURO's pairwise relevance queries (b).

2.2 Ranking Model and Selection Strategy

AURO utilizes a label ranking model that learns to order relevant labels above irrelevant ones. It incorporates a "dummy label" ($y_0$) to serve as a threshold: labels ranked above $y_0$ are predicted as relevant, while those below are irrelevant.

To maximize information gain, AURO employs a specific selection strategy:

  1. Instance Selection: It picks the least-queried instance to ensure broad exploration of the data space.
  2. Label Selection: It selects $y_1$ (the label the model is most confident is relevant) and $y_2$ (the label the model is most uncertain about, i.e., closest to the $y_0$ threshold). This forces the model to resolve its greatest ambiguities regarding label boundaries.

3. Experimental Evaluation

The authors evaluated AURO against state-of-the-art active learning methods (QUIRE, AUDI, MMC) on 20 benchmark datasets. These datasets spanned diverse domains, including image classification (Corel5K), text categorization (Reuters), and biological data (Genebase). A key component of the evaluation was the MSRA dataset, which provides human-ranked ground truth for 1,868 images across 19 labels, allowing for a direct assessment of ranking accuracy.

4. Key Results and Insights

AURO consistently outperformed all baseline methods across all 20 datasets. Statistical t-tests at a 95% confidence level showed that AURO beat the MMC algorithm in 98% of test cases and QUIRE in 87% of cases.

Performance comparison (Micro-F1) of AURO against baseline methods as a function of query cost.

4.1 Query Type Dominance

A striking finding was that even a version of AURO with random instance selection (AURO-r) often outperformed sophisticated instance selection methods using traditional queries. This reinforces the paper's core thesis: the query type is more important than the selection criterion.

4.2 Human Efficiency and Label Correlation

While a pairwise query takes slightly longer for a human to answer than a single-label query (1.43s vs 1.38s), the information density is significantly higher. By asking for relative order, the query inherently captures label correlations that independent binary queries miss. Furthermore, ranking is often cognitively easier for humans than absolute classification, especially in ambiguous or complex domains.

5. Conclusion

AURO demonstrates that pairwise relevance ordering is a powerful query type for multi-label active learning. By focusing on how we ask questions, we can build more accurate models with significantly less human effort. Future work will explore combining AURO with more advanced instance selection techniques and scaling pairwise queries to datasets with extremely large label sets.

Summary updated

Abstract

Active learning reduces the labeling cost by selectively querying the most valuable information from the annotator. It is essentially important for multilabel learning, where the labeling cost is rather high because each object may be associated with multiple labels. Existing multi-label active learning (MLAL) research mainly focuses on the task of selecting instances to be queried. In this paper, we disclose for the first time that the query type, which decides what information to query for the selected instance, is more important. Based on this observation, we propose a novel MLAL framework to query the relevance ordering of label pairs, which gets richer information from each query and requires less expertise of the annotator. By incorporating a simple selection strategy and a label ranking model into our framework, the proposed approach can reduce the labeling effort of annotators significantly. Experiments on 20 benchmark datasets and a manually labeled real data validate that our approach not only achieves superior performance on classification, but also provides accurate ranking for relevant labels.
BibTeX
@inproceedings{huang2015multi,
  author = {Sheng-Jun Huang and Songcan Chen and Zhi-Hua Zhou},
  title = {{Multi-Label Active Learning: Query Type Matters}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2015},
}
Figure