Learning by Actively Querying Strong Modal Features

Yang Yang, De-Chuan Zhan, Yuan Jiang

ACQUEST (ACtive QUErying STrong modalities) is a novel training strategy that reduces the cost of multi-modal learning by selectively querying expensive "strong" features instead of human-provided labels. By using an "inverse prediction" technique to bridge the gap between features and labels, it achieves higher classification accuracy with lower overall data acquisition costs compared to traditional active learning.

The Multi-Modal Cost Challenge

In machine learning, we often have "multi-modal" data—information coming from different sources. "Strong" modalities (like high-resolution CT scans) are highly informative but expensive to collect or process. "Weak" modalities (like low-resolution X-rays) are cheap but less accurate. Traditional active learning tries to save money by asking humans to label only the most informative examples. However, human labels are often the most expensive "feature" of all.

ACQUEST asks: Instead of querying labels, can we actively query the strong features of a few key instances to help the model learn the relationship between cheap and expensive data?

The ACQUEST Method

ACQUEST treats the acquisition of strong features as an active selection process. It relies on three core components:

  • Inverse Prediction: The authors propose that strong features and ground-truth labels are intrinsically linked. They develop a technique to "invert" this relationship, allowing the model to treat strong features as a proxy for labels within a unified optimization framework.
  • Active Selection Criterion: Rather than picking instances where the label is uncertain, ACQUEST selects instances that are both informative (close to the decision boundary) and representative (helpful for mapping weak features to strong ones).
  • Unified Optimization: The model minimizes a loss function that accounts for classification error using weak features and the "reconstruction" error between predicted and actual queried strong features.

Once trained, the model only requires the cheap, weak modalities to make predictions, as it has learned to leverage the underlying structure of the strong modalities.

Experimental Evaluation

The researchers evaluated ACQUEST on 12 image classification tasks derived from three major datasets: NUS-WIDE, MSRA-MM, and Animal (ANIM). In these tests, Color Histograms were typically treated as the "weak" modality, while more complex features like PHOG (Pyramid Histogram of Oriented Gradients) were treated as "strong" modalities.

Learning curves across 12 datasets comparing ACQUEST with baseline methods.

Key Results and Insights

ACQUEST outperformed five state-of-the-art active learning methods (like QUIRE and DUAL) and a specialized multi-modal method (ARM) on 8 out of 12 real-world datasets. As shown in the learning curves above, ACQUEST often reaches high accuracy within just 10 queries, whereas competing methods require significantly more data or remain unstable.

Final classification accuracy (mean ± standard deviation) for all methods across the 12 tasks.

Key takeaways include:

  • Features as "Soft" Labels: Strong features can be cheaper than human oracles but more informative than raw weak data.
  • Bridging Modalities: By querying strong features for selected instances, the model learns a mapping that allows it to "hallucinate" the benefits of strong features at test time using only weak inputs.
  • Stability: Unlike some active learning methods that fluctuate, ACQUEST showed a stable performance increase, suggesting the "inverse prediction" provides a robust regularization effect.

Limitations and Future Work

While promising, the current ACQUEST model assumes a linear relationship between modalities and labels, which may not capture complex, non-linear dependencies in data like raw video. Additionally, the experimental validation focused on binary classification; extending the framework to multi-class applications and developing automated ways to identify "strong" features are key directions for future research.

Summary updated

Abstract

Complex objects are usually with multiple modal features. In multi-modal learning, modalities closely related to the target tasks are known as strong modalities. While collecting strong modalities of all instances is often expensive, and current multi-modal learning techniques hardly take the strong modal feature extraction expenses into consideration. On the other hand, active learning is proposed to reduce the labeling expenses by querying the ground truths for specific selected instances. In this paper, we propose a training strategy, ACQUEST (ACtive QUErying STrong modalities), which exploits strong modal information by actively querying the strong modal feature values of "selected" instances rather than their corresponding ground truths. In ACQUEST, only the informative instances are selected for strong modal feature acquisition. An inverse prediction technique is also proposed to make the ACQUEST a unified optimization form. Experiments on image datasets show that ACQUEST achieves better classification performance than conventional active learning and multi-modal learning methods with less feature acquisition costs and labeling expenses.
BibTeX
@inproceedings{yang2016learning,
  author = {Yang Yang and De-Chuan Zhan and Yuan Jiang},
  title = {{Learning by Actively Querying Strong Modal Features}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2016},
}
Figure