Multiview semi-supervised ranking for automatic image annotation

Ali Fakeri-Tabrizi, Massih-Reza Amini, Patrick GallinariACM Multimedia 2013 · DOI

TL;DR

This paper introduces a novel multiview semi-supervised ranking approach to improve automatic image annotation, especially for datasets with limited labeled examples and imbalanced classes. By leveraging multiple "views" of image data and encouraging agreement among view-specific rankers on unlabeled data, the method significantly boosts annotation accuracy compared to existing techniques.

What Problem Does This Paper Address?

Manually labeling large image collections is a tedious and incomplete process. This results in datasets where many images have few or no labels, and some categories have very few representative examples (imbalanced classes). Existing automated image annotation systems struggle with these challenges. The paper aims to develop a more robust annotation method that can effectively learn from a small amount of labeled data and a large amount of unlabeled data, while also handling the inherent imbalance in real-world image datasets.

Methods

The core of the paper is a multiview semi-supervised bipartite ranking framework. Here's a breakdown:

  • Multiview Learning: Images are represented by multiple distinct feature sets (e.g., color histograms, texture features, text descriptions). Each feature set is considered a "view."
  • Semi-Supervised Learning: The method leverages both labeled and unlabeled data. It uses the labeled data to train initial "rankers" for each view and then uses unlabeled data to improve them.
  • Bipartite Ranking: Instead of directly classifying an image into a category, the system learns to rank images based on their relevance to a category. This is known to be more robust to class imbalance than traditional classification.
  • Agreement Mechanism: The key innovation is how unlabeled data is used. The system trains multiple view-specific rankers. It then encourages these rankers to agree on the relative ordering of unlabeled image pairs. If multiple rankers consistently rank one image higher than another for a specific category, that pair is "pseudo-labeled" and added to the training set. This agreement measure helps reduce disagreement between views and improves generalization.
  • Iterative Refinement: The process is iterative: rankers are trained, pseudo-labels are generated based on their agreement, and then rankers are retrained on the expanded dataset. This continues until the disagreement measure stops decreasing.

Datasets

The experiments were conducted on the NUS-WIDE-LITE dataset. This is a subset of the larger NUS-WIDE dataset, containing 28,807 images for training and 28,808 for testing, spanning 81 classes. The paper notes that this dataset is particularly useful for evaluating performance under class imbalance, as it contains both classes with high positive example percentages and highly imbalanced classes.

Key Insights and Ideas

  • Ranking is Better for Imbalance: The paper reinforces the idea that bipartite ranking is inherently more robust to class imbalance than traditional classification, a finding supported by previous work.
  • View Agreement as a Signal: The core insight is that forcing different "views" of the data to agree on the relative ordering of unlabeled examples is a powerful way to learn from unlabeled data and improve model performance. This agreement acts as a strong, implicit supervision signal.
  • Iterative Improvement is Key: The iterative nature of the algorithm, where pseudo-labels are generated and used to refine the rankers, allows for a progressive learning process that adapts to the data.
  • Handling Heterogeneous Data: The multiview approach is crucial for real-world scenarios where images are described by diverse types of features (visual, textual, etc.).

Key Results

The proposed method, R_SemSUP, consistently outperformed several baseline methods:

  • Significant Improvement: R_SemSUP achieved higher Accuracy and AUC scores compared to supervised baselines (R_SUP, C_SUP, R_Concat) and a semi-supervised multiview classification model (C_SemSUP).
  • Robustness to Imbalance: The improvements were particularly pronounced in scenarios with imbalanced classes, especially when the number of labeled examples was low.
  • Performance Across Labeled Set Sizes: The method showed strong performance across various labeled training set sizes (50, 100, 200, 1000), indicating its effectiveness even with very limited initial supervision.

Limitations and Future Work

  • Unanimity Requirement: The paper acknowledges that requiring strict unanimity among all view-specific rankers for pseudo-labeling might be too restrictive, especially with a large number of views. However, they found it worked well in practice.
  • Computational Cost: While the algorithm is designed to be efficient (O(n+m)), the paper mentions that a naive implementation of generating pseudo-labels over all pairs of unlabeled data would be computationally prohibitive (O(m²)). Their approach of sampling pairs mitigates this.
  • Future Directions: The authors suggest that the generic nature of their approach makes it applicable to other multi-modal image applications.

Summary updated

BibTeX
@inproceedings{fakeritabrizi2013multiview,
  author = {Ali Fakeri-Tabrizi and Massih-Reza Amini and Patrick Gallinari},
  title = {{Multiview semi-supervised ranking for automatic image annotation}},
  booktitle = {ACM Multimedia},
  year = {2013},
  doi = {10.1145/2502081.2502136},
}
Figure