Appearance-Based Curriculum for Semi-Supervised Learning with Multi-Angle Unlabeled Data

Yuki Tanaka, Shuhei M. Yoshida, Takashi Shibata, Makoto Terao, Takayuki Okatani, Masashi Sugiyama

Abstract

This paper introduces Appearance-Based Curriculum (ABC), a novel semi-supervised learning framework designed for scenarios where labeled data is restricted to specific viewpoints (like front-facing catalog images) while unlabeled data comes from diverse, unconstrained angles. By incrementally feeding unlabeled data into the training process—starting with images most similar to the labeled set—ABC maintains high pseudo-label quality and significantly outperforms state-of-the-art methods like FixMatch in multi-angle recognition tasks.

Introduction: The Challenge of Viewpoint Shift

In many practical computer vision applications, there is a massive distribution shift between labeled and unlabeled data. Consider product recognition: a company might have a "digital catalog" with one or two perfectly lit, front-facing labeled images per product. However, the unlabeled data collected from in-store security cameras features the same products from every possible angle, often obscured or rotated.

Standard Semi-Supervised Learning (SSL) algorithms, such as FixMatch or VAT, typically assume that unlabeled data follows the same distribution as labeled data. They rely on data augmentation to bridge gaps, but standard augmentations (like flipping or cropping) cannot transform a front-facing photo into a view from the back. When these models are trained on all unlabeled data simultaneously, they generate "noisy" (incorrect) pseudo-labels for the difficult angles, which eventually degrades the model's performance.

The Appearance-Based Curriculum (ABC) Framework

The authors propose a two-phase framework called Appearance-Based Curriculum (ABC) to bridge the gap between catalog photos and real-world views.

Overview of the Appearance-Based Curriculum (ABC) framework, showing curriculum construction and the incremental training process.
  1. Curriculum Construction: The unlabeled data is ranked from "easy" to "hard" based on its similarity to the labeled set.
  2. Training Phase: Instead of using all unlabeled data at once, the model (using a base SSL algorithm like FixMatch) is trained in steps. It starts with the most similar unlabeled images. Once the model reaches a performance plateau (the "Feed-on-Plateau" criterion), the next batch of slightly more "difficult" (dissimilar) images is added to the training pool.

This "easy-to-hard" approach ensures that the model first learns to classify images that look like the catalog photos, then uses that knowledge to accurately pseudo-label images from slightly different angles, gradually working its way around the object.

Distance Functions for Ranking

The authors propose three distance functions ($d$) to measure the similarity between labeled and unlabeled data, allowing the curriculum to be constructed effectively:

The three distance functions used for curriculum construction: d_angle (metadata), d_local (keypoints), and d_global (feature similarity).
  • $d_{angle}$: Uses actual camera angle metadata (elevation and azimuth) if available.
  • $d_{local}$: Uses hand-crafted local descriptors (AKAZE) to match keypoints between images. This works well when objects have distinct textures, such as beverage labels.
  • $d_{global}$: Uses a feature extractor pretrained on the small labeled set to compute cosine similarity between global image features.

Experimental Evaluation

The researchers evaluated ABC on three object recognition benchmarks: MIRO (120 objects, 160 angles), COIL-100 (100 objects, 72 angles), and DRINK, a newly introduced private dataset of 65 Japanese beverage brands.

Key Results

The results demonstrate that ABC provides a massive boost in accuracy over existing state-of-the-art SSL methods:

  • MIRO Dataset: ABC ($d_{global}$) achieved up to 86.5% accuracy, whereas the base FixMatch algorithm struggled at 65.4%.
  • DRINK Dataset: ABC ($d_{local}$) reached 81.4% accuracy, nearly doubling the performance of the supervised-only baseline (46.4%) and significantly beating FixMatch (48.3%).
  • Robustness: ABC showed it could handle various starting angles for the labeled data, consistently outperforming other methods regardless of whether the "front" or "top" of the object was labeled first.

Conclusion

ABC avoids the "pseudo-label quality trap" by "growing" the model's expertise. By the time the model sees a side-view, it has already learned from "near-front" views, making the side-view easier to classify. While the success of ABC relies heavily on choosing an appropriate distance function and involves some computational overhead due to multi-stage training, it offers a robust solution for real-world distribution shifts. Future work may investigate if this curriculum approach can be applied to other shifts, such as lighting or background changes.

Summary updated

Abstract

We propose an appearance-based curriculum (ABC) for a semi-supervised learning scenario where labeled images taken from limited angles and unlabeled ones taken from various angles are available for training. A common approach to semi-supervised learning relies on pseudo-labeling and data augmentation, but it struggles with large visual variations that cannot be covered by data augmentation. To solve this problem, ABC incrementally expands the pool of unlabeled images fed to a base semi-supervised learner so that newly added data are the ones most similar to those already in the pool. This way, the learner can assign pseudo-labels to the new data with high accuracy, keeping the quality of pseudo-labels higher than that when all the unlabeled data are processed at once, as customarily done in existing semi-supervised learning methods. We conducted extensive experiments and confirmed that our method outperforms the state-of-the-art semi-supervised learning methods in our scenario.
BibTeX
@inproceedings{tanaka2024appearance,
  author = {Yuki Tanaka and Shuhei M. Yoshida and Takashi Shibata and Makoto Terao and Takayuki Okatani and Masashi Sugiyama},
  title = {{Appearance-Based Curriculum for Semi-Supervised Learning with Multi-Angle Unlabeled Data}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00276},
}
Figure