Hardness Sampling for Self-Training Based Transductive Zero-Shot Learning

Bo Liu, Qiulei Dong, Zhanyi HuCVPR 2021 · DOI

Abstract

Transductive Zero-Shot Learning (T-ZSL) leverages unlabeled data from unseen classes to mitigate the domain shift between training and test distributions. This paper introduces Self-Training with Hardness Sampling (STHS), a framework that iteratively trains on the "hardest" unlabeled samples. Contrary to the common practice of selecting high-confidence "easy" samples, we demonstrate that samples from classes the model finds difficult to predict carry less label noise and provide more critical information for bridging the domain gap. STHS achieves state-of-the-art performance on AWA2, CUB, and SUN benchmarks.

Introduction

Zero-Shot Learning (ZSL) challenges models to recognize objects from classes never seen during training by utilizing semantic attributes. The primary obstacle is the domain shift: models optimized on "seen" classes often fail to generalize to the visual distribution of "unseen" classes. Transductive ZSL (T-ZSL) addresses this by providing access to unlabeled images from unseen classes during training.

The core challenge in T-ZSL is determining which unlabeled samples to trust for self-training. Most existing methods prioritize "easy" samples—those with high prediction confidence. However, this paper argues that this approach is suboptimal and proposes that "hard" samples are actually more beneficial for model adaptation.

The STHS Framework

The proposed STHS (Self-Training with Hardness Sampling) framework iteratively selects a subset of unlabeled data to incorporate into the training set via pseudo-labeling. The selection process is governed by several key components:

  1. Uneven Prediction Analysis: Initial models naturally exhibit varying performance across unseen classes, identifying some as inherently "harder" than others.
  2. Class-Frequency Based Selection (CFBS): This algorithm identifies hard classes by monitoring prediction frequency; classes that are rarely predicted are designated as hard.
  3. Prior-Normalized CFBS (PN-CFBS): To account for dataset imbalance, the framework uses a Classification-Clustering Consistency (3C) method. This combines unsupervised clustering with model predictions to estimate class priors, ensuring that "hardness" isn't merely a reflection of low sample count.
  4. Iterative Self-Training: Starting from an inductive ZSL baseline, the model repeatedly selects the hardest samples, assigns pseudo-labels, and retrains to refine the visual-semantic mapping.

The Hard-Class Advantage and the Noise Paradox

A central finding of this work is the Hard-Class Advantage. Training on classes with low initial accuracy forces the model to drastically adjust its mapping between visual features and semantic descriptions, effectively "pulling" the model's distribution toward the unseen domain.

Analysis of per-class accuracy and the performance gap between easy and hard classes across different baseline models.

Furthermore, the authors identify a Noise Paradox: pseudo-labels for hard classes are often less noisy than those for easy classes. In ZSL, "easy" classes frequently act as "sinks" that attract incorrect predictions from various other categories, whereas hard classes, though rarely predicted, tend to be predicted correctly when they are selected.

Experimental Results

The STHS framework was evaluated on three standard benchmarks: AWA2 (Animals with Attributes 2), CUB (Birds), and SUN (Scenes). The researchers also introduced a "Strict T-GZSL" setting to ensure a more realistic evaluation where the model does not know whether a test sample belongs to a seen or unseen class.

Comparison of accuracy and precision trends over iterative self-training steps using Hard vs. Easy sampling strategies.

Key performance highlights include:

  • State-of-the-Art Accuracy: STHS-WGAN achieved 94.9% accuracy on the AWA2 dataset in conventional ZSL settings.
  • Generalized ZSL (GZSL): STHS-WGAN improved the Harmonic Mean (balancing seen and unseen accuracy) by approximately 4.5% on AWA2 and 4.9% on CUB compared to previous leading methods.
  • Robustness: The PN-CFBS method proved vital for maintaining performance on unbalanced datasets, highlighting the importance of class distribution priors.

Conclusion and Future Work

STHS demonstrates that intentionally training on "hard" samples is a superior strategy for transductive self-training in ZSL. By focusing on the most challenging classes, the model more effectively overcomes domain shift and reduces the impact of label noise. While the current framework relies on approximate prior estimation via clustering, future research could explore more robust methods for prior estimation in fully unlabeled settings or extend the "hardness" philosophy to broader semi-supervised learning tasks.

Summary updated

BibTeX
@inproceedings{liu2021hardness,
  author = {Bo Liu and Qiulei Dong and Zhanyi Hu},
  title = {{Hardness Sampling for Self-Training Based Transductive Zero-Shot Learning}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2021},
  doi = {10.1109/cvpr46437.2021.01623},
}
Figure