Semi-supervised learning (SSL) has seen significant progress by leveraging large amounts of unlabeled data alongside limited labeled samples. However, a persistent challenge in SSL is "class collapse," where models focus primarily on inter-class differences while ignoring the rich diversity within a single category. This WACV 2024 paper introduces PivotAlign, a framework designed to explicitly model intra-class heterogeneity to improve classification performance and robustness.
Addressing Intra-Class Heterogeneity
Most contemporary SSL methods prioritize inter-class heterogeneity—ensuring that an "airplane" is distinct from a "bird." This is typically achieved by pulling all samples of a class toward a single central prototype. While effective for simple cases, this approach often leads to class collapse, where fine-grained semantic information is lost.
As shown in Figure 1, an airplane viewed from different angles or a bird in flight versus perched can look vastly different. Forcing these diverse images into a single generic "airplane" bucket makes the model vulnerable to "hard" samples. For instance, a flying bird might be semantically closer to a generic airplane center than its own class center, leading to misclassification. PivotAlign addresses this by learning multiple sub-prototypes, or "pivots," per class.
The PivotAlign Framework
The PivotAlign architecture consists of two primary blocks: a Basic Block for standard feature extraction and an Alignment Block for hierarchical learning and debiasing.
Hierarchical Representation Learning
Instead of a single center per class, PivotAlign maintains a set of pivots. The model employs an Online Pivot Assignment strategy using an Optimal Transport (OT) formulation, solved via the Sinkhorn-Knopp algorithm. This ensures that unlabeled samples are assigned to pivots in a balanced manner within each mini-batch, preventing the collapse of all samples into a single dominant pivot. A hierarchical contrastive loss ($L_c$) is then used to pull features toward their assigned pivots while maintaining separation between different sub-groups.
Label Refinement and Debiasing
SSL models are often biased toward "easy-to-learn" classes. PivotAlign mitigates this through a two-step process:
- Refinement: Pseudo-labels are refined by comparing samples to their nearest pivots, providing a more localized and accurate label estimate.
- Debiasing: To prevent the model from over-predicting dominant classes, another OT step is applied to force the distribution of pseudo-labels to remain balanced across the entire dataset.
Experimental Results
PivotAlign was evaluated on several standard SSL benchmarks, including CIFAR-10/100, STL-10, SVHN, and ImageNet-1k. The results demonstrate its superiority over existing state-of-the-art (SOTA) methods:
- CIFAR-100: With only 400 labels, PivotAlign achieved 63.25% accuracy, significantly outperforming previous leaders like CoMatch and SimMatch (~58%).
- ImageNet-1k: On the 1% labeled subset, the model reached 69.6% Top-1 accuracy, surpassing other cluster-based methods such as ProtoCon (65.5%) and Suave (66.2%).
The use of sub-prototypes allows the model to create more nuanced decision boundaries. t-SNE visualizations confirm that PivotAlign successfully groups sub-clusters (e.g., different viewpoints of a car) that standard methods like FixMatch often scatter randomly.
Conclusion
PivotAlign represents a shift in SSL from simple class-level prototypes to a more granular, hierarchical representation. By explicitly modeling intra-class heterogeneity and using Optimal Transport for balanced assignment and debiasing, it effectively handles ambiguous samples and alleviates class bias. While the number of pivots remains a hyperparameter, the framework's consistent performance across diverse datasets highlights the importance of capturing fine-grained details in semi-supervised visual recognition.
Summary updated