PivotAlign: Improve Semi-Supervised Learning by Learning Intra-Class Heterogeneity and Aligning with Pivots

Lingjie Yi, Tao Sun, Yikai Zhang, Songzhu Zheng, Weimin Lyu, Haibin Ling, Chao Chen

Semi-supervised learning (SSL) has seen significant progress by leveraging large amounts of unlabeled data alongside limited labeled samples. However, a persistent challenge in SSL is "class collapse," where models focus primarily on inter-class differences while ignoring the rich diversity within a single category. This WACV 2024 paper introduces PivotAlign, a framework designed to explicitly model intra-class heterogeneity to improve classification performance and robustness.

Addressing Intra-Class Heterogeneity

Most contemporary SSL methods prioritize inter-class heterogeneity—ensuring that an "airplane" is distinct from a "bird." This is typically achieved by pulling all samples of a class toward a single central prototype. While effective for simple cases, this approach often leads to class collapse, where fine-grained semantic information is lost.

Comparison between class-level representations (a) and hierarchical representations (b). Standard methods often collapse diverse intra-class samples into a single prototype, whereas PivotAlign maintains sub-prototypes (pivots) to capture variety.

As shown in Figure 1, an airplane viewed from different angles or a bird in flight versus perched can look vastly different. Forcing these diverse images into a single generic "airplane" bucket makes the model vulnerable to "hard" samples. For instance, a flying bird might be semantically closer to a generic airplane center than its own class center, leading to misclassification. PivotAlign addresses this by learning multiple sub-prototypes, or "pivots," per class.

The PivotAlign Framework

The PivotAlign architecture consists of two primary blocks: a Basic Block for standard feature extraction and an Alignment Block for hierarchical learning and debiasing.

The PivotAlign framework architecture, featuring the Basic Block for feature extraction and the Alignment Block for hierarchical representation learning and label refinement.

Hierarchical Representation Learning

Instead of a single center per class, PivotAlign maintains a set of pivots. The model employs an Online Pivot Assignment strategy using an Optimal Transport (OT) formulation, solved via the Sinkhorn-Knopp algorithm. This ensures that unlabeled samples are assigned to pivots in a balanced manner within each mini-batch, preventing the collapse of all samples into a single dominant pivot. A hierarchical contrastive loss ($L_c$) is then used to pull features toward their assigned pivots while maintaining separation between different sub-groups.

Label Refinement and Debiasing

SSL models are often biased toward "easy-to-learn" classes. PivotAlign mitigates this through a two-step process:

  1. Refinement: Pseudo-labels are refined by comparing samples to their nearest pivots, providing a more localized and accurate label estimate.
  2. Debiasing: To prevent the model from over-predicting dominant classes, another OT step is applied to force the distribution of pseudo-labels to remain balanced across the entire dataset.

Experimental Results

PivotAlign was evaluated on several standard SSL benchmarks, including CIFAR-10/100, STL-10, SVHN, and ImageNet-1k. The results demonstrate its superiority over existing state-of-the-art (SOTA) methods:

  • CIFAR-100: With only 400 labels, PivotAlign achieved 63.25% accuracy, significantly outperforming previous leaders like CoMatch and SimMatch (~58%).
  • ImageNet-1k: On the 1% labeled subset, the model reached 69.6% Top-1 accuracy, surpassing other cluster-based methods such as ProtoCon (65.5%) and Suave (66.2%).

The use of sub-prototypes allows the model to create more nuanced decision boundaries. t-SNE visualizations confirm that PivotAlign successfully groups sub-clusters (e.g., different viewpoints of a car) that standard methods like FixMatch often scatter randomly.

Conclusion

PivotAlign represents a shift in SSL from simple class-level prototypes to a more granular, hierarchical representation. By explicitly modeling intra-class heterogeneity and using Optimal Transport for balanced assignment and debiasing, it effectively handles ambiguous samples and alleviates class bias. While the number of pivots remains a hyperparameter, the framework's consistent performance across diverse datasets highlights the importance of capturing fine-grained details in semi-supervised visual recognition.

Summary updated

Abstract

Self-supervised learning plays an important role in current state-of-the-art semi-supervised learning (SSL) methods. These methods learn inter-class heterogeneity among data and generate pseudo-labels based on class level representations. However, they often neglect intra-class heterogeneity, resulting in the under-exploitation of finer-grained semantic relationships within classes. To address this limitation, we introduce PivotAlign, a novel SSL approach that aims to 1) learn hierarchical representations to detect both interclass and intra-class semantic relationships, and 2) refine pseudo-labels based on learned representations with a class-debiasing strategy. Specifically, we first learn a set of pivots as sub-prototypes of classes. We then train representations so that features align with the assigned pivot and are hierarchically grouped based on both inter-class and intra-class heterogeneity. This allows us to capture both inter-class and intra-class semantic relationships among data and leverage them to better assign and refine pseudo-labels. Additionally, since SSL methods are prone to bias toward classes that are easier to learn, we further re-balance class predictions to alleviate this class bias. We demonstrate the effectiveness of PivotAlign on various SSL benchmarks, where PivotAlign achieves state-of-the-art performances. The source code will be released upon publication of the work.
BibTeX
@inproceedings{yi2025pivotalign,
  author = {Lingjie Yi and Tao Sun and Yikai Zhang and Songzhu Zheng and Weimin Lyu and Haibin Ling and Chao Chen},
  title = {{PivotAlign: Improve Semi-Supervised Learning by Learning Intra-Class Heterogeneity and Aligning with Pivots}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
  doi = {10.1109/wacv61041.2025.00769},
}
Figure