Superclass Learning with Representation Enhancement

Zeyu Gan, Suyun Zhao, Jinlong Kang, Liyuan Shang, Hong Chen, Cuiping Li

Abstract

Standard image classification models typically rely on the "smoothness assumption," where visual similarity implies shared labels. However, real-world tasks often require grouping visually heterogeneous objects into broad "superclasses" (e.g., refuse sorting). We introduce Superclass Learning with Representation Enhancement (SCLRE), a framework that leverages Cross-Instance Attention (CIA) and a novel tripartite loss function to discover semantic links across visually diverse instances. Our results demonstrate that SCLRE significantly improves classification accuracy in superclass scenarios across multiple benchmarks, including CIFAR-100 and real-world datasets like FMoW.

Introduction

Most computer vision models excel at basic-level classification, such as distinguishing between specific species or objects. These models operate under the assumption that images with similar visual features should reside near each other in the embedding space. This assumption fails in the context of Superclasses. For instance, in waste management, "kitchen waste" encompasses items as visually distinct as eggshells, bones, and rotten fruit. Conversely, a "toy apple" (recyclable) may be visually indistinguishable from a "real apple" (kitchen waste).

Conceptual overview of the embedding space challenge where visually distinct items share superclass labels while mimics reside in different domains.

Traditional models struggle because sub-categories within a superclass are often scattered, and boundaries are blurred by visual mimics. SCLRE addresses this by shifting focus from individual image features to relational semantic features within a data batch.

Methodology

The SCLRE framework moves beyond isolated image processing by implementing two primary components:

Cross-Instance Attention (CIA)

Instead of extracting features from a single image, SCLRE employs a self-attention mechanism across an entire data batch. This allows the model to "mix" representations, facilitating the discovery of high-level semantic links between visually disparate items (e.g., a bone and an eggshell) that share a superclass label.

Tripartite Loss Strategy

To reshape the embedding space effectively, SCLRE utilizes three loss functions:

  1. Category Classification Loss ($L_{ce}$): Standard cross-entropy to maintain baseline predictive accuracy.
  2. Contrastive Adjustment Loss ($L_{ca}$): Encourages representations of the same superclass to cluster while pushing different superclasses apart.
  3. Targeted Adjustment Loss ($L_{ta}$): This key innovation defines fixed "target anchors" (category centers) in the embedding space. Samples are forced to migrate toward these anchors, preventing the "scattered" distribution typical of complex superclasses.
Comparison of (a) Image Feature Domain, (b) Separate Fine Domain, and (c) the learned Superclass Concept Domain where diverse features converge.

Experimental Evaluation

The authors evaluated SCLRE on several benchmarks reorganized to simulate superclass challenges:

  • CIFAR-100: Grouped into 3, 4, and 7 superclasses (e.g., "mammals," "household items").
  • High-Resolution Benchmarks: mini-ImageNet and VOC.
  • Real-World Datasets: Functional Map of the World (FMoW) for regional building style prediction and Adience for age group estimation.

Results and Discussion

SCLRE consistently outperformed baselines and standard contrastive learning methods. On CIFAR100-3, SCLRE achieved 80.1% accuracy, surpassing the ResNet baseline (72.8%) and SupCon (78.1%). Notably, on the complex FMoW dataset, SCLRE improved performance by over 8% (64.8% vs 56.5%), demonstrating robustness in non-idealized scenarios.

Visualizations using t-SNE confirmed that while baseline models produce scattered "clouds" for superclasses, SCLRE generates tight, well-separated clusters. This suggests that the combination of CIA and targeted anchors successfully collapses sub-category boundaries in favor of semantic super-structures.

Conclusion

SCLRE provides a theoretically grounded and practically effective approach to superclass learning. By breaking the reliance on individual visual features and utilizing batch-level attention, the model captures the underlying semantic essence of broad categories. Future work will focus on reducing batch dependency and exploring semi-supervised methods to discover these super-structures automatically without extensive expert labeling.

Summary updated

Abstract

In many real scenarios, data are often divided into a handful of artificial super categories in terms of expert knowledge rather than the representations of images. Concretely, a superclass may contain massive and various raw categories, such as refuse sorting. Due to the lack of common semantic features, the existing classification techniques are intractable to recognize superclass without raw class labels, thus they suffer severe performance damage or require huge annotation costs. To narrow this gap, this paper proposes a superclass learning framework, called SuperClass Learning with Representation Enhancement(SCLRE), to recognize super categories by leveraging enhanced representation. Specifically, by exploiting the self-attention technique across the batch, SCLRE collapses the boundaries of those raw categories and enhances the representation of each superclass. On the enhanced representation space, a superclassaware decision boundary is then reconstructed. Theoretically, we prove that by leveraging attention techniques the generalization error of SCLRE can be bounded under superclass scenarios. Experimentally, extensive results demonstrate that SCLRE outperforms the baseline and other contrastive-based methods on CIFAR-100 datasets and four high-resolution datasets.
BibTeX
@inproceedings{gan2023superclass,
  author = {Zeyu Gan and Suyun Zhao and Jinlong Kang and Liyuan Shang and Hong Chen and Cuiping Li},
  title = {{Superclass Learning with Representation Enhancement}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2023},
  doi = {10.1109/cvpr52729.2023.02304},
}
Figure