Group-Wise Contrastive Bottleneck for Weakly-Supervised Visual Representation Learning

Boon Peng Yap, Beng Koon NgWACV 2024

TL;DR

This paper introduces CLRC, a novel framework for visual representation learning that leverages "weak" labels (like image captions or attributes) by grouping them semantically. CLRC uses a "group-wise contrastive bottleneck" to learn representations that are more robust and efficient, outperforming prior methods on various downstream tasks.

What Problem Does This Paper Address?

Training deep learning models often requires vast amounts of labeled data, which can be expensive and time-consuming to acquire. Weakly supervised learning aims to overcome this by using less precise labels, such as image captions or coarse object categories. However, existing methods often struggle to effectively utilize the inherent semantic relationships within these weak labels, especially when dealing with sparse or high-dimensional attribute data. This paper addresses the challenge of extracting rich, robust visual representations from such weak labels by explicitly modeling the semantic grouping of attributes.

Methods

The core of CLRC is a group-wise contrastive bottleneck module. This module takes a global image representation and disentangles it into multiple group-specific representations. These group-specific representations are then fed into a contrastive learning objective, where positive pairs are defined based on shared labels within each semantic group. This approach ensures a sufficient number of positive pairs, even with sparse labels.

To further enhance robustness, CLRC incorporates a reconstruction loss. The disentangled group representations are used to reconstruct the original global image representation. This reconstructed representation is then encouraged to be consistent with a representation from a differently augmented view of the same image. This process, inspired by autoencoders, helps learn more robust features and implicitly captures unobserved attributes. The overall loss function combines the group-wise contrastive loss and the reconstruction loss.

Datasets

The paper evaluates CLRC on several benchmark datasets:

  • UT Zappos 50k: Visual attributes dataset.
  • WIDER Attribute: Visual attributes dataset.
  • CUB-200-2011: Visual attributes dataset.
  • Fitzpatrick17k: Sensitive attributes (skin type) dataset.
  • ImageNet-100: Hierarchical labels dataset.

Key Insights and Ideas

  • Semantic Grouping is Key: The paper highlights that simply treating each weak label independently is suboptimal. Grouping related attributes (e.g., "red," "blue," "green" under "color") unlocks more effective representation learning.
  • Bottleneck for Disentanglement: The proposed bottleneck module effectively disentangles global features into group-specific representations, allowing for targeted contrastive learning.
  • Reconstruction for Robustness: The reconstruction loss acts as a powerful regularizer, forcing the model to learn representations that are invariant to augmentations and capture underlying semantic information.
  • Parameter Efficiency: CLRC's bottleneck module is shown to be more parameter-efficient compared to standard MLP projectors used in other contrastive learning methods.
  • Flexibility Across Label Types: CLRC demonstrates adaptability to various types of weak labels, including visual attributes, sensitive attributes, and hierarchical labels.

Key Results

CLRC consistently outperforms prior weakly-supervised methods across various datasets and tasks:

  • Visual Attributes: CLRC achieves the best overall performance on WIDER Attribute and CUB-200-2011, and competitive results on UT Zappos 50k. For CUB-200-2011, it shows a significant performance gap (4.88% and 5.55% in top-1 and top-5 accuracy, respectively) over the second-best method.
  • Fair Representation Learning: On the Fitzpatrick17k dataset, CLRC significantly improves worst-case AUC and reduces the AUC gap, demonstrating its ability to learn fairer representations for underrepresented subgroups compared to methods that improve overall AUC but widen the gap.
  • Hierarchical Labels: CLRC achieves the best performance on ImageNet-100 for hierarchical label learning, outperforming methods specifically designed for hierarchical relationships.

Limitations and Future Work

The authors acknowledge that CLRC is primarily designed for discrete labels. Applying it directly to continuous labels, such as text embeddings, is a limitation. Future work could explore adapting the grouping and weighting mechanisms for continuous settings. Additionally, the authors suggest investigating the integration of CLRC's weakly-supervised pretraining framework into compositional zero-shot learning, which could lead to improved convergence speed and downstream performance.

Summary updated

BibTeX
@inproceedings{yap2024group,
  author = {Boon Peng Yap and Beng Koon Ng},
  title = {{Group-Wise Contrastive Bottleneck for Weakly-Supervised Visual Representation Learning}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure