Information Theory-Guided Heuristic Progressive Multi-View Coding

Jiangmeng Li, Wenwen Qiang, Hang Gao, Bing Su, Farid Razzak, Jie Hu, Changwen Zheng, Hui Xiong

TL;DR

This paper introduces IPMC, a novel information-theory-guided framework for multi-view representation learning. By addressing limitations in existing contrastive learning methods, IPMC learns more consistent and sufficient representations from multiple data views, leading to state-of-the-art performance on classification tasks.

What Problem Does This Paper Address?

Multi-view representation learning aims to extract shared information from different "views" of the same data (e.g., images from different camera angles, or different modalities like text and audio). While contrastive learning (CL) has been successful, current pairwise CL approaches struggle with:

  • View-specific noise: Learning representations that are too dependent on a single view, obscuring the shared information.
  • Misclassified negatives: Treating "fake negative" pairs (samples from the same class as the positive but different views) the same as "real negative" pairs (samples from different classes), leading to suboptimal learning.
  • Optimization instability: Uniformly measuring similarities between all pairs can hinder effective optimization and lead to overly complex representations.
  • Lack of theoretical grounding: Few works provide a strong theoretical framework, especially for learning from more than two views.

Methods

IPMC tackles these issues with a three-tier progressive architecture, guided by an information-theoretic framework:

  1. Distribution-tier (View Distribution Alignment): This tier aims to reduce view-specific noise by aligning the distributions of different views. It uses a discrepancy metric, specifically Wasserstein distance, to minimize the differences between view representations, encouraging them to capture shared semantic content.
  2. Set-tier (Self-Adjusted Pool Contrast): Instead of fixed pairwise contrast, IPMC uses dynamic pools. A "view filter" adaptively identifies and transfers "fake negative" terms (which are actually similar to positives) into the positive pool. This creates more challenging and informative contrastive pairs. A memory bank is used to store past positive features, enriching the positive pool.
  3. Instance-tier (Unified Loss Calculation): A designed unified loss function is employed to learn discriminative representations and reduce gradient interference. This loss function aims to push positive pairs closer together and negative pairs further apart, with specific terms to modulate the impact of similarities.

Datasets

The paper evaluates IPMC on several standard benchmark datasets for classification:

  • Tiny ImageNet
  • STL-10
  • CIFAR-10
  • CIFAR-100

Key Insights and Ideas

  • Information Theory as a Guiding Principle: The authors propose a generalized information-theoretic framework for multi-view learning, which provides a solid foundation for their proposed architecture.
  • Beyond Pairwise Contrast: The shift from fixed pairwise contrast to dynamic, self-adjusted pools in the set-tier is crucial for handling "fake negatives" and improving learning efficiency.
  • Wasserstein Distance for Alignment: Using Wasserstein distance for distribution alignment proves more effective than KL-divergence, offering stable gradients and better performance.
  • Three-Tier Progressive Architecture: The hierarchical approach (distribution, set, instance) allows for progressive refinement of representations, tackling different aspects of multi-view learning at each stage.

Key Results

  • State-of-the-Art Performance: IPMC consistently outperforms existing state-of-the-art unsupervised multi-view learning methods across various datasets, achieving significant accuracy improvements. For instance, on CIFAR10 and STL-10, IPMC(Sap + DA) achieved 89.74% and 91.14% accuracy respectively, surpassing other methods.
  • Ablation Studies: Experiments confirm the effectiveness of each component: the distribution alignment (DA) and the self-adjusted pool (Sap) both contribute positively to IPMC's performance.
  • Robustness to Encoders: IPMC demonstrates strong adaptability, performing well with different backbone encoders like AlexNet and ResNet-50.
  • Visualizations: Representation visualizations show that IPMC learns more discriminative and structured representations compared to CMC and DIM, with neighboring representations corresponding to visually similar images.

Limitations and Future Work

  • Hyperparameter Sensitivity: While the paper explores hyperparameter influences (β, γ, δ), finding optimal values might still require careful tuning. The paper notes that the effect of parameter α (leveraging factor) can be sensitive and sometimes requires attenuation.
  • Computational Cost: While IPMC is efficient, the distribution alignment step can introduce some computational overhead, though it is argued to be manageable and beneficial for convergence.
  • Future Work: The paper suggests that exploring more sophisticated data augmentation strategies and investigating the theoretical implications of the proposed framework for more complex multi-view scenarios could be fruitful directions. The authors also mention that their framework can be generalized to more than three views.

Summary updated

Abstract

Multi-view representation learning captures comprehensive information from multiple views of a shared context. Recent works intuitively apply contrastive learning (CL) to learn representations, regarded as a pairwise manner, which is still scalable: view-specific noise is not filtered in learning view-shared representations; the fake negative pairs, where the negative terms are actually within the same class as the positive, and the real negative pairs are coequally treated; and evenly measuring the similarities between terms might interfere with optimization. Importantly, few works research the theoretical framework of generalized self-supervised multi-view learning, especially for more than two views. To this end, we rethink the existing multi-view learning paradigm from the information theoretical perspective and then propose a novel information theoretical framework for generalized multi-view learning. Guided by it, we build a multi-view coding method with a three-tier progressive architecture, namely Information theory-guided heuristic Progressive Multi-view Coding (IPMC). In the distribution-tier, IPMC aligns the distribution between views to reduce view-specific noise. In the set-tier, IPMC builds self-adjusted pools for contrasting, which utilizes a view filter to adaptively modify the pools. Lastly, in the instance-tier, we adopt a designed unified loss to learn discriminative representations and reduce the gradient interference. Theoretically and empirically, we demonstrate the superiority of IPMC over state-of-the-art methods.
BibTeX
@inproceedings{li2023information,
  author = {Jiangmeng Li and Wenwen Qiang and Hang Gao and Bing Su and Farid Razzak and Jie Hu and Changwen Zheng and Hui Xiong},
  title = {{Information Theory-Guided Heuristic Progressive Multi-View Coding}},
  booktitle = {Neural Networks},
  year = {2023},
  url = {https://openreview.net/forum?id=P6xjmcKxmS},
}
Figure