Information theory-guided heuristic progressive multi-view coding

Jiangmeng Li, Hang Gao, Wenwen Qiang, Changwen Zheng

TL;DR

The paper introduces IPMC, a three-tier progressive framework for multi-view self-supervised learning grounded in information theory. By aligning view distributions, dynamically filtering "fake" negative samples, and employing a unified loss function, IPMC achieves state-of-the-art representation quality across diverse image and video benchmarks.

What Problem Does This Paper Address?

Self-supervised multi-view learning (MVL) aims to learn useful representations by capturing shared information across different "views" of the same data (e.g., the RGB, luminance, and color-opponent channels of an image). Most current methods rely on contrastive learning (CL), which compares pairs of samples. However, this approach faces three major hurdles:

  1. View-Specific Noise: Standard CL often fails to filter out information that is unique to a single view and irrelevant to the underlying semantic task.
  2. "Fake" Negatives: In CL, any sample other than the positive pair is treated as a negative. Often, these "negatives" actually belong to the same class as the positive sample, which confuses the model and biases learning.
  3. Optimization Instability: Treating all sample similarities equally can lead to gradient interference and slow convergence.

Furthermore, most existing theoretical frameworks for MVL are limited to just two views, leaving a gap in our understanding of generalized multi-view scenarios.

Methods

The authors propose a novel information-theoretic framework that rethinks MVL as a process of maximizing mutual information between views while minimizing task-irrelevant noise. This theory is realized through the Information theory-guided heuristic Progressive Multi-view Coding (IPMC) method, which operates in three progressive tiers:

  • Distribution-tier: Instead of just sample-to-sample comparisons, IPMC aligns the global feature distributions of different views. It uses the Wasserstein distance as a discrepancy metric because it provides more consistent and robust gradients than standard metrics like KL-divergence, effectively "washing away" view-specific noise.
  • Set-tier: This tier introduces Self-adjusted pool contrast (Sap). A designed "view filter" identifies potential "fake negatives" in the negative pool—samples that are semantically similar to the anchor—and moves them to the positive pool. This dynamically "cleans" the supervision signal during training.
  • Instance-tier: Finally, a unified loss function ($L_{UniSap}$) is applied at the individual sample level. It uses adaptive leveraging factors to focus the model's attention on "hard" samples that are far from their optimal similarity, reducing gradient interference and stabilizing optimization.
The proposed three-tier progressive architecture pipeline for IPMC.

Datasets

The authors benchmarked IPMC against a wide array of datasets:

  • Image Classification: Tiny ImageNet, STL-10, CIFAR-10, CIFAR-100, and the large-scale ImageNet.
  • Action Recognition (Video): UCF-101 and HMDB-51.
  • Views used: For images, they typically used RGB, Luminance (L), and ab-color channels. For video, they used ventral (RGB frames) and dorsal (optical flow) streams.

Key Insights and Ideas

  • Distribution Alignment is a Filter: Aligning the overall "shape" of feature distributions across views (Figure 3) is a powerful way to ensure the model focuses on shared semantic content rather than view-specific artifacts.
  • The Negative Pool is "Dirty": The assumption that any random sample is a "negative" is a major bottleneck in self-supervised learning. Dynamically reclassifying these samples using a "view filter" (Figure 4) significantly boosts the discriminative power of the learned features.
  • Wasserstein > KL-Divergence: For distribution alignment, the Wasserstein distance is superior because its gradients don't vanish even when distributions have non-overlapping supports, ensuring consistent updates throughout the training process.
  • Paradigm Shift: The paper moves from a simple pairwise contrastive paradigm to a generalized framework that can naturally handle more than two views by considering the conjunction of information across all available signals.
Comparison between the conventional pairwise paradigm (a) and the proposed generalized multi-view paradigm (b).

Key Results

IPMC consistently outperformed state-of-the-art (SOTA) methods like CMC, SimCLR, and SwAV across all benchmarks:

  • CIFAR-10: IPMC achieved a top-1 accuracy of 88.29% (using an fc backbone), surpassing the previous best (CMC) by nearly 5%.
  • ImageNet: It reached 44.6% accuracy, outperforming other unsupervised methods like MoCo and BYOL.
  • Action Recognition: On the UCF-101 dataset, IPMC achieved 59.5% accuracy, demonstrating excellent task transferability from unsupervised pre-training to downstream applications.
  • Robustness: IPMC showed superior performance even with smaller batch sizes compared to methods like SimCLR, which typically require very large batches to be effective.

Limitations and Future Work

The authors acknowledge that the three-tier architecture and the calculation of Wasserstein distances increase the computational complexity and training time compared to simpler models like Deep InfoMax (DIM). While the performance gains are substantial, this added overhead is a trade-off. Future research could focus on optimizing these calculations or extending the framework to handle more diverse modalities, such as combining text, audio, and video in a single unified representation space.

Summary updated

Abstract

Multi-view representation learning aims to capture comprehensive information from multiple views of a shared context. Recent works intuitively apply contrastive learning to different views in a pairwise manner, which is still scalable: view-specific noise is not filtered in learning view-shared representations; the fake negative pairs, where the negative terms are actually within the same class as the positive, and the real negative pairs are coequally treated; evenly measuring the similarities between terms might interfere with optimization. Importantly, few works study the theoretical framework of generalized self-supervised multi-view learning, especially for more than two views. To this end, we rethink the existing multi-view learning paradigm from the perspective of information theory and then propose a novel information theoretical framework for generalized multi-view learning. Guided by it, we build a multi-view coding method with a three-tier progressive architecture, namely Information theory-guided heuristic Progressive Multi-view Coding (IPMC). In the distribution-tier, IPMC aligns the distribution between views to reduce view-specific noise. In the set-tier, IPMC constructs self-adjusted contrasting pools, which are adaptively modified by a view filter. Lastly, in the instance-tier, we adopt a designed unified loss to learn representations and reduce the gradient interference. Theoretically and empirically, we demonstrate the superiority of IPMC over state-of-the-art methods.
BibTeX
@inproceedings{li2023information,
  author = {Jiangmeng Li and Hang Gao and Wenwen Qiang and Changwen Zheng},
  title = {{Information theory-guided heuristic progressive multi-view coding}},
  booktitle = {Neural Networks},
  year = {2023},
  url = {https://openreview.net/forum?id=P6xjmcKxmS},
}

Loading related content...

Figure