Information Theory-Guided Heuristic Progressive Multi-View Coding

Jiangmeng Li, Hang Gao, Wenwen Qiang, Changwen Zheng

Abstract

Multi-view representation learning aims to capture comprehensive information from multiple views of a shared context. Recent works intuitively apply contrastive learning to different views in a pairwise manner, which is still scalable: view-specific noise is not filtered in learning view-shared representations; the fake negative pairs, where the negative terms are actually within the same class as the positive, and the real negative pairs are coequally treated; evenly measuring the similarities between terms might interfere with optimization. Importantly, few works study the theoretical framework of generalized self-supervised multi-view learning, especially for more than two views. To this end, we rethink the existing multi-view learning paradigm from the perspective of information theory and then propose a novel information theoretical framework for generalized multi-view learning. Guided by it, we build a multi-view coding method with a three-tier progressive architecture, namely Information theory-guided hierarchical Progressive Multi-view Coding (IPMC). In the distribution-tier, IPMC aligns the distribution between views to reduce view-specific noise. In the set-tier, IPMC constructs self-adjusted contrasting pools, which are adaptively modified by a view filter. Lastly, in the instance-tier, we adopt a designed unified loss to learn representations and reduce the gradient interference. Theoretically and empirically, we demonstrate the superiority of IPMC over state-of-the-art methods.

Introduction

Self-supervised multi-view learning (MVL) seeks to extract meaningful representations by identifying shared information across various data "views," such as RGB and depth channels of an image. While contrastive learning (CL) is a popular approach, it suffers from several limitations:

  1. View-Specific Noise: CL often fails to distinguish between task-relevant shared information and noise unique to a single view.
  2. "Fake" Negatives: Standard CL treats all non-positive samples as negatives, even if they share the same semantic class, leading to biased learning.
  3. Optimization Instability: Uniform treatment of sample similarities can cause gradient interference.

Furthermore, existing theoretical frameworks often struggle to generalize beyond two views.

Theoretical Framework

The proposed framework reinterprets MVL through information theory, aiming to maximize mutual information between views while filtering out task-irrelevant noise. Unlike conventional pairwise paradigms that focus on simple overlaps, the generalized paradigm considers the conjunction of information across all available signals to learn consistent and sufficient representations.

Comparison between the conventional pairwise paradigm (a) and the proposed generalized multi-view paradigm (b).

The IPMC Architecture

Guided by this theoretical foundation, the Information theory-guided hierarchical Progressive Multi-view Coding (IPMC) method employs a three-tier progressive architecture:

  1. Distribution-tier: Aligns global feature distributions using Wasserstein distance to robustly filter view-specific noise. This metric provides more consistent gradients than standard KL-divergence, ensuring the model focuses on shared semantic content.
  2. Set-tier: Implements Self-adjusted pool contrast (Sap), where a "view filter" identifies and reclassifies "fake negatives" into the positive pool, cleaning the supervision signal during training.
  3. Instance-tier: Utilizes a unified loss function ($L_{UniSap}$) with adaptive factors to focus on "hard" samples and stabilize optimization.
The proposed three-tier progressive architecture pipeline for IPMC.

Experimental Results

IPMC was evaluated across diverse datasets, including Tiny ImageNet, STL-10, CIFAR-10/100, and ImageNet for images, as well as UCF-101 and HMDB-51 for video action recognition.

Key findings include:

  • Superior Accuracy: IPMC achieved 88.29% top-1 accuracy on CIFAR-10 and 44.6% on ImageNet, outperforming SOTA methods like CMC, MoCo, and BYOL.
  • Transferability: On UCF-101, it reached 59.5% accuracy, demonstrating strong performance in downstream tasks.
  • Efficiency: The model remains robust even with smaller batch sizes, unlike many CL-based methods that require massive batches to be effective.

Conclusion

IPMC addresses the core challenges of multi-view self-supervised learning by integrating information theory with a progressive hierarchical architecture. While the use of Wasserstein distance and a three-tier structure increases computational complexity, the significant gains in representation quality and optimization stability establish it as a robust framework for generalized multi-view coding.

Summary updated

Abstract

Multi-view representation learning aims to capture comprehensive information from multiple views of a shared context. Recent works intuitively apply contrastive learning to different views in a pairwise manner, which is still scalable: view-specific noise is not filtered in learning view-shared representations; the fake negative pairs, where the negative terms are actually within the same class as the positive, and the real negative pairs are coequally treated; evenly measuring the similarities between terms might interfere with optimization. Importantly, few works study the theoretical framework of generalized self-supervised multi-view learning, especially for more than two views. To this end, we rethink the existing multi-view learning paradigm from the perspective of information theory and then propose a novel information theoretical framework for generalized multi-view learning. Guided by it, we build a multi-view coding method with a three-tier progressive architecture, namely Information theory-guided hierarchical Progressive Multi-view Coding (IPMC). In the distribution-tier, IPMC aligns the distribution between views to reduce view-specific noise. In the set-tier, IPMC constructs self-adjusted contrasting pools, which are adaptively modified by a view filter. Lastly, in the instance-tier, we adopt a designed unified loss to learn representations and reduce the gradient interference. Theoretically and empirically, we demonstrate the superiority of IPMC over state-of-the-art methods.
BibTeX
@inproceedings{li2023information,
  author = {Jiangmeng Li and Hang Gao and Wenwen Qiang and Changwen Zheng},
  title = {{Information Theory-Guided Heuristic Progressive Multi-View Coding}},
  booktitle = {Neural Networks},
  year = {2023},
  url = {https://openreview.net/forum?id=P6xjmcKxmS},
}
Figure