TL;DR
The paper introduces IPMC, a three-tier progressive framework for multi-view self-supervised learning grounded in information theory. By aligning view distributions, dynamically filtering "fake" negative samples, and employing a unified loss function, IPMC achieves state-of-the-art representation quality across diverse image and video benchmarks.
What Problem Does This Paper Address?
Self-supervised multi-view learning (MVL) aims to learn useful representations by capturing shared information across different "views" of the same data (e.g., the RGB, luminance, and color-opponent channels of an image). Most current methods rely on contrastive learning (CL), which compares pairs of samples. However, this approach faces three major hurdles:
- View-Specific Noise: Standard CL often fails to filter out information that is unique to a single view and irrelevant to the underlying semantic task.
- "Fake" Negatives: In CL, any sample other than the positive pair is treated as a negative. Often, these "negatives" actually belong to the same class as the positive sample, which confuses the model and biases learning.
- Optimization Instability: Treating all sample similarities equally can lead to gradient interference and slow convergence.
Furthermore, most existing theoretical frameworks for MVL are limited to just two views, leaving a gap in our understanding of generalized multi-view scenarios.
Methods
The authors propose a novel information-theoretic framework that rethinks MVL as a process of maximizing mutual information between views while minimizing task-irrelevant noise. This theory is realized through the Information theory-guided heuristic Progressive Multi-view Coding (IPMC) method, which operates in three progressive tiers:
- Distribution-tier: Instead of just sample-to-sample comparisons, IPMC aligns the global feature distributions of different views. It uses the Wasserstein distance as a discrepancy metric because it provides more consistent and robust gradients than standard metrics like KL-divergence, effectively "washing away" view-specific noise.
- Set-tier: This tier introduces Self-adjusted pool contrast (Sap). A designed "view filter" identifies potential "fake negatives" in the negative pool—samples that are semantically similar to the anchor—and moves them to the positive pool. This dynamically "cleans" the supervision signal during training.
- Instance-tier: Finally, a unified loss function ($L_{UniSap}$) is applied at the individual sample level. It uses adaptive leveraging factors to focus the model's attention on "hard" samples that are far from their optimal similarity, reducing gradient interference and stabilizing optimization.
Datasets
The authors benchmarked IPMC against a wide array of datasets:
- Image Classification: Tiny ImageNet, STL-10, CIFAR-10, CIFAR-100, and the large-scale ImageNet.
- Action Recognition (Video): UCF-101 and HMDB-51.
- Views used: For images, they typically used RGB, Luminance (L), and ab-color channels. For video, they used ventral (RGB frames) and dorsal (optical flow) streams.
Key Insights and Ideas
- Distribution Alignment is a Filter: Aligning the overall "shape" of feature distributions across views (Figure 3) is a powerful way to ensure the model focuses on shared semantic content rather than view-specific artifacts.
- The Negative Pool is "Dirty": The assumption that any random sample is a "negative" is a major bottleneck in self-supervised learning. Dynamically reclassifying these samples using a "view filter" (Figure 4) significantly boosts the discriminative power of the learned features.
- Wasserstein > KL-Divergence: For distribution alignment, the Wasserstein distance is superior because its gradients don't vanish even when distributions have non-overlapping supports, ensuring consistent updates throughout the training process.
- Paradigm Shift: The paper moves from a simple pairwise contrastive paradigm to a generalized framework that can naturally handle more than two views by considering the conjunction of information across all available signals.
Key Results
IPMC consistently outperformed state-of-the-art (SOTA) methods like CMC, SimCLR, and SwAV across all benchmarks:
- CIFAR-10: IPMC achieved a top-1 accuracy of 88.29% (using an fc backbone), surpassing the previous best (CMC) by nearly 5%.
- ImageNet: It reached 44.6% accuracy, outperforming other unsupervised methods like MoCo and BYOL.
- Action Recognition: On the UCF-101 dataset, IPMC achieved 59.5% accuracy, demonstrating excellent task transferability from unsupervised pre-training to downstream applications.
- Robustness: IPMC showed superior performance even with smaller batch sizes compared to methods like SimCLR, which typically require very large batches to be effective.
Limitations and Future Work
The authors acknowledge that the three-tier architecture and the calculation of Wasserstein distances increase the computational complexity and training time compared to simpler models like Deep InfoMax (DIM). While the performance gains are substantial, this added overhead is a trade-off. Future research could focus on optimizing these calculations or extending the framework to handle more diverse modalities, such as combining text, audio, and video in a single unified representation space.
Summary updated