Introduction
Self-supervised learning (SSL) has emerged as a powerful paradigm for learning representations from unlabeled data. A common strategy involves multi-view learning, where different "views" of the same instance (e.g., different augmentations or color channels) are used to train a model. The prevailing objective in these methods is the maximization of Mutual Information (MI) between views, based on the assumption that shared information across views is the most relevant for downstream tasks.
However, this paper identifies a fundamental limitation in the "consistency-only" approach. While maximizing shared information helps filter out view-specific noise, it often leads to the inadvertent loss of view-specific information that is task-relevant. For example, in fine-grained classification, unique features present in only one view (like specific color patterns) might be crucial for distinguishing between classes, yet standard MI maximization would encourage the model to ignore them.
Theoretical Motivation: Consistency vs. Complementarity
The core challenge in multi-view learning is to distinguish between three types of information:
- View-shared information: Features present across all views (e.g., object outlines).
- View-specific task-relevant information: Features unique to a specific view that aid in classification (e.g., specific color distributions).
- View-specific task-irrelevant noise: Features unique to a view that do not aid in classification (e.g., background artifacts).
As illustrated in Figure 1, traditional methods focus on the intersection of views (consistency), potentially missing the task-relevant information residing in the non-overlapping regions (complementarity). The proposed CoCoNet framework aims to capture both the shared and the complementary task-relevant information while suppressing noise.
The CoCoNet Framework
The Consistency and Complementarity Network (CoCoNet) introduces a dual-module architecture designed to balance these two objectives.
Global Consistency Module (GloCo)
The GloCo module ensures that the global representations of different views are aligned. Unlike standard contrastive methods that use KL-divergence or simple cosine similarity, GloCo employs the Generalized Sliced Wasserstein Distance (GSWD). GSWD is particularly effective for high-dimensional distribution alignment because it provides stable gradients even when distributions have non-overlapping supports—a common occurrence in early training or with highly disparate views. This module enforces a global consistency that captures the underlying structure shared by all views.
Local Complementarity Module (LoCo)
The LoCo module is designed to preserve view-specific, task-relevant details. It operates by constructing a "complementarity-factor"—a syncretic feature vector that aggregates both high-level and low-level information from all available views. By maximizing the mutual information between this aggregate factor and each individual view's features through a contrastive objective, the model is forced to retain unique details that are not present in other views but are essential for the overall representation.
Experimental Evaluation
CoCoNet was evaluated across a wide range of benchmarks to demonstrate its versatility and effectiveness.
Image Classification
On the ImageNet-1K dataset, CoCoNet achieved state-of-the-art results, outperforming the Contrastive Multiview Coding (CMC) baseline by an average of 1.1% across different layers. Similar improvements were observed on smaller datasets like CIFAR-10, CIFAR-100, and STL-10. On STL-10, CoCoNet reached an accuracy of 84.58%, surpassing other competitive SSL methods like SimCLR and SwAV.
Graph and Video Benchmarks
The framework's robustness was further tested on non-image data. In graph property prediction tasks from the Open Graph Benchmark (OGB), CoCoNet showed consistent gains. Similarly, in action recognition tasks on UCF-101 and HMDB-51, the dual strategy of consistency and complementarity proved superior to consistency-only baselines.
Real-World Application: WHEBD-759
To validate the model in a practical setting, the authors introduced the WHEBD-759 dataset, featuring waterfowl in complex habitats. CoCoNet was used to pre-train encoders for bird detection and classification. When integrated with a Faster-RCNN detector, the CoCoNet-pre-trained features significantly improved detection performance in semi-supervised scenarios, highlighting its utility for real-world, unlabeled data.
Conclusion
CoCoNet addresses a critical oversight in multi-view self-supervised learning by explicitly modeling both global consistency and local complementarity. By leveraging GSWD for distribution alignment and a syncretic complementarity factor for feature preservation, it achieves superior representation learning across diverse domains. Future work will focus on optimizing the computational complexity of the framework and exploring automated view generation strategies.
Summary updated