Modeling Multiple Views via Implicitly Preserving Global Consistency and Local Complementarity

Jiangmeng Li, Wenwen Qiang, Changwen Zheng, Bing Su, Farid Razzak, Ji-Rong Wen, Hui Xiong

Introduction

Self-supervised learning (SSL) has emerged as a powerful paradigm for learning representations from unlabeled data. A common strategy involves multi-view learning, where different "views" of the same instance (e.g., different augmentations or color channels) are used to train a model. The prevailing objective in these methods is the maximization of Mutual Information (MI) between views, based on the assumption that shared information across views is the most relevant for downstream tasks.

However, this paper identifies a fundamental limitation in the "consistency-only" approach. While maximizing shared information helps filter out view-specific noise, it often leads to the inadvertent loss of view-specific information that is task-relevant. For example, in fine-grained classification, unique features present in only one view (like specific color patterns) might be crucial for distinguishing between classes, yet standard MI maximization would encourage the model to ignore them.

Theoretical Motivation: Consistency vs. Complementarity

The core challenge in multi-view learning is to distinguish between three types of information:

  1. View-shared information: Features present across all views (e.g., object outlines).
  2. View-specific task-relevant information: Features unique to a specific view that aid in classification (e.g., specific color distributions).
  3. View-specific task-irrelevant noise: Features unique to a view that do not aid in classification (e.g., background artifacts).
Information decomposition in multi-view learning: distinguishing between view-shared information, view-specific task-relevant information, and view-specific noise.

As illustrated in Figure 1, traditional methods focus on the intersection of views (consistency), potentially missing the task-relevant information residing in the non-overlapping regions (complementarity). The proposed CoCoNet framework aims to capture both the shared and the complementary task-relevant information while suppressing noise.

The CoCoNet Framework

The Consistency and Complementarity Network (CoCoNet) introduces a dual-module architecture designed to balance these two objectives.

Global Consistency Module (GloCo)

The GloCo module ensures that the global representations of different views are aligned. Unlike standard contrastive methods that use KL-divergence or simple cosine similarity, GloCo employs the Generalized Sliced Wasserstein Distance (GSWD). GSWD is particularly effective for high-dimensional distribution alignment because it provides stable gradients even when distributions have non-overlapping supports—a common occurrence in early training or with highly disparate views. This module enforces a global consistency that captures the underlying structure shared by all views.

Local Complementarity Module (LoCo)

The LoCo module is designed to preserve view-specific, task-relevant details. It operates by constructing a "complementarity-factor"—a syncretic feature vector that aggregates both high-level and low-level information from all available views. By maximizing the mutual information between this aggregate factor and each individual view's features through a contrastive objective, the model is forced to retain unique details that are not present in other views but are essential for the overall representation.

The CoCoNet architecture, integrating the Global Consistency Module (GloCo) and the Local Complementarity Module (LoCo).

Experimental Evaluation

CoCoNet was evaluated across a wide range of benchmarks to demonstrate its versatility and effectiveness.

Image Classification

On the ImageNet-1K dataset, CoCoNet achieved state-of-the-art results, outperforming the Contrastive Multiview Coding (CMC) baseline by an average of 1.1% across different layers. Similar improvements were observed on smaller datasets like CIFAR-10, CIFAR-100, and STL-10. On STL-10, CoCoNet reached an accuracy of 84.58%, surpassing other competitive SSL methods like SimCLR and SwAV.

Graph and Video Benchmarks

The framework's robustness was further tested on non-image data. In graph property prediction tasks from the Open Graph Benchmark (OGB), CoCoNet showed consistent gains. Similarly, in action recognition tasks on UCF-101 and HMDB-51, the dual strategy of consistency and complementarity proved superior to consistency-only baselines.

Real-World Application: WHEBD-759

To validate the model in a practical setting, the authors introduced the WHEBD-759 dataset, featuring waterfowl in complex habitats. CoCoNet was used to pre-train encoders for bird detection and classification. When integrated with a Faster-RCNN detector, the CoCoNet-pre-trained features significantly improved detection performance in semi-supervised scenarios, highlighting its utility for real-world, unlabeled data.

Conclusion

CoCoNet addresses a critical oversight in multi-view self-supervised learning by explicitly modeling both global consistency and local complementarity. By leveraging GSWD for distribution alignment and a syncretic complementarity factor for feature preservation, it achieves superior representation learning across diverse domains. Future work will focus on optimizing the computational complexity of the framework and exploring automated view generation strategies.

Summary updated

Abstract

While self-supervised learning techniques are often used to mine hidden knowledge from unlabeled data via modeling multiple views, it is unclear how to perform effective representation learning in a complex and inconsistent context. To this end, we propose a new multi-view self-supervised learning method, namely <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">consistency and complementarity network</i> (CoCoNet), to comprehensively learn global inter-view consistent and local cross-view complementarity-preserving representations from multiple views. To capture crucial common knowledge which is implicitly shared among views, CoCoNet employs a global consistency module that aligns the probabilistic distribution of views by utilizing an efficient discrepancy metric based on the generalized sliced Wasserstein distance. To incorporate cross-view complementary information, CoCoNet proposes a heuristic complementarity-aware contrastive learning approach, which extracts a complementarity-factor jointing cross-view discriminative knowledge and uses it as the contrast to guide the learning of view-specific encoders. Theoretically, the superiority of CoCoNet is verified by our information-theoretical-based analyses. Empirically, our thorough experimental results show that CoCoNet outperforms the state-of-the-art self-supervised methods by a significant margin, for instance, CoCoNet beats the best benchmark method by an average margin of 1.1% on ImageNet.
BibTeX
@article{li2022modeling,
  author = {Jiangmeng Li and Wenwen Qiang and Changwen Zheng and Bing Su and Farid Razzak and Ji-Rong Wen and Hui Xiong},
  title = {{Modeling Multiple Views via Implicitly Preserving Global Consistency and Local Complementarity}},
  journal = {IEEE Trans. Knowl. Data Eng.},
  year = {2022},
  doi = {10.1109/tkde.2022.3198746},
}
Figure