Discovering common information in multi-view data

Qi Zhang, Mingfei Lu, Shujian Yu, Jingmin Xin, Badong Chen

Introduction

In machine learning, "multi-view" data refers to information about the same entity collected from disparate sources, such as audio and visual tracks of a video. Effective learning from such data relies on two fundamental principles: consensus (identifying shared information across views) and complementarity (leveraging unique details from each view).

Existing methods, such as Canonical Correlation Analysis (CCA), often conflate correlation with "common information." Furthermore, many deep learning approaches rely on variational inference, which can be mathematically imprecise and difficult to scale. The CUMI (Common and Unique Multi-view Information) framework addresses these gaps by introducing a mathematically rigorous approach to separate shared consensus information from view-specific unique information.

The CUMI Framework

The CUMI framework is built upon three key technical pillars designed to ensure the purity and independence of extracted features.

The architecture of the CUMI framework, illustrating the separation of common and unique information through specialized encoders and a total correlation constraint.
  1. Gács-Körner (GK) Common Information: Rather than simply maximizing mutual information, CUMI utilizes a generalized version of GK common information. It defines common information as a variable extractable from any view via a deterministic function while maintaining maximum entropy.
  2. Total Correlation (TC) Constraint: To prevent "leakage" between common features ($C$) and unique features ($U$), the model explicitly minimizes Total Correlation. This ensures statistical independence, which is theoretically proven to be essential for faithfully discovering true common information.
  3. Matrix-based Rényi’s $\alpha$-order Entropy: To avoid the pitfalls of estimating probability distributions in high-dimensional spaces, CUMI employs a matrix-based functional. This calculates entropy directly from the eigenspectrum of data kernel matrices, bypassing unreliable variational approximations and enhancing mathematical robustness.

The architecture employs specific encoders for common and unique features, a reconstruction network to preserve information, and a final classifier for downstream tasks.

Experimental Evaluation

The framework was evaluated against state-of-the-art methods (including DCCA, MEIB, and TMC) using a variety of synthetic and real-world benchmarks, such as Caltech101, MSRC, and XRMB (acoustic/articulation mapping).

Key Results

CUMI consistently outperformed competing methods across all seven real-world datasets. Notable performance gains include:

  • Accuracy: On the N-W-OBJECT dataset, CUMI achieved 48.51% accuracy, surpassing MEIB's 45.21%. On the MSRC dataset, CUMI reached 95.71% accuracy, significantly outperforming traditional CCA (62.86%).
  • Signal Separation: Synthetic tests demonstrated CUMI's superior ability to physically separate overlapping signals. As shown in the figure below, CUMI was the only method to perfectly recover distinct frequencies of common and unique signals, whereas CCA and DCCA produced mixed, "blurry" results.
Comparison of signal separation on synthetic data: CUMI (f) accurately recovers the ground truth (a) frequencies, unlike traditional methods like CCA (b) or DCCA (c).

Discussion and Future Work

A primary insight of this work is that correlation does not equal consensus. By leveraging the Gács-Körner definition and ensuring independence through TC minimization, CUMI extracts a "pure" shared signal. Furthermore, the use of matrix-based entropy allows for complex information-theoretic operations without the noise introduced by variational hurdles.

The main limitation of the current framework is its time complexity ($O(n^3)$) due to the required eigenvalue decompositions. Future research will focus on integrating randomized approximations to reduce complexity to $O(n^2)$ or lower and extending the framework to handle incomplete multi-view data where some views may be missing.

Summary updated

BibTeX
@article{zhang2024discovering,
  author = {Qi Zhang and Mingfei Lu and Shujian Yu and Jingmin Xin and Badong Chen},
  title = {{Discovering common information in multi-view data}},
  journal = {Information Fusion},
  year = {2024},
  doi = {10.1016/j.inffus.2024.102400},
}
Figure