CPM-Nets: Cross Partial Multi-View Networks

Changqing Zhang, Zongbo Han, Yajie Cui, Huazhu Fu, Joey Tianyi Zhou, Qinghua Hu

Introduction

CPM-Nets (Cross Partial Multi-View Networks) is a novel framework designed to handle multi-view data where some views are missing—a pervasive challenge in real-world machine learning applications. By learning a "complete" and "versatile" latent representation capable of reconstructing available views, and by utilizing a clustering-like classification loss, CPM-Nets achieves state-of-the-art performance and remarkable robustness even when a large percentage of data is missing.

The Challenge of Partial Data

In an ideal scenario, multi-view learning (MVL) involves datasets where every sample possesses every "view" (e.g., an image, a text description, and an audio clip). In reality, data is often "partial." A medical patient might have an MRI but no CT scan; a webpage might have text but no video.

Traditional MVL algorithms usually assume completeness. When views are missing, researchers typically resort to two flawed strategies:

  1. Imputation: Attempting to fill in the blanks. This is difficult because missing views are often "blockwise" (entire modalities are missing), making low-rank completion or simple mean-filling ineffective.
  2. Grouping: Sorting samples by their specific missing-view patterns. This approach scales poorly as the number of views increases, leading to fragmented models that cannot share information effectively.

The authors address the challenge of Partial Multi-View Classification (PMVC): how to learn a unified, discriminative representation from samples with arbitrary missing-view patterns without relying on a "primary" complete view.

The CPM-Nets Approach

The core innovation of CPM-Nets is shifting from mapping views to a latent space to a "degradation" perspective.

Architecture of CPM-Nets
  • Completeness through Reconstruction: Instead of trying to find a function that maps available views to a latent code $h$, the model learns a latent code $h$ that can reconstruct any available view. If $h$ can "degrade" back into the original observations, it must contain all the essential information, ensuring completeness.
  • Versatility: The authors provide a formal definition and theoretical proof that their latent representation is "versatile." This means it contains enough information to perform as well as (or better than) any single-view representation.
  • Structured Latent Representation: To make the latent space useful for classification, the model avoids standard cross-entropy. Instead, it employs a clustering-like loss. This forces the latent codes of samples from the same class to cluster together while pushing different classes apart with a defined margin, ensuring the representation is "structured" and interpretable.
  • Test-Time Consistency: Because training involves optimizing latent codes directly, the authors introduce a fine-tuning strategy for the testing phase. This ensures that the representation extracted from a new, unseen test sample remains consistent with the space learned during training.

Key Insights

  • The "Degradation" Flip: The most clever move is treating the latent-to-view mapping as a communication channel. By mimicking data transmission (where a comprehensive signal is sent and partial data is received), the model naturally handles missing views by simply ignoring the reconstruction loss for views that are absent.
  • Nonparametric Classification: By using a clustering-like loss rather than a parametric classifier (like a standard softmax layer), the model reduces the burden of parameter tuning and performs better in "small-sample-size" regimes.
  • Geometry Matters: The paper emphasizes that for multi-view learning to work, the latent space shouldn't just be a bucket of features; it needs a specific geometric structure where class separation is maximized.

Key Results

The model was evaluated on six diverse datasets (including ORL, PIE, YaleB, CUB, Handwritten digits, and Animal datasets), simulating missing data by randomly removing views at various "missing rates" ($\eta$).

Accuracy vs. Missing Rate
  • Robustness to Missing Data: CPM-Nets significantly outperformed competitors (like CCA, DCCA, and DMF-MVC) as the missing rate increased. On datasets like ORL, while other methods' accuracy plummeted as $\eta$ went from 0.0 to 0.5, CPM-Nets remained remarkably stable, losing minimal accuracy even at high missing rates.
  • Superiority over Imputation: The authors compared CPM-Nets against state-of-the-art imputation methods (like Cascaded Residual Autoencoders). CPM-Nets consistently performed better, proving that learning a robust latent space is often superior to trying to "guess" what the missing data looked like.
  • Visual Proof: t-SNE visualizations of the latent space showed that CPM-Nets produced much tighter, more distinct clusters for different classes compared to other unsupervised or supervised representation learning methods.

Limitations and Future Work

  • Computational Overhead: The requirement to fine-tune or optimize the latent representation for each test instance could be computationally expensive for real-time applications dealing with massive data streams.
  • Independence Assumption: The framework assumes views are conditionally independent given the shared latent representation. In some complex datasets, there might be residual correlations between views that this model doesn't explicitly capture.
  • Scaling Views: While more flexible than grouping, the model still requires a separate reconstruction network for every view. Scaling this architecture to hundreds of distinct views might present significant engineering challenges.

Summary updated

Abstract

Despite multi-view learning progressed fast in past decades, it is still challenging due to the difficulty in modeling complex correlation among different views, especially under the context of view missing. To address the challenge, we propose a novel framework termed Cross Partial Multi-View Networks (CPM-Nets). In this framework, we first give a formal definition of completeness and versatility for multi-view representation and then theoretically prove the versatility of the latent representation learned from our algorithm. To achieve the completeness, the task of learning latent multi-view representation is specifically translated to degradation process through mimicking data transmitting, such that the optimal tradeoff between consistence and complementarity across different views could be achieved. In contrast with methods that either complete missing views or group samples according to view-missing patterns, our model fully exploits all samples and all views to produce structured representation for interpretability. Extensive experimental results validate the effectiveness of our algorithm over existing state-of-the-arts.
BibTeX
@inproceedings{zhang2019cpm,
  author = {Changqing Zhang and Zongbo Han and Yajie Cui and Huazhu Fu and Joey Tianyi Zhou and Qinghua Hu},
  title = {{CPM-Nets: Cross Partial Multi-View Networks}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
}
Figure