CMF: Collective Matrix Factorization

Arto Klami, Lauri VäreOpenalex Unknown 2014 · DOI · OpenAlex

TL;DR

This paper introduces a robust Bayesian framework for Canonical Correlation Analysis (CCA) that excels in high-dimensional settings where samples are scarce. By re-casting CCA as a generative "Inter-Battery Factor Analysis" model and applying group-wise sparsity priors, the authors create a method that automatically separates shared signals from view-specific noise, outperforming classical and regularized approaches in both speed and accuracy.

What Problem Does This Paper Address?

Canonical Correlation Analysis (CCA) is the "multi-view" cousin of PCA; it looks for linear relationships between two different sets of variables (like gene expression vs. DNA copy number) measured on the same samples.

However, classical CCA has a "small $N$, large $D$" problem: when you have thousands of features but only a few dozen samples (common in genomics), the model overfits and finds correlations that are just noise. Previous Bayesian attempts to fix this were computationally heavy—requiring the inversion of massive covariance matrices ($O(D^3)$ complexity)—making them impractical for modern, high-dimensional datasets. The authors set out to create a Bayesian CCA that is both statistically robust and computationally efficient.

Methods

The authors bridge the gap between CCA and Factor Analysis (FA) using a model called Inter-Battery Factor Analysis (IBFA).

The Core Innovation: Group-wise Sparsity

Instead of trying to estimate a full, complex covariance matrix, they treat the data as being generated by three types of latent variables:

  1. Shared components: Variation present in both datasets.
  2. View-specific components: Variation unique to dataset A or B.
  3. Residual noise: Isotropic (simple) noise.

To make this work without manually guessing how many components belong to each group, they use Group-wise Automatic Relevance Determination (ARD). They pack all potential components into one large projection matrix $W$ and apply a prior that pushes entire columns toward zero if they aren't needed. This allows the model to "learn" the structure: a component might be active for both views (shared), active for only one (specific), or shut down entirely (unused).

Inference

The paper provides two ways to solve the model:

  • Variational Bayes (VB): A fast, deterministic approximation. They include a "parameter-expansion" trick (optimizing for rotation) that significantly speeds up convergence.
  • Gibbs Sampler: A more computationally intensive sampling method using a "spike-and-slab" prior for even sharper sparsity.

Datasets

The authors validated their approach on several fronts:

  • Artificial Data: Used to prove the model can correctly recover known shared and specific latent signals.
  • Cancer Gene Prioritization: The Pollack and Hyman datasets, which involve thousands of genes but fewer than 100 samples. The goal was to find genes where DNA alterations correlate with expression changes.
  • Multi-label Prediction: 10 benchmark datasets from the Mulan library (e.g., emotions, scene, yeast, enron, delicious). These were used to show that CCA is a powerful tool for predicting multiple related labels simultaneously.

Key Insights and Ideas

  • Low-Rank is Key: By modeling view-specific variation as low-rank components rather than a full-rank covariance matrix, the computational complexity drops from $O(D^3)$ to being linear in the number of dimensions $D$. This is what makes the model "high-dimensional ready."
  • The "Squeeze" Effect: For prediction tasks, the model "squeezes" information through the shared latent variables. It ignores the view-specific variation, which acts as a sophisticated filter for removing structured noise that would otherwise confuse a regressor.
  • Automatic Complexity: You don't need to know the "true" number of correlations in advance. You can set a high upper bound for the number of components, and the ARD prior will prune the unnecessary ones.

Key Results

  • Superior Accuracy: In the cancer gene mining task, the Bayesian IBFA (BIBFA) achieved higher Area Under Curve (AUC) scores than specialized bioinformatics tools like pint.
  • Better Generalization: On the Mulan datasets, BIBFA outperformed standard multi-label algorithms (like RAKEL and MLKNN) on datasets with a high number of labels, proving that capturing dependencies between labels is vital.
  • Massive Speedups: On high-dimensional datasets (like bibtex or delicious), the previous "full-covariance" Bayesian CCA was either infeasible or took hours, while the new BIBFA converged in minutes.
  • Robustness: Unlike classical CCA, which breaks down completely when the number of features exceeds the number of samples ($D > N$), BIBFA remained stable and accurate.

Limitations and Future Work

  • Linearity: The model is fundamentally linear. While the authors mention kernel-based and neural network extensions, this specific paper focuses on the linear case.
  • Component Allocation: While ARD is powerful, in extremely small sample sizes, the model can be conservative, sometimes missing weak shared components and categorizing them as noise.
  • Scaling to Many Views: The paper primarily focuses on two datasets (views). The authors have since extended this work into "Group Factor Analysis" (GFA) to handle $M$ datasets, which is a natural next step for this research.

Summary updated

Abstract

Collective matrix factorization (CMF) finds joint low-rank representations for a collection of matrices with shared row or column entities. This code learns a variational Bayesian approximation for CMF, supporting multiple likelihood potentials and missing data, while identifying both factors shared by multiple matrices and factors private for each matrix. For further details on the method see Klami et al. (2014) &lt;<a href="https://doi.org/10.48550/arXiv.1312.5921" target="_top">doi:10.48550/arXiv.1312.5921</a>&gt;. The package can also be used to learn Bayesian canonical correlation analysis (CCA) and group factor analysis (GFA) models, both of which are special cases of CMF. This is likely to be useful for people looking for CCA and GFA solutions supporting missing data and non-Gaussian likelihoods. See Klami et al. (2013) &lt;<a href="https://research.cs.aalto.fi/pml/online-papers/klami13a.pdf" target="_top">https://research.cs.aalto.fi/pml/online-papers/klami13a.pdf</a>&gt; and Virtanen et al. (2012) &lt;<a href="http://proceedings.mlr.press/v22/virtanen12.html" target="_top">http://proceedings.mlr.press/v22/virtanen12.html</a>&gt; for details on Bayesian CCA and GFA, respectively.
BibTeX
@inproceedings{klami2014cmf,
  author = {Arto Klami and Lauri Väre},
  title = {{CMF: Collective Matrix Factorization}},
  booktitle = {Openalex Unknown},
  year = {2014},
  doi = {10.32614/cran.package.cmf},
}

Loading related content...

Figure