Abstract
Integrating various data modalities brings valuable insights into underlying phenomena. Multimodal factor analysis (FA) uncovers shared axes of variation underlying different simple data modalities, where each sample is represented by a vector of features. However, FA is not suited for structured data modalities, such as text or single cell sequencing data, where multiple data points are measured per each sample and exhibit a clustering structure. To overcome this challenge, we introduce FACTM, a novel, multi-view and multi-structure Bayesian model that combines FA with correlated topic modeling and is optimized using variational inference. Additionally, we introduce a method for rotating latent factors to enhance interpretability with respect to binary features. On text and video benchmarks as well as real-world music and COVID-19 datasets, we demonstrate that FACTM outperforms other methods in identifying clusters in structured data, and integrating them with simple modalities via the inference of shared, interpretable factors.
Introduction
In multi-modal data integration, Factor Analysis (FA) serves as a primary tool for identifying shared axes of variation across high-dimensional data sources. While effective for "simple" data where each sample is a single feature vector, standard FA fails to account for "structured" data. Structured modalities, such as text documents or biological samples, consist of collections of data points with internal clustering (e.g., topics in text or cell types in tissue).
Traditional approaches often rely on a disconnected two-step process: first vectorizing structured data and then applying FA. This paper argues that such a disjointed methodology loses critical information and introduces FACTM to perform joint inference across simple and structured views simultaneously.
Methods
FACTM is a multi-view Bayesian framework that bridges Multimodal Factor Analysis and Correlated Topic Modeling (CTM). The model identifies shared latent factors ($Z$) across all views while simultaneously capturing the internal cluster structures within structured views.
The core innovation is a sample-specific modification variable ($\mu_n$) that links global shared factors to the internal cluster proportions of structured data. This allows the model to learn how global states (e.g., a patient's disease state) shift internal distributions (e.g., cell type proportions). To address the rotation invariance common in FA models, the authors propose a "Supervised Rotation" technique using the Kabsch-Umeyama algorithm. This aligns latent factors with known binary features to enhance human interpretability. The entire model is optimized using Variational Inference (VI) for scalability.
Results and Key Insights
Evaluation across simulated data and real-world benchmarks (CMU-MOSI, Mirex music data, and COVID-19 patient data) revealed several key advantages:
- Superior Joint Inference: Jointly learning factors and topics significantly outperforms two-step approaches. In simulations, FACTM achieved higher Spearman correlations with ground-truth factors compared to standard FA, PCA, or MOFA.
- Predictive Performance: On the Mirex music dataset, FACTM achieved a ROC-AUC of 0.64 for emotion classification, outperforming standalone CTM (0.57) and LDA (0.62).
- Biological Grounding: In COVID-19 datasets, FACTM’s inferred topics naturally aligned with expert-annotated cell types without prior supervision.
- Enhanced Interpretability: The supervised rotation method successfully mapped abstract latent factors to clear, interpretable categories.
Conclusion
FACTM provides a robust framework for integrating simple and structured data modalities. By utilizing joint inference and supervised rotation, the model not only improves predictive accuracy but also offers a more interpretable latent space. Future work will focus on addressing current limitations, such as the assumption of linear dependencies and sensitivity to hyperparameters, potentially through the integration of Variational Autoencoders for non-linear modeling.
Summary updated