Probabilistic semi-nonnegative matrix factorization: a Skellam-based framework

Benoit Fuentes, Gaël RichardarXiv 2021 · arXiv

Abstract

Semi-nonnegative matrix factorization (SNMF) is a critical tool for decomposing real-valued data into non-negative activations and unconstrained atoms. Despite its utility in fields like energy efficiency and gene clustering, SNMF has historically lacked a rigorous probabilistic foundation compared to its non-negative counterpart (NMF). This paper introduces Skellam-SNMF, a novel framework that models signed data using the Skellam distribution—the difference between two independent Poisson variables. We derive efficient Expectation-Maximization (EM) and Variational Bayes (VB) algorithms and introduce a new divergence measure that generalizes Kullback-Leibler (KL) divergence to real-valued data.

Introduction

Matrix Factorization (MF) decomposes a data matrix $X$ into factors $W$ (atoms) and $\lambda$ (activations). While Nonnegative Matrix Factorization (NMF) is well-suited for purely positive data, many real-world signals are signed while maintaining inherently non-negative activations. Existing SNMF methods often rely on the Euclidean norm, which lacks the sparsity-promoting properties of KL divergence and makes it difficult to incorporate prior knowledge or handle missing data. Skellam-SNMF addresses these gaps by providing a hierarchical generative model for real-valued data.

The Skellam-SNMF Framework

The core of the proposed model is the representation of the unconstrained factor $W$ as the difference of two non-negative tensors ($\theta_0 - \theta_1$). The observed data $X$ is then modeled as being drawn from a Skellam distribution parameterized by these factors.

Technical Innovations

  1. The "Diffnomial" Distribution: The authors introduce the diffnomial distribution, which describes the posterior of hidden Poisson sources given an observed Skellam mixture. This serves as the signed-data equivalent of the multinomial distribution used in Poisson-NMF.
  2. Generalized Divergence ($\mathcal{D}$): By taking the limit of the model as it moves from integer to real-valued data, a new divergence measure is derived. This function acts as a mathematical bridge, allowing the application of KL-like logic to data containing negative values.
  3. Inference via VBEM: The Variational Bayes EM (VBEM) algorithm allows for full Bayesian inference. This enables the model to perform automated hyperparameter tuning, effectively "learning how to learn" by updating prior shapes and rates during the inference process.
Convergence of hyperparameter estimation and mean squared error (MSE) during online learning.

Experimental Evaluation

The framework was evaluated using synthetic data to test parameter estimation accuracy and four real-world datasets from the UCI repository (Ionosphere, Wave, Image, and Shuttle) for automatic clustering tasks. The authors compared Skellam-SNMF against several baselines, including the classic SNMF algorithm by Ding et al. (2010) and K-means.

Acronyms and descriptions of the algorithms used for benchmarking in the study.

Key Results

  • Clustering Performance: Skellam-SNMF significantly outperformed the classic SNMF algorithm on the Ionosphere and Wave datasets. On the Wave dataset, Skellam-SNMF achieved approximately 64% accuracy, compared to 61% for Ding '10 and 50.2% for K-means.
  • Integer vs. Continuous Modeling: The model uses a parameter $M$ to toggle between integer data ($M=1$) and real-valued data ($M \to \infty$). While the integer model ($Sk_1$) is faster, the continuous model ($Sk_\infty$) provides exact results for real-valued data, albeit with slower convergence.
  • Robustness: The probabilistic framework demonstrated lower variance in results compared to standard optimization-based SNMF, particularly when utilizing Variational Bayes for hyperparameter estimation.

Discussion and Future Work

While Skellam-SNMF provides a robust probabilistic foundation for SNMF, challenges remain regarding convergence speed for the continuous ($Sk_\infty$) model. Future work will focus on further optimizing the "parabolic EM" acceleration techniques and exploring the framework's ability to automatically estimate the model rank $K$ and handle missing data more extensively. Despite these challenges, the derivation of the divergence $\mathcal{D}$ represents a significant step forward in applying sparsity-promoting objectives to signed real-valued datasets.

Summary updated

BibTeX
@misc{fuentes2021probabilistic,
  author = {Benoit Fuentes and Gaël Richard},
  title = {{Probabilistic semi-nonnegative matrix factorization: a Skellam-based framework}},
  year = {2021},
  eprint = {2107.03317},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2107.03317},
}
Figure