Undirected Probabilistic Model for Tensor Decomposition

Zerui Tao, Toshihisa Tanaka, Qibin Zhao

Introduction

Researchers have developed EnergyTD, a novel tensor decomposition framework that swaps traditional directed generative models for undirected deep Energy-Based Models (EBMs). By discarding rigid structural and distributional assumptions, this approach allows neural networks to learn complex, non-Gaussian data patterns and unified representations for both static and dynamic datasets.

The Challenge with Traditional Tensor Decompositions

Tensor Decompositions (TDs) are the multi-dimensional extensions of matrix factorization, used to find latent patterns in "multiway" data (such as a 3D block of user-item-time interactions).

Standard TD methods typically rely on two major assumptions:

  1. Structure: They assume a specific "contraction rule" (like CP or Tucker decomposition) that dictates how latent factors combine to create the data.
  2. Distribution: They often assume the data follows a simple distribution, usually Gaussian.

In real-world scenarios, the "true" structure or distribution of data is rarely known. Incorrect assumptions can lead to biased models and poor performance. While some non-linear TDs exist, most still rely on directed graphical models that struggle with multi-modal or highly skewed distributions.

Introducing EnergyTD

The authors introduce EnergyTD, which treats tensor decomposition as an undirected modeling problem. Instead of assuming that latent factors cause the observations, EnergyTD models the joint probability of the data ($x$) and latent factors ($z$) using an energy function: $p(x, z) \propto \exp(-f(x, z))$.

Key technical highlights include:

  • Deep EBMs: Multi-layer perceptrons (MLPs) are used to parameterize the energy function $f(x, z)$. This flexibility allows the model to learn nearly any underlying data distribution or structural correlation from scratch.
  • Unified Energy Function: For dynamic tensors (data with timestamps), a temporal encoding layer is added to the energy function, enabling the model to learn continuous-time trajectories without altering the core architecture.
  • VCNCE Objective: Training EBMs is notoriously difficult because the probability is "unnormalized" (calculating the total probability requires an intractable integral). Adding latent variables makes it "doubly intractable." To overcome this, the authors derived a Variational Conditional Noise-Contrastive Estimation (VCNCE) objective—an upper bound that trains the model by teaching it to distinguish real data from "conditional noise."

Datasets and Evaluation

The model was evaluated on a mix of synthetic and high-dimensional real-world datasets:

  • Synthetic: Data generated from Beta, Mixture of Gaussians (MoG), and Exponential distributions to test distributional flexibility.
  • Sparse Tensor Completion: Alog (file access logs) and ACC (code repository management).
  • Continuous-Time Completion: Air (Beijing air quality) and Click (advertisement click-through data).

Key Insights and Results

EnergyTD consistently outperformed state-of-the-art baselines (like GPTF, CoSTCo, and NONFAT) across various tasks.

  • Breaking the Gaussian Habit: Most probabilistic TDs are constrained by Gaussian assumptions. EnergyTD can capture sharp edges or multiple peaks in data density that Gaussian models typically smooth over.
  • Undirected Flexibility: By modeling the joint energy rather than a directed generative path, the model learns more general correlations between tensor modes.
  • Time as a Feature: Instead of treating time as a separate dimension to be decomposed, EnergyTD treats it as an auxiliary feature in the energy function, simplifying the handling of continuous-time data.

Distributional Accuracy

Traditional Gaussian-based methods often fail to capture the true shape of non-Gaussian data. EnergyTD closely tracks the ground truth, especially in complex cases like Mixture of Gaussians.

Simulation results for Mixture of Gaussians. EnergyTD (green) significantly outperforms Gaussian-based GPTF (red) in matching the ground truth (blue).

Continuous-Time Completion

The model demonstrates a strong ability to recover missing data in time-series tensors. It captures both low-frequency trends and high-frequency fluctuations better than specialized dynamic baselines.

Continuous-time tensor decomposition results. EnergyTD (green) accurately recovers trajectories even in missing regions (red).

On real-world datasets like Air and Click, EnergyTD achieved the lowest Root Mean Square Error (RMSE) and Mean Absolute Error (MAE) compared to six competitive baselines, often by statistically significant margins.

Limitations and Future Work

While powerful, the authors note a few areas for potential improvement:

  • Variational Tightness: The current loss function is not a "fully" variational upper bound because it uses importance sampling for noise samples. A tighter bound could further improve training efficiency.
  • Interpretability: As with many deep learning models, the learned latent factors in EnergyTD are harder to interpret than those in classical linear CP or Tucker models.
  • Architecture: The current version uses simple MLPs. Incorporating convolutional or transformer-based architectures into the energy function could yield even better results for specific data modalities.

Summary updated

Abstract

Tensor decompositions (TDs) serve as a powerful tool for analyzing multiway data. Traditional TDs incorporate prior knowledge about the data into the model, such as a directed generative process from latent factors to observations. In practice, selecting proper structural or distributional assumptions beforehand is crucial for obtaining a promising TD representation. However, since such prior knowledge is typically unavailable in real-world applications, choosing an appropriate TD model can be challenging. This paper aims to address this issue by introducing a flexible TD framework that discards the structural and distributional assumptions, in order to learn as much information from the data. Specifically, we construct a TD model that captures the joint probability of the data and latent tensor factors through a deep energy-based model (EBM). Neural networks are then employed to parameterize the joint energy function of tensor factors and tensor entries. The flexibility of EBM and neural networks enables the learning of underlying structures and distributions. In addition, by designing the energy function, our model unifies the learning process of different types of tensors, such as static tensors and dynamic tensors with time stamps. The resulting model presents a doubly intractable nature due to the presence of latent tensor factors and the unnormalized probability function. To efficiently train the model, we derive a variational upper bound of the conditional noise-contrastive estimation objective that learns the unnormalized joint probability by distinguishing data from conditional noises. We show advantages of our model on both synthetic and several real-world datasets.
BibTeX
@inproceedings{tao2023undirected,
  author = {Zerui Tao and Toshihisa Tanaka and Qibin Zhao},
  title = {{Undirected Probabilistic Model for Tensor Decomposition}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=3NWWgB2SuF},
}
Figure