Introduction
Researchers have developed EnergyTD, a novel tensor decomposition framework that swaps traditional directed generative models for undirected deep Energy-Based Models (EBMs). By discarding rigid structural and distributional assumptions, this approach allows neural networks to learn complex, non-Gaussian data patterns and unified representations for both static and dynamic datasets.
The Challenge with Traditional Tensor Decompositions
Tensor Decompositions (TDs) are the multi-dimensional extensions of matrix factorization, used to find latent patterns in "multiway" data (such as a 3D block of user-item-time interactions).
Standard TD methods typically rely on two major assumptions:
- Structure: They assume a specific "contraction rule" (like CP or Tucker decomposition) that dictates how latent factors combine to create the data.
- Distribution: They often assume the data follows a simple distribution, usually Gaussian.
In real-world scenarios, the "true" structure or distribution of data is rarely known. Incorrect assumptions can lead to biased models and poor performance. While some non-linear TDs exist, most still rely on directed graphical models that struggle with multi-modal or highly skewed distributions.
Introducing EnergyTD
The authors introduce EnergyTD, which treats tensor decomposition as an undirected modeling problem. Instead of assuming that latent factors cause the observations, EnergyTD models the joint probability of the data ($x$) and latent factors ($z$) using an energy function: $p(x, z) \propto \exp(-f(x, z))$.
Key technical highlights include:
- Deep EBMs: Multi-layer perceptrons (MLPs) are used to parameterize the energy function $f(x, z)$. This flexibility allows the model to learn nearly any underlying data distribution or structural correlation from scratch.
- Unified Energy Function: For dynamic tensors (data with timestamps), a temporal encoding layer is added to the energy function, enabling the model to learn continuous-time trajectories without altering the core architecture.
- VCNCE Objective: Training EBMs is notoriously difficult because the probability is "unnormalized" (calculating the total probability requires an intractable integral). Adding latent variables makes it "doubly intractable." To overcome this, the authors derived a Variational Conditional Noise-Contrastive Estimation (VCNCE) objective—an upper bound that trains the model by teaching it to distinguish real data from "conditional noise."
Datasets and Evaluation
The model was evaluated on a mix of synthetic and high-dimensional real-world datasets:
- Synthetic: Data generated from Beta, Mixture of Gaussians (MoG), and Exponential distributions to test distributional flexibility.
- Sparse Tensor Completion:
Alog(file access logs) andACC(code repository management). - Continuous-Time Completion:
Air(Beijing air quality) andClick(advertisement click-through data).
Key Insights and Results
EnergyTD consistently outperformed state-of-the-art baselines (like GPTF, CoSTCo, and NONFAT) across various tasks.
- Breaking the Gaussian Habit: Most probabilistic TDs are constrained by Gaussian assumptions. EnergyTD can capture sharp edges or multiple peaks in data density that Gaussian models typically smooth over.
- Undirected Flexibility: By modeling the joint energy rather than a directed generative path, the model learns more general correlations between tensor modes.
- Time as a Feature: Instead of treating time as a separate dimension to be decomposed, EnergyTD treats it as an auxiliary feature in the energy function, simplifying the handling of continuous-time data.
Distributional Accuracy
Traditional Gaussian-based methods often fail to capture the true shape of non-Gaussian data. EnergyTD closely tracks the ground truth, especially in complex cases like Mixture of Gaussians.
Continuous-Time Completion
The model demonstrates a strong ability to recover missing data in time-series tensors. It captures both low-frequency trends and high-frequency fluctuations better than specialized dynamic baselines.
On real-world datasets like Air and Click, EnergyTD achieved the lowest Root Mean Square Error (RMSE) and Mean Absolute Error (MAE) compared to six competitive baselines, often by statistically significant margins.
Limitations and Future Work
While powerful, the authors note a few areas for potential improvement:
- Variational Tightness: The current loss function is not a "fully" variational upper bound because it uses importance sampling for noise samples. A tighter bound could further improve training efficiency.
- Interpretability: As with many deep learning models, the learned latent factors in EnergyTD are harder to interpret than those in classical linear CP or Tucker models.
- Architecture: The current version uses simple MLPs. Incorporating convolutional or transformer-based architectures into the energy function could yield even better results for specific data modalities.
Summary updated