Nonparametric Factor Trajectory Learning for Dynamic Tensor Decomposition

Zheng Wang, Shandian Zhe

Introduction

Tensor decomposition is a fundamental framework for analyzing multi-dimensional data, such as user-item-location interactions or spatiotemporal environmental readings. Traditional methods, including CANDECOMP/PARAFAC (CP) and Tucker decomposition, typically assume that the latent factors representing entities are static. However, in real-world applications, entities are rarely stationary; user preferences shift, product trends evolve, and environmental conditions fluctuate.

To address this, researchers have developed dynamic tensor decomposition models. Yet, existing approaches often rely on discretizing time into bins—which sacrifices fine-grained detail and struggles with data sparsity—or utilize time-domain Gaussian Processes (GPs) that suffer from "mean reversion," where predictions become unreliable and decay toward zero when extrapolating beyond observed timestamps. This paper introduces NONFAT (Nonparametric FActor Trajectory learning), a framework that learns continuous, time-varying latent factors by operating in the frequency domain.

The NONFAT Framework

NONFAT moves beyond static representations by modeling the temporal evolution of factors as continuous trajectories. The core innovation lies in a bi-level Bayesian nonparametric approach that leverages frequency-domain modeling to ensure robust long-term extrapolation.

1. Frequency Domain Factor Learning

Instead of modeling factor trajectories directly in the time domain, NONFAT places GP priors on functions within the frequency domain. By doing so, the model can capture complex temporal patterns through spectral representations. To return to the time domain, the authors apply an Inverse Fourier Transform (IFT). Because Fourier bases (sines and cosines) are periodic and do not naturally decay to zero, the resulting trajectories are significantly more robust for long-term forecasting than standard time-domain kernels (like RBF).

2. Gauss-Laguerre Quadrature

Since the IFT integral is often analytically intractable, NONFAT employs Gauss-Laguerre quadrature to approximate the transformation efficiently. This allows the model to sample trajectory values at specific time points while maintaining mathematical rigor and computational stability.

3. Bi-level Gaussian Process Structure

Once the factor trajectories are established, a second-level GP maps these latent factors to the actual observed tensor entry values. This hierarchical structure allows the model to capture non-linear interactions between different entities across the tensor modes, providing a flexible and expressive representation of the data.

Scalable Inference via Matrix Gaussian Structures

A significant challenge in GP-based tensor models is computational complexity. Standard GP inference scales cubically with the number of observations, which is prohibitive for large tensors. NONFAT overcomes this by leveraging Matrix Gaussian structures. By exploiting the grid-like relationship between factors and quadrature nodes, the authors decompose massive covariance matrices. This reduces the computational complexity from $O((d_k C)^3)$ to $O(C^3 + d_k^3)$, where $d_k$ is the number of entities and $C$ is the number of quadrature nodes. This optimization, combined with a nested sparse variational learning algorithm, allows NONFAT to scale to large-scale real-world datasets.

Experimental Evaluation

The authors evaluated NONFAT against state-of-the-art baselines, including CPCT, GPCT, and various RNN-based dynamic models, across three diverse datasets:

  • Beijing Air Quality: A 2-mode tensor (pollutant, site) tracking six pollutants across 12 sites over four years.
  • Mobile Ads: A 3-mode tensor (banner-position, site domain, mobile app) containing 50,000 click-through records.
  • DBLP: A 3-mode tensor (author, conference, keyword) representing bibliographic records over a decade.

Key Results

  • Predictive Accuracy: NONFAT consistently achieved the lowest Root Mean Square Error (RMSE) across all datasets, demonstrating superior predictive power in both interpolation and extrapolation tasks.
  • Uncertainty Quantification: The model showed a significant lead in test log-likelihood, indicating that its probabilistic framework provides more reliable uncertainty estimates than deterministic or simpler probabilistic models.
  • Qualitative Insights: The model successfully recovered interpretable temporal patterns, such as the periodic cycles of air quality sensors and the specific lifecycle trends of different pollutants.

Conclusion

NONFAT represents a significant step forward in dynamic tensor decomposition. By shifting latent factor modeling to the frequency domain and utilizing efficient Matrix Gaussian inference, it provides a robust, scalable, and continuous-time solution for multi-dimensional data analysis. While the bi-level GP structure introduces more complexity than static models, its ability to handle data sparsity and provide stable long-term extrapolations makes it a powerful tool for modern spatiotemporal and behavioral analytics.

Summary updated

Abstract

Tensor decomposition is a fundamental framework to analyze data that can be represented by multi-dimensional arrays. In practice, tensor data is often accompanied by temporal information, namely the time points when the entry values were generated. This information implies abundant, complex temporal variation patterns. However, current methods always assume the factor representations of the entities in each tensor mode are static, and never consider their temporal evolution. To fill this gap, we propose NONparametric FActor Trajectory learning for dynamic tensor decomposition (NONFAT). We place Gaussian process (GP) priors in the frequency domain and conduct inverse Fourier transform via Gauss-Laguerre quadrature to sample the trajectory functions. In this way, we can overcome data sparsity and obtain robust trajectory estimates across long time horizons. Given the trajectory values at specific time points, we use a second-level GP to sample the entry values and to capture the temporal relationship between the entities. For efficient and scalable inference, we leverage the matrix Gaussian structure in the model, introduce a matrix Gaussian posterior, and develop a nested sparse variational learning algorithm. We have shown the advantage of our method in several real-world applications.
BibTeX
@inproceedings{wang2022nonparametric,
  author = {Zheng Wang and Shandian Zhe},
  title = {{Nonparametric Factor Trajectory Learning for Dynamic Tensor Decomposition}},
  booktitle = {International Conference on Machine Learning},
  year = {2022},
  url = {https://openreview.net/forum?id=gkCzDNS84Q},
}
Figure