Introduction
Tensor decomposition is a fundamental framework for analyzing multi-dimensional data, such as user-item-location interactions or spatiotemporal environmental readings. Traditional methods, including CANDECOMP/PARAFAC (CP) and Tucker decomposition, typically assume that the latent factors representing entities are static. However, in real-world applications, entities are rarely stationary; user preferences shift, product trends evolve, and environmental conditions fluctuate.
To address this, researchers have developed dynamic tensor decomposition models. Yet, existing approaches often rely on discretizing time into bins—which sacrifices fine-grained detail and struggles with data sparsity—or utilize time-domain Gaussian Processes (GPs) that suffer from "mean reversion," where predictions become unreliable and decay toward zero when extrapolating beyond observed timestamps. This paper introduces NONFAT (Nonparametric FActor Trajectory learning), a framework that learns continuous, time-varying latent factors by operating in the frequency domain.
The NONFAT Framework
NONFAT moves beyond static representations by modeling the temporal evolution of factors as continuous trajectories. The core innovation lies in a bi-level Bayesian nonparametric approach that leverages frequency-domain modeling to ensure robust long-term extrapolation.
1. Frequency Domain Factor Learning
Instead of modeling factor trajectories directly in the time domain, NONFAT places GP priors on functions within the frequency domain. By doing so, the model can capture complex temporal patterns through spectral representations. To return to the time domain, the authors apply an Inverse Fourier Transform (IFT). Because Fourier bases (sines and cosines) are periodic and do not naturally decay to zero, the resulting trajectories are significantly more robust for long-term forecasting than standard time-domain kernels (like RBF).
2. Gauss-Laguerre Quadrature
Since the IFT integral is often analytically intractable, NONFAT employs Gauss-Laguerre quadrature to approximate the transformation efficiently. This allows the model to sample trajectory values at specific time points while maintaining mathematical rigor and computational stability.
3. Bi-level Gaussian Process Structure
Once the factor trajectories are established, a second-level GP maps these latent factors to the actual observed tensor entry values. This hierarchical structure allows the model to capture non-linear interactions between different entities across the tensor modes, providing a flexible and expressive representation of the data.
Scalable Inference via Matrix Gaussian Structures
A significant challenge in GP-based tensor models is computational complexity. Standard GP inference scales cubically with the number of observations, which is prohibitive for large tensors. NONFAT overcomes this by leveraging Matrix Gaussian structures. By exploiting the grid-like relationship between factors and quadrature nodes, the authors decompose massive covariance matrices. This reduces the computational complexity from $O((d_k C)^3)$ to $O(C^3 + d_k^3)$, where $d_k$ is the number of entities and $C$ is the number of quadrature nodes. This optimization, combined with a nested sparse variational learning algorithm, allows NONFAT to scale to large-scale real-world datasets.
Experimental Evaluation
The authors evaluated NONFAT against state-of-the-art baselines, including CPCT, GPCT, and various RNN-based dynamic models, across three diverse datasets:
- Beijing Air Quality: A 2-mode tensor (pollutant, site) tracking six pollutants across 12 sites over four years.
- Mobile Ads: A 3-mode tensor (banner-position, site domain, mobile app) containing 50,000 click-through records.
- DBLP: A 3-mode tensor (author, conference, keyword) representing bibliographic records over a decade.
Key Results
- Predictive Accuracy: NONFAT consistently achieved the lowest Root Mean Square Error (RMSE) across all datasets, demonstrating superior predictive power in both interpolation and extrapolation tasks.
- Uncertainty Quantification: The model showed a significant lead in test log-likelihood, indicating that its probabilistic framework provides more reliable uncertainty estimates than deterministic or simpler probabilistic models.
- Qualitative Insights: The model successfully recovered interpretable temporal patterns, such as the periodic cycles of air quality sensors and the specific lifecycle trends of different pollutants.
Conclusion
NONFAT represents a significant step forward in dynamic tensor decomposition. By shifting latent factor modeling to the frequency domain and utilizing efficient Matrix Gaussian inference, it provides a robust, scalable, and continuous-time solution for multi-dimensional data analysis. While the bi-level GP structure introduces more complexity than static models, its ability to handle data sparsity and provide stable long-term extrapolations makes it a powerful tool for modern spatiotemporal and behavioral analytics.
Summary updated