Introduction
Tensors are the standard mathematical structure for representing multi-dimensional data, such as "User $\times$ Ad $\times$ Device" interaction logs. Standard tensor decomposition breaks these arrays into latent factors to predict missing values or uncover hidden patterns. However, most existing methods treat these factors as "frozen" in time. In reality, entities evolve: a user’s interests shift, a product’s popularity fluctuates, and environmental conditions change.
While some dynamic models exist, they typically discretize time into rigid buckets or assume the factors themselves are static while only their weights change. The core challenge is learning a continuous trajectory for every factor. Doing this directly in the time domain is difficult due to data sparsity; if an entity is unobserved for a period, standard Gaussian Process (GP) models often revert to a zero-mean prediction with high uncertainty.
To address this, researchers introduce NONFAT (NONparametric FActor Trajectory learning), a framework that breaks the static assumption by modeling latent factors as continuous, time-varying trajectories.
The NONFAT Methodology
NONFAT employs a bi-level inference framework and frequency-domain modeling to make continuous trajectory learning robust and scalable.
- Frequency Domain Modeling: Instead of modeling the trajectory $u(t)$ directly, NONFAT models the frequency function $\hat{u}(\omega)$ using a Gaussian Process. By recovering the trajectory via an Inverse Fourier Transform (IFT), the model leverages trigonometric bases. Unlike standard kernels that decay to zero far from training data, these bases allow the model to maintain robust estimates and capture periodic patterns over long horizons.
- Gauss-Laguerre Quadrature: To handle the complex integral required for the IFT, the authors use Gauss-Laguerre quadrature. This transforms the integral into a weighted sum of function values at specific nodes, ensuring computational tractability.
- Bi-level GP Structure: The model is hierarchical. The first level generates the latent factor trajectories from the frequency domain. Given the trajectory values at specific time points, a second-level GP maps these time-varying factors to the actual observed tensor entry values, capturing non-linear temporal relationships between entities.
- Matrix Gaussian Inference: To prevent the computational cost from exploding, the authors exploit the Matrix Gaussian structure of their priors. This allows them to decompose massive covariance matrices into smaller pieces, significantly speeding up the Variational Evidence Lower Bound (ELBO) calculations.
Key Insights
- The "Extrapolation" Advantage: Modeling in the frequency domain acts as a natural regularizer for time-series data. By learning underlying spectral components, the model extrapolates much more gracefully into the future compared to time-domain GPs, which often provide flat predictions outside the training window.
- Entity Embeddings for Scalability: Rather than learning a separate GP for thousands of individual entities, NONFAT learns a single function that takes an entity embedding and a frequency as input, allowing the model to share knowledge across similar entities.
- Calibrated Uncertainty: As a Bayesian model, NONFAT provides a measure of confidence alongside its predictions. Its uncertainty estimates are highly calibrated, showing smaller variance near observed data and growing steadily further away.
Experimental Results
The model was evaluated on three real-world datasets:
- Beijing Air Quality: A 2-mode tensor (pollutant $\times$ site) with 10K measurements from 2013 to 2017.
- Mobile Ads: A 3-mode tensor (banner-position $\times$ site $\times$ app) containing 50K click-through records.
- DBLP: A 3-mode tensor (author $\times$ conference $\times$ keyword) tracking paper publication counts.
NONFAT consistently outperformed state-of-the-art baselines (including CPCT and various neural-network-based dynamic models) across all datasets. In terms of Root Mean Square Error (RMSE), NONFAT achieved the lowest error in nearly every test case. For example, on the Beijing Air Quality data, it improved RMSE by approximately 10-15% over the next best GP-based baseline. Furthermore, the model showed a massive advantage in test log-likelihood, indicating highly reliable probabilistic predictions.
Limitations and Future Work
While highly effective, NONFAT's bi-level GP is more computationally intensive than simple matrix factorization, making scaling to tensors with millions of modes a remaining challenge. Additionally, mapping the learned frequency functions back to human-understandable events (e.g., identifying a specific frequency as a "weekly shopping cycle") is not yet automated. Future work will investigate the semantic meaning of these learned frequency patterns and explore applications in domains like medical patient monitoring and financial forecasting.
Summary updated