When data possesses multiple distinct dimensions—such as a recommendation system tracking (user, item, time) or a medical scan tracking (spatial-x, spatial-y, spatial-z)—we typically organize it as a tensor. Traditional multiway data analysis relies on classical tensor decompositions like CANDECOMP/PARAFAC (CP) or Tucker decomposition.
However, these classical methods suffer from a fundamental structural assumption: they assume that the interactions between different modes are strictly multilinear. In a standard CP decomposition, the value of an entry is modeled as the sum of element-wise products of latent vectors across modes:
$$\mathcal{X}_{ijk} \approx \sum_{r=1}^R u_{ir} v_{jr} w_{kr}$$
This multiplicative formulation implies that the relationship between dimensions is strictly proportional. If a user's affinity for a movie changes, or if a temporal effect saturates, a purely linear or multiplicative model cannot naturally capture it. Real-world dynamics often exhibit thresholds, diminishing returns, or highly irregular non-linear dependencies. As noted by Zenglin Xu et al., relying solely on multilinear factorization prevents models from accurately representing the complex, coupled structures inherent in modern datasets.
To overcome this limitation, a prominent research direction has emerged that replaces rigid multilinear mappings with flexible, non-linear functions. Instead of multiplying latent factors, these approaches feed latent representations into non-linear operators—such as Gaussian Processes, kernels, or deep neural networks—to model multiway interactions.
The Nonparametric Bayesian Approach: Tensor Gaussian Processes
One of the most mathematically elegant ways to introduce non-linearity into tensor decomposition is to place a Gaussian Process (GP) over the interaction space. Instead of a core tensor or a direct product, the multiway interaction is governed by a continuous, non-linear function $f(\mathbf{u}_i, \mathbf{v}_j, \mathbf{w}_k)$ sampled from a GP.
Zenglin Xu et al. pioneered this concept with InfTucker (Infinite Tucker Decomposition: Nonparametric Bayesian Models for Multiway Data Analysis), which elevates classical Tucker decomposition into an infinite-dimensional feature space. By leveraging latent Gaussian and $t$-processes, the model can learn arbitrary smooth interaction surfaces between entities. Heishiro Kanagawa et al. later provided a rigorous statistical foundation for this paradigm, proving that Gaussian Process tensor estimators achieve minimax optimality, meaning they provide the best possible theoretical generalization error when estimating complex, multi-way interactions (Gaussian process nonparametric tensor estimator and its minimax optimality).
Overcoming the Computational Bottleneck
While GPs offer exceptional flexibility, they scale poorly with data size, typically requiring $\mathcal{O}(N^3)$ time complexity. In tensor settings, where the number of entries explodes combinatorially with the number of modes, standard GPs quickly become intractable.
To scale non-linear tensor analysis to massive datasets, researchers introduced local and distributed approximations:
- Local GPs and Variational Inference: Shandian Zhe et al. developed a scalable framework that replaces a single global GP with a collection of local approximations paired with an online Variational Bayes learning algorithm (Scalable Nonparametric Multiway Data Analysis).
- Distributed Architectures: Shandian Zhe et al. introduced DinTucker (DinTucker: Scaling Up Gaussian Process Models on Large Multidimensional Arrays), which distributes local GPs across large-scale clusters to handle arrays with billions of entries.
- Sparsity and Discrete Data: Real-world tensors are often extremely sparse or contain discrete counts rather than continuous values. Conor Tillinghast et al. combined Completely Random Measures with GPs to explicitly model structural sparsity (Nonparametric Decomposition of Sparse Tensors), while Zerui Tao et al. utilized Polya-Gamma augmentation to efficiently handle binary and count data without losing non-linear expressivity (Efficient Nonparametric Tensor Decomposition for Binary and Count Data).
Kernelized Tensor Learning for Structured Data
When applying tensor decomposition to supervised learning tasks—such as classifying 3D neuroimages or fMRI scans—flattening the tensor into a vector destroys vital spatial and structural correlations. As explained by Lei Wang et al., maintaining the multi-mode structure is essential for preserving the data's inherent meaning (Subspace Kernel Learning on Tensor Sequences).
To bridge tensor structures with non-linear decision boundaries, researchers developed Kernelized Support Tensor Machines (KSTMs). Instead of operating on flattened vectors, these models apply kernel functions directly to the tensor modes.
Lifang He et al. introduced the DuSK (Dual Structure-preserving Kernel) framework (DuSK: A Dual Structure-preserving Kernel for Supervised Tensor Learning with Applications to Neuroimages) and later expanded it to joint optimization frameworks that learn a kernelized tensor factorization alongside a maximum-margin classifier (Kernelized Support Tensor Machines). By mapping tensor components into a high-dimensional Reproducing Kernel Hilbert Space (RKHS), complex biological patterns become linearly separable while preserving structural geometry.
To optimize these non-convex kernel objectives efficiently, Taiji Suzuki et al. demonstrated that an Alternating Minimization Procedure (AMP) can achieve linear convergence rates and reach minimax optimality, bypassing the slow optimization speeds typically associated with non-parametric tensor learning (Minimax Optimal Alternating Minimization for Kernel Nonparametric Tensor Learning).
Deep and Additive Alternatives
Beyond GPs and kernels, researchers have integrated deep neural networks and additive spline models to capture non-linear multiway dynamics.
Neural Tensor Networks
Deep learning architectures can be fused with tensor factorization layers to learn hierarchical, non-linear representations. For instance, in recommendation systems, Huiyuan Chen et al. proposed the Neural Tensor Model (NTM) (Neural Tensor Model for Learning Multi-Aspect Factors in Recommender Systems), which pairs a standard "wide" CP layer with a "deep" tensorized Multi-Layer Perceptron (MLP) to capture complex interactions between user reviews, visual styles, and temporal contexts.
For inverse problems in multi-dimensional imaging, Yisi Luo et al. introduced HLRTF (HLRTF: Hierarchical Low-Rank Tensor Factorization for Inverse Problems in Multi-Dimensional Imaging), which replaces fixed linear transforms with an unsupervised deep neural network to discover highly compact, non-linear low-rank redundancies directly from degraded images.
Additive and Spline Models
When the goal is to maintain interpretability while allowing non-linear effects like saturation or diminishing returns, additive models offer a compelling alternative. Botao Hao et al. developed Sparse Tensor Additive Regression (STAR) (Sparse Tensor Additive Regression), which integrates B-spline approximations with low-rank constraints to model non-linear relationships between multi-dimensional inputs and a scalar outcome.
Similarly, to beat the curse of dimensionality in high-dimensional tensor regression, Masaaki Imaizumi et al. proposed Additive-Multiplicative Nonparametric Regression (AMNR) (Doubly Decomposing Nonparametric Tensor Regression), which decomposes a complex multiway regression function into a sum of simpler local functions, balancing structural efficiency with non-linear flexibility.
Key Foundational Work
Infinite Tucker Decomposition: Nonparametric Bayesian Models for Multiway Data Analysis (2012) — contributed 1 concept elements to this direction.
Kernel-Based Learning from Infinite Dimensional 2-Way Tensors (2010) — contributed 1 concept elements to this direction.