Data is rarely flat. While classical machine learning relies heavily on matrices (two-dimensional arrays) to represent relationships between samples and features, real-world data frequently manifests as multi-dimensional arrays, or tensors. A video sequence is a 3D tensor (width $\times$ height $\times$ time); a color video is 4D (adding a channel dimension); and multi-temporal remote sensing or 5D light-field imaging pushes the dimensionality even further.
For years, the standard approach to handling multi-dimensional data was either to flatten it into a massive matrix or to apply specialized third-order (3D) tensor tools. However, flattening destroys the intrinsic multi-way structural dependencies of the data, while 3D tensor tools hit a hard mathematical wall when forced to scale further.
The frontier of tensor research focuses on breaking this "order barrier"—generalizing foundational matrix and 3D tensor operations to arbitrary higher-order ($N$-order) spaces.
The Algebraic Breakdown: Why 3D Tools Fail in $N$-D
To understand why generalizing to higher orders is difficult, we must look at how we handle 3D tensors. A popular framework for 3D tensors is the tensor Singular Value Decomposition (t-SVD), which relies on the t-product. The intuition behind the t-product is to treat a 3D tensor as a matrix where every single entry is a "tube" extending into the third dimension. This allows researchers to mimic matrix algebra by performing operations in the Fourier domain along that third axis.
However, this elegant trick is structurally rigid. As Ju et al. demonstrate when attempting to define geometric means for positive definite tensors, these frameworks are strictly bound to third-order structures due to the specific algebraic nature of the underlying product operations. When you add a fourth or fifth dimension, there is no longer a single, unambiguous "tube" axis.
Furthermore, low-order tensor products suffer from a phenomenon known as Slice Permutations Variability (SPV). As identified in Handling Slice Permutations Variability in Tensor Recovery, simply reordering the frames of a video sequence can drastically alter the performance and stability of t-product algorithms. In higher-order tensors, this permutation instability compounds exponentially across multiple axes. Without a true $N$-order product definition, maintaining consistent geometric and algebraic properties across arbitrary dimensions remains an open theoretical challenge.
The Twin Bottlenecks: Parameter Explosion and Optimization Landscapes
Beyond the algebraic hurdles, scaling to higher orders introduces severe computational and optimization challenges.
The first challenge is parameter explosion. In natural language processing, Fried et al. showed that while transitive verbs can be modeled efficiently using low-rank 3D tensors, moving to more complex linguistic structures like ditransitive verbs requires higher-order tensors. As the order increases, the number of parameters grows exponentially, demanding aggressive and robust compression techniques.
The second challenge lies in the optimization landscape. In matrix land, we minimize rank using the nuclear norm, which is convex and well-behaved. But in higher-order spaces, defining and minimizing "rank" becomes highly non-convex and plagued by local minima. Zhao et al. and Wang et al. note that most algebraic frameworks for rank minimization are rigidly designed for third-order tensors. When generalized to $N$-order tensors, standard convex surrogates introduce massive estimation bias.
Matrix (2D) 3D Tensor (e.g., t-SVD) Higher-Order Tensor (N-D)
[ x x ] [ x x ] / x [ x x ] / x ... / x
[ x x ] [ x x ] / x [ x x ] / x ... / x
Convex Rank "Tube" Algebra Exponential Parameters
Global Minima Permutation Sensitive Highly Non-Convex Landscape
Even spectral methods struggle to scale. For instance, Ding et al. developed fast "lifting" techniques to decompose overcomplete third-order tensors, but these mathematical liftings do not automatically translate to fourth-order tensors and beyond. Escaping local minima in these higher-order, highly non-convex landscapes requires entirely new optimization paradigms, a difficulty that mirrors the challenges found in non-convex matrix factorizations described by Lahat et al..
Pathways to True Higher-Order Scalability
To bypass these limitations, researchers are developing frameworks that natively accommodate arbitrary dimensions without relying on low-order reductions.
Advanced Structural Networks
Instead of forcing a higher-order tensor into a single low-rank approximation, frameworks are shifting toward hierarchical structures like Tensor Trains or Tree Tensor Networks. As Ceruti et al. demonstrate, evolving basis matrices and core coefficients simultaneously within these tree structures allows for a high degree of concurrent sub-problem execution. This parallel integration bypasses the serial bottlenecks that typically plague matrix differential equations, offering a scalable path for dynamical approximations.
Randomized Low-Rank Approximations
To combat the computational cost of evaluating high-order arrays, researchers are turning to randomized algorithms. Wenjin Qin introduced a framework that pairs double non-convex models with randomized low-rank approximations. By sketching the tensor through random projections, the algorithm drastically cuts down the cost of multi-dimensional Alternating Direction Method of Multipliers (ADMM) updates, enabling robust tensor completion to scale to 5D datasets efficiently.
Continuous Functional Representations
Perhaps the most radical departure from classical tensor algebra is treating tensors not as discrete grids of numbers, but as continuous functions. Xi Zhang proposed Online Functional Tensor Decomposition, which parameterizes tensor decompositions using Implicit Neural Representations (INRs). By modeling streaming data as a continuous function over coordinate spaces, this approach circumvents the rigid structural constraints of classical Canonical Polyadic (CP) or Tucker decompositions, allowing higher-order representations to adapt dynamically over time.