Tensors are the natural mathematical language for multi-dimensional data, offering a powerful way to bypass the "curse of dimensionality" by compressing massive datasets into compact representations (Andrzej Cichocki et al., Tensor Networks for Dimensionality Reduction and Large-scale Optimization: Part 2 Applications and Future Perspectives). However, a long-standing computational bottleneck has limited their scalability on modern hardware: the reliance on sequential optimization.
Traditionally, decomposing a tensor requires fixing all factor matrices except one, updating that single matrix, and then moving to the next mode. This alternating approach creates strict sequential dependencies that leave modern multi-core processors and GPUs heavily underutilized.
To unlock true hardware acceleration, a prominent research direction focuses on simultaneous factor updates. By mathematically restructuring optimization frameworks, these methods break the sequential chain, allowing all factor matrices, tensor modes, or network cores to be updated concurrently.
The Alternating Bottleneck: Gauss-Seidel vs. Jacobi
To understand why simultaneous updates are necessary, consider the difference between two classic iterative strategies: Gauss-Seidel and Jacobi.
Traditional tensor optimization algorithms—such as Alternating Least Squares (ALS) or standard Alternating Direction Method of Multipliers (ADMM)—operate like a Gauss-Seidel sweep. If a tensor has three modes ($A$, $B$, and $C$), the algorithm updates $A$, uses the new $A$ to update $B$, and uses both to update $C$. Because each step requires the immediate output of the previous one, the process cannot be parallelized across the modes.
Simultaneous factor updates shift this paradigm toward a Jacobi-style execution. In a Jacobi framework, updates for $A$, $B$, and $C$ are calculated concurrently using only the information available from the previous iteration.
Sequential (Gauss-Seidel):
[Update A] ----> [Update B] ----> [Update C]
Simultaneous (Jacobi):
/---> [Update A] ---\
[Previous State] ----> [Update B] ----> [Next State]
\---> [Update C] ---/
The authors of Generalized Higher-Order Tensor Decomposition via Parallel ADMM (Fanhua Shang et al.) leveraged this insight by introducing a Jacobi-type parallelization of ADMM. By eliminating the sequential dependency between modes, their framework allows high-order tensor factors to be computed at the exact same time across distributed computing architectures.
This transition from sequential dependencies to parallel Jacobian updates has broad utility beyond pure tensor decomposition. For instance, it has been successfully adapted to accelerate massive MIMO signal detection in low-latency communications (Ali A. Elghariani et al., A Proximal Jacobian ADMM Approach for Fast Massive MIMO Signal Detection in Low-Latency Communications) and to optimize joint energy-bandwidth allocations (Vaneet Aggarwal et al., Joint Energy-Bandwidth Allocation for Multi-User Channels with Cooperating Hybrid Energy Nodes).
Decoupling Modes via Expectation-Maximization
Another elegant strategy for achieving simultaneous updates is the Expectation-Maximization (EM) framework. While tensor decomposition under certain divergences is traditionally considered computationally hard, reformulating the problem through a probabilistic lens can decouple the parameters.
In Kullback-Leibler Principal Component for Tensors is not NP-hard, Kejun Huang et al. demonstrated that an EM framework allows for the simultaneous update of all factor matrices in multi-component tensor approximations. By treating the latent tensor components as hidden variables, the M-step of the algorithm naturally separates into independent subproblems for each dimension. Instead of alternating through modes, the optimization architecture updates every factor matrix at once, providing massive parallelization gains on large datasets.
Row-Wise and Mode-Wise Independence
For extremely large or sparse tensors, updating entire factor matrices simultaneously can still introduce memory bottlenecks. To combat this, researchers have drilled down to finer grains of independence within the tensor structure itself.
One approach is to exploit row-wise independence. In Scalable Tucker Factorization for Sparse Tensors - Algorithms and Discoveries, Namyong Park et al. developed a row-wise update rule for Tucker factorization. Because the update for each individual row of a factor matrix depends only on observed entries in the sparse tensor, every row can be processed completely independently. This allows multi-core processors to dynamically schedule and update rows concurrently, maximizing CPU utilization.
Alternatively, the tensor can be decomposed into independent matrix factorization problems for each mode. Haijin Zeng et al. utilized this in Enhanced Nonconvex Low-Rank Approximation of Tensor Multi-Modes for Tensor Completion, performing parallel matrix factorizations for each mode of the tensor independently. By avoiding a direct, centralized full tensor decomposition, they bypassed expensive singular value decompositions (SVDs) and distributed the computational workload efficiently.
Scaling Up Advanced Tensor Networks
The paradigm of simultaneous updates becomes even more critical when moving from standard decompositions (like CP or Tucker) to advanced tensor networks, such as Tensor Trains (TT) and Tree Tensor Networks (TTN). These networks represent high-dimensional data as a chain or tree of interconnected low-order tensors (cores).
Traditional algorithms for constructing Tensor Trains, such as the standard Tensor-Train SVD (TTSVD), are inherently sequential—they compute one core, deflate the tensor, and move to the next. To break this sequence, Tianyi Shi et al. uncovered mathematical relationships between different tensor unfoldings in Parallel Algorithms for Computing the Tensor-Train Decomposition. By proving that the column spaces of different unfoldings can be computed simultaneously rather than sequentially, the authors unlocked a fully parallelizable approach to TT core generation. This structural independence is similarly exploited when training polynomial classifiers, where individual TT-cores can be updated concurrently via distributed linear systems or convex optimization (Zhongming Chen et al., Parallelized Tensor Train Learning of Polynomial Classifiers).
Sequential TT-SVD:
[Core 1] ----> [Core 2] ----> [Core 3] ----> [Core 4]
Parallel Basis Update:
[Unfolding 1] ---> [Core 1]
[Unfolding 2] ---> [Core 2] (All computed simultaneously)
[Unfolding 3] ---> [Core 3]
For Tree Tensor Networks, the sequential bottleneck is even more severe because updates traditionally crawl level-by-level through the tree hierarchy. In A Parallel Basis Update and Galerkin Integrator for Tree Tensor Networks, Gianluca Ceruti et al. solved this by introducing a parallel time-stepping integrator. By integrating the underlying differential equations for augmented basis and coefficient matrices concurrently, their algorithm updates all basis matrices and connecting tensors within the network simultaneously within a single time step.
Summary
The shift from sequential alternating updates to simultaneous factor updates represents a fundamental re-engineering of tensor mathematics for the parallel computing era. Whether by replacing Gauss-Seidel steps with Jacobi updates, leveraging EM formulations to decouple latent dimensions, or exploiting row-wise and unfolding independence, this research direction ensures that high-dimensional tensor operations can fully exploit modern hardware.
Key Foundational Work
Generalized Higher-Order Tensor Decomposition via Parallel ADMM (2014) — contributed 1 concept elements to this direction.
Parallelized factor analysis and feature normalization for automatic speaker verification (2008) — contributed 1 concept elements to this direction.