In modern scientific computing, we often encounter data that lives in many dimensions simultaneously. A video is a 3D array (width $\times$ height $\times$ time); a color video is 4D (adding a color channel); and a quantum simulation or a complex climate model might involve dozens of dimensions. These multi-way arrays are called tensors.
The fundamental problem with tensors is the "curse of dimensionality." If you have a 10-dimensional tensor where each dimension has a size of 100, storing it would require $100^{10}$ entries—more than the number of atoms in the human body. As Tianyi Shi and Alex Townsend explore in On the compressibility of tensors, this exponential growth makes raw storage impossible.
Low-rank tensor network decompositions solve this by breaking a massive, high-dimensional tensor into a set of small, interconnected "cores." Instead of storing every entry, we store the rules for how these small cores interact to reconstruct the data.
The Intuition of Factorization
To understand tensor networks, start with a simple matrix (a 2D tensor). If a matrix has a "low rank," it means its rows or columns are redundant. We can factorize a large $M \times N$ matrix into two skinny matrices ($M \times r$ and $r \times N$).
Tensor networks extend this logic to $d$ dimensions. The most common format is the Tensor Train (TT). In a Tensor Train, a $d$-order tensor is represented as a chain of 3D cores. Each core is linked to its neighbors by an internal "rank." If these ranks are small, the total number of parameters scales linearly with the number of dimensions ($d$), rather than exponentially.
Ivan Oseledets and colleagues have shown that many analytical functions can be "compiled" directly into these sparse TT cores, as detailed in Constructive TT-representation of the tensors given as index interaction functions with applications. This allows researchers to perform complex calculations, like computing matrix permanents or game theory equilibria, without ever forming the full, massive tensor.
Beyond the Chain: Rings and Hierarchies
While the Tensor Train is efficient, it has a weakness: it is sensitive to the order of dimensions. If you swap the first and last dimension of your data, the TT rank might explode.
To fix this, Masashi Sugiyama and Qibin Zhao introduced the Tensor Ring (TR) decomposition in Learning Efficient Tensor Representations with Ring Structure Networks. By connecting the two ends of the "train" to form a loop, the representation becomes invariant to circular shifts of the dimensions and often achieves much higher compression ratios.
Other researchers look toward hierarchical structures. Holger Rauhut and Reinhold Schneider utilize Hierarchical Tucker formats, which organize dimensions into a tree structure. In Tensor completion in hierarchical tensor representations, they demonstrate that these hierarchies allow for "compressed sensing" of tensors—recovering the entire high-dimensional structure from just a few random measurements.
Scaling with Randomization and Sketching
Even with these compressed formats, the process of finding the decomposition (the "fitting" step) can be slow. Traditional methods like the Singular Value Decomposition (SVD) or Alternating Least Squares (ALS) usually require looking at every entry of the tensor.
To bypass this, the field has moved toward randomized algorithms. The intuition is similar to political polling: you don't need to ask every citizen their opinion to understand the trend; you just need a representative sample.
- Leverage Score Sampling: Osman Asif Malik and Stephen Becker developed sublinear algorithms for Tensor Rings in A Sampling-Based Method for Tensor Ring Decomposition. By sampling entries based on their "importance" (leverage scores), they can decompose a tensor without ever reading most of its values.
- Tensor Sketching: Thomas D. Ahle and Jakob Bæk Tejs Knudsen use "sketches"—random projections that squash the tensor into a smaller space while preserving its essential geometry. Their work in Almost Optimal Tensor Sketch provides a way to perform dimensionality reduction that doesn't grow exponentially with the tensor's order.
- Oblivious Embeddings: Bei Jiang has shown in Oblivious subspace embeddings for compressed Tucker decompositions that we can use random matrices to compress tensors "blindly"—without knowing the data distribution beforehand—and still maintain rigorous accuracy guarantees.
Multi-resolution and Fusion
Recent work has pushed these ideas into multi-scale analysis. Oscar Mickelin and Antonio G. Marques have proposed formats that represent a tensor as a sum of components at different resolutions. In A Multi-resolution Low-rank Tensor Decomposition, they show that by approximating data on increasingly coarse grids, they can capture both global trends and local details more efficiently than standard flat decompositions.
Furthermore, these networks can be used for "feature fusion." Giuseppe G. Calvi and Ilia Kisil describe a method in The sum of tensor networks to stack the cores of different networks. This allows multiple datasets to be merged into a single compressed representation, which is particularly useful for machine learning tasks where data is scarce but high-dimensional.
By shifting from dense arrays to these structured, low-rank networks, researchers can now process datasets that were previously considered "uncomputable," bridging the gap between theoretical physics, high-performance computing, and modern machine learning.