Computing Directly on Compressed Tensor Representations

This direction is merged into another direction: tensor decomposition scalability. Its last version stays readable and citable here.

Version 1 of 115 papers16 conceptsfrozen 2026-10-04

The "curse of dimensionality" is a fundamental wall in modern computing. As we move from 2D images to 3D physical simulations or high-dimensional data arrays (tensors), the memory required to store these objects grows exponentially. A grid with $2^{60}$ points is physically impossible to store in raw form.

Traditionally, we solve this by compressing the data after the fact. However, a new research direction focuses on a more radical approach: performing the entire computation—optimization, linear system solving, and training—directly within the compressed format. By never expanding the tensor into its dense, high-dimensional form, researchers can solve problems with millions or even quintillions of degrees of freedom on standard hardware.

The Memory Wall and the Tensor Core

When we represent a high-dimensional tensor using formats like the Tensor Train (TT) or Tucker decomposition, we break a massive multi-dimensional array into a sequence of small, interconnected "cores."

The breakthrough in this field is the realization that we don't need the full tensor to perform linear algebra. As Alexandra Bünger and Sergey Dolgov demonstrate in their work on PDE-constrained optimization, complex 3D system matrices can be treated as a sum of separable components. Instead of multiplying a massive matrix by a vector, they perform "dimensional sweeps," updating the compressed factors locally. This allows them to bypass the "memory wall," solving problems that would otherwise require supercomputers on a simple desktop.

Sketching the Subspace

A parallel strategy involves sketching: using random projections to create a low-dimensional "summary" of a tensor. The intuition is that high-dimensional data often sits on a much simpler, low-rank manifold.

Yining Wang, Anima Anandkumar, and their colleagues showed that tensor contractions—the core of most optimization algorithms—can be executed directly on these sketches. By performing computations in the Fourier domain, they achieved 100x speedups.

This isn't just a heuristic; it has rigorous theoretical backing. Bei Jiang and Linglong Kong established guarantees for "oblivious" embeddings—dimensionality reduction tools that don't even need to see the data to be effective. Their HOOI-RE algorithm maintains latent factors in a reduced state throughout the entire optimization process, ensuring that the computational overhead stays low from start to finish.

Iterative Optimization in Compressed Space

The most powerful application of this idea is in solving differential equations and large-scale simulations. Standard solvers require storing the entire solution vector, which is impossible for ultra-fine meshes.

Researchers like Ivan Oseledets and Maxim Rakhuba have pioneered the use of the Quantized Tensor Train (QTT) format. By treating the indices of a vector as a tensor itself, they can solve 2D diffusion equations on grids with $2^{60}$ points. The solver works by adaptively seeking the best low-rank approximation of the solution using alternating optimization.

Similarly, Ngoc Quoc Tran and a team at Los Alamos National Laboratory developed TT-IGA, which uses the Alternating Minimal Energy algorithm to solve 3D Poisson problems with 500 million degrees of freedom. The solver never "sees" the 500 million points; it only ever manipulates the small tensor cores that represent them.

Beyond Simulations: Neural Networks and Clustering

This "compute-in-compressed-space" philosophy is also transforming machine learning.

  • Neural Networks: Shiva Kasiviswanathan and Nina Narodytska use tensor sketching to approximate neural network layers. By reshaping weights into tensors and projecting them, they can train models from scratch in a compressed format, making them suitable for embedded devices.
  • Clustering: Hui Chen and Wanquan Liu developed data-adaptive transforms that incorporate subspace projections into iterative clustering. This reduces the dimensionality early, allowing for robust analysis of noisy, high-dimensional datasets.
  • Sublinear Decomposition: Osman Asif Malik and Stephen Becker pushed this to the limit with a sampling-based method for Tensor Ring decomposition. Their algorithm has sublinear complexity, meaning it can decompose a massive tensor without even looking at every entry, by constructing "sketched" least squares problems directly from the compressed cores.

By shifting the focus from "compressing data" to "compressing the computation itself," these researchers have turned the curse of dimensionality into a manageable engineering challenge.

Papers in this version

History & citationVersion 1 · 2026-10-04

Merged into another direction · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_computing_directly_on_compressed_tensor__259e8932,
  title = {Computing Directly on Compressed Tensor Representations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/computing-directly-on-compressed-tensor-representations/rdr_259e8932380144f584d16ba91692a2ad},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_computing_directly_on_compressed_tensor__259e8932_v1,
  title = {Computing Directly on Compressed Tensor Representations},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/computing-directly-on-compressed-tensor-representations/rdr_259e8932380144f584d16ba91692a2ad/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}