Tensor Train Decomposition on TensorFlow (T3F)

Alexander Novikov, Pavel Izmailov, Valentin Khrulkov, Michael Figurnov, Ivan V. Oseledets

TL;DR

T3F is an open-source library that brings Tensor Train (TT) decomposition—a powerful low-rank tensor compression format—natively into TensorFlow. It bridges a major tooling gap for machine learning researchers by combining GPU acceleration, automatic differentiation, mini-batch parallel processing, and specialized Riemannian manifold optimization within a single framework.

What Problem Does This Paper Address?

High-dimensional tensors suffer from the "curse of dimensionality": their parameter counts scale exponentially with dimension ($n^d$). The Tensor Train (TT) format (also known as Matrix Product States in quantum physics) solves this by decomposing massive tensors into products of 3-way core tensors, drastically reducing parameter complexity to $\mathcal{O}(n d r^2)$, where $r$ is the TT-rank.

While widely applied to neural network compression, theoretical analysis of deep nets, and tensor completion, machine learning researchers frequently had to reimplement TT operations from scratch. Existing scientific libraries lacked deep learning necessities—specifically GPU acceleration, automatic differentiation (autodiff), parallel mini-batch processing, and optimization routines that respect the Riemannian manifold structure of fixed-rank tensors.

Methods

T3F implements tensor operations directly within the TensorFlow ecosystem:

  • Core Data Structures: Introduces TensorTrain and TensorTrainBatch classes that wrap TT-cores as native tf.Tensor objects, supporting standard TensorFlow APIs (.get_shape(), .name, etc.) across both graph and eager execution.
  • Batch Processing: Operations (e.g., t3f.matmul) support broadcasting and parallel execution across the batch dimension, enabling efficient mini-batch training.
  • Riemannian Optimization: The set of tensors of a fixed TT-rank forms a Riemannian manifold. T3F includes primitives for projecting onto tangent spaces ($P_B A$) and optimizes tailored operations such as project_sum. By leveraging the linearity of tangent space projections, project_sum reduces the asymptotic complexity of projecting a batch sum from $\mathcal{O}(b d r_A r_B n (r_A b + r_B))$ to $\mathcal{O}(b d r_A r_B n (r_A + r_B))$.
  • Automatic Riemannian Differentiation: T3F provides out-of-the-box computation for Riemannian gradients ($P_X \frac{\partial f}{\partial X}$) and linearized Riemannian Hessian-vector products ($P_{T_X}(\nabla^2 f(X)) P_{T_X} Z$) with optimal asymptotic complexity.

Datasets

The authors did not use external empirical ML datasets; instead, they evaluated performance using synthetically generated TT benchmarks:

  • Batches of 100 random TT-matrices of size $10^{10} \times 10^{10}$ ($d = 10$, TT-rank $r = 10$).
  • Batches of 100 random TT-vectors of size $10^{10} \times 1$.

Key Insights and Ideas

  • Manifold Optimization Without the Math Overhead: Optimizing low-rank factorized tensors directly via standard SGD can suffer from numerical instability and gauge ambiguity. By automating tangent-space projections and Riemannian autodiff, T3F enables principled manifold optimization with theoretical guarantees without requiring manual calculus from practitioners.
  • Algorithmic Batch Fusion: Treating batch tensor projections as a unified algebraic operation (project_sum) rather than a sequence of independent operations yields dramatic asymptotic savings by eliminating the batch factor $b$ inside the rank-scaling term.
  • Drop-in Neural Network Layers: TT operations wrap transparently into TensorFlow, allowing users to swap dense linear layers with compressed TT-matrix multiplication (t3f.matmul(tt_matrix, x)).

Key Results

Benchmarked on an NVIDIA DGX-1 (single Tesla V100 GPU, double precision) against the established CPU library ttpy:

Operation TTPY (1 obj, CPU) T3F (1 obj, CPU) T3F (1 obj, GPU) T3F (100 objs, CPU)* T3F (100 objs, GPU)*
matvec 11.142 ms 1.190 ms 0.744 ms 1.885 ms 0.140 ms
matmul 86.191 ms 9.849 ms 0.950 ms 17.483 ms 1.461 ms
norm 3.790 ms 2.136 ms 1.019 ms 0.253 ms 0.044 ms
gram 0.145 ms 0.606 ms 0.973 ms 0.021 ms 0.001 ms
project 116.868 ms 3.001 ms 13.239 ms 1.645 ms 0.226 ms

*Reported per single object within the 100-object batch.

Batch GPU execution yields massive per-object speedups: ~80× for matrix-by-vector products, ~59× for matrix multiplication, and over 500× for tangent projections compared to baseline CPU operations in ttpy. The library ships with 94% test coverage.

Limitations and Future Work

  • Missing Specialized Physics Algorithms: T3F does not support specialized tensor algorithms such as DMRG (Density Matrix Renormalization Group) or AMen (Alternating Minimal Energy), which are available in domain-specific physics and numerical toolboxes.
  • Single-Framework Lock-in: The library is tightly coupled to TensorFlow and does not provide native multi-backend support (e.g., PyTorch or JAX).

Summary updated

Abstract

Tensor Train decomposition is used across many branches of machine learning. We present T3F—a library for Tensor Train decomposition based on TensorFlow. T3F supports GPU execution, batch processing, automatic differentiation, and versatile functionality for the Riemannian optimization framework, which takes into account the underlying manifold structure to construct efficient optimization methods. The library makes it easier to implement machine learning papers that rely on the Tensor Train decomposition. T3F includes documentation, examples and 94% test coverage.
BibTeX
@article{novikov2020tensor,
  author = {Alexander Novikov and Pavel Izmailov and Valentin Khrulkov and Michael Figurnov and Ivan V. Oseledets},
  title = {{Tensor Train Decomposition on TensorFlow (T3F)}},
  journal = {J. Mach. Learn. Res.},
  year = {2020},
  url = {https://openreview.net/forum?id=NuuD64ilSvj},
}

Loading related content...

Figure