Abstract
Tensor Train decomposition is used across many branches of machine learning. We present T3F -- a library for Tensor Train decomposition based on TensorFlow. T3F supports GPU execution, batch processing, automatic differentiation, and versatile functionality for the Riemannian optimization framework, which takes into account the underlying manifold structure to construct efficient optimization methods. The library makes it easier to implement machine learning papers that rely on the Tensor Train decomposition. T3F includes documentation, examples and 94% test coverage.
Introduction
High-dimensional tensors are essential for representing complex data relationships in domains like recommender systems and deep neural networks. However, they suffer from the "curse of dimensionality," where storage and computational costs grow exponentially with the number of dimensions. Tensor Train (TT) decomposition mitigates this by factorizing massive tensors into a chain of smaller, manageable "cores."
While TT decomposition is theoretically powerful, previous implementations often lacked critical features for modern machine learning. T3F (Tensor Train on TensorFlow) was developed to bridge this gap, providing an open-source library that integrates TT methods into the TensorFlow ecosystem. It specifically addresses the need for GPU acceleration, automatic differentiation for neural network integration, efficient batch processing, and Riemannian optimization.
Methods
T3F is built directly on top of TensorFlow, supporting both graph-based and eager execution. It introduces two primary classes: TensorTrain and TensorTrainBatch to handle single objects and batches respectively.
Riemannian Geometry and Optimization
A core innovation of T3F is its treatment of tensors with a fixed TT-rank as a Riemannian manifold. This allows for Riemannian gradient descent, which often converges faster than standard gradient descent on individual cores by respecting the geometric structure of the low-rank space. T3F is the first library to automate the projection of gradients onto the manifold's tangent space, simplifying the implementation of complex optimization-based research.
Optimized Operations
To handle large-scale data, T3F includes a tailored project_sum operation. While naive batch summation and projection are computationally expensive, T3F leverages the linearity of the projection to reduce costs significantly. The library also maintains a standard API similar to tf.Tensor (supporting .op, .name, .get_shape()), allowing users to easily replace standard layers with TT-compressed alternatives.
Results
The performance of T3F was evaluated using synthetic benchmarks involving random TT-matrices (size $10^{10} \times 10^{10}$) and TT-vectors (size $10^{10} \times 1$). These massive dimensions are impossible to store in a dense format but are easily handled by T3F due to the TT-format's efficiency.
Performance Benchmarks
T3F was compared against TTPY, a popular CPU-based Python library for Tensor Train operations. The results, summarized in the table below, demonstrate significant speedups across various operations.
For a single matrix-vector multiplication (matvec), T3F on a CPU is approximately 10x faster than TTPY. The most dramatic improvements appear during batch processing. When handling 100 objects, T3F on a GPU achieved a per-object speedup of nearly 80x compared to TTPY (0.140 ms vs 11.142 ms). This efficiency highlights T3F's suitability for modern ML workflows that rely on large mini-batches and hardware acceleration.
Discussion and Conclusion
T3F makes the theoretical efficiency of TT-decomposition—reducing parameters from $n^d$ to $O(d n r^2)$—practically accessible for deep learning. By treating batching as a first-class citizen and leveraging Riemannian geometry, it provides a robust framework for high-dimensional optimization.
While T3F currently focuses on optimization-based methods and does not yet support some classical algorithms like DMRG or AMen, its integration with TensorFlow's ecosystem and high test coverage (94%) make it a valuable tool for the research community. Future work may include automated rank selection and broader algorithm support.
Summary updated