Tensor Train Low-rank Approximation (TT-LoRA): Democratizing AI with Accelerated LLMs

Afia Anjum, Maksim Ekin Eren, Ismael Boureima, Boian S. Alexandrov, Manish Bhattarai

Abstract

In recent years, Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide range of natural language processing (NLP) tasks. However, the ever-growing complexity of LLMs demands immense computational resources, hindering the broader research and application of these models. To address this, various parameter-efficient fine-tuning (PEFT) strategies, such as Low-Rank Approximation (LoRA) and Adapters, have been developed. Despite their potential, these methods often face limitations in compressibility. Specifically, LoRA struggles to scale effectively with the increasing number of trainable parameters in modern large-scale LLMs. This paper introduces Tensor Train Low-Rank Approximation (TT-LoRA), a novel PEFT approach that extends previous tensor-based methods with optimized tensor train (TT) decomposition integration. By eliminating Adapters and traditional LoRA-based structures, TT-LoRA achieves greater model compression without compromising downstream task performance, along with reduced inference latency and computational overhead. Our results demonstrate significant compression of LLMs while maintaining comparable performance to larger models, facilitating their deployment on resource-constrained platforms.

Introduction

Fine-tuning LLMs like LLaMA-3 or DeBERTa is the standard approach for specializing models for domain-specific tasks. However, "Full Fine-Tuning"—updating every single parameter—is prohibitively expensive in terms of GPU memory and storage. Existing solutions like LoRA (Low-Rank Adaptation) mitigate this by training small low-rank matrices. Yet, as models scale to hundreds of billions of parameters, even LoRA’s trainable components become substantial. Other methods, such as Adapters, introduce additional layers that increase inference latency. TT-LoRA addresses these challenges by achieving extreme compression without the "latency tax" associated with sequential adapter layers.

Methodology

The core innovation of TT-LoRA is the application of Tensor Train (TT) decomposition directly to the weight update matrix ($\Delta W$). Unlike traditional methods that use 2D matrix factorization, TT-LoRA treats weight updates as multi-dimensional objects.

The architectural integration of Tensor Train decomposition into the transformer encoder layer, showing the decomposition of weight updates into TT-format cores.

Tensor Train Decomposition

A large weight matrix is reshaped into a high-dimensional tensor and subsequently broken down into a sequence of interconnected 3D tensor "cores." This direct parameterization allows the model to represent complex weight updates using a fraction of the parameters required by LoRA. During training, the original pre-trained weights ($W_0$) remain frozen, and only these tiny TT cores are updated. To optimize this process, the researchers employed automated hyperparameter optimization (Ray Tune and HyperBand) to determine the ideal tensor shapes and ranks for specific model architectures.

Experimental Setup

TT-LoRA was evaluated across several standard benchmarks to test its versatility:

  • GLUE: Used for BERT-family models (DeBERTa-base, RoBERTa-base) on tasks like sentiment analysis (SST-2).
  • SuperGLUE: A more rigorous benchmark for larger models like LLaMA-2-7B and LLaMA-3-8B, featuring complex reasoning tasks.

Results and Analysis

TT-LoRA demonstrated superior performance-to-parameter efficiency across all tested models. For LLaMA-3-8B, TT-LoRA utilized only 0.2M trainable parameters compared to 3.41M for LoRA—a 170.5x reduction—while actually improving average accuracy from 73.96% to 79.55%.

Performance Percentage vs. Number of Trainable Parameters across various PEFT methods, highlighting TT-LoRA's superior efficiency.

Storage efficiency was equally dramatic. On DeBERTa, TT-LoRA required only 39 KB of storage for its trainable parameters, whereas standard LoRA required approximately 600 KB and Full Fine-Tuning required 278 MB. On the LLaMA-2-7B SuperGLUE benchmark, TT-LoRA achieved an average score of 80.19%, significantly outperforming both Full Fine-Tuning (75.03%) and LoRA (74.04%).

Discussion

The results suggest a "smaller is better" paradox: in several instances, TT-LoRA outperformed larger fine-tuning methods. This indicates that the extreme low-rank constraint of Tensor Trains may serve as a powerful regularizer, preventing the model from overfitting to small downstream datasets. By reducing storage requirements from megabytes to kilobytes, TT-LoRA enables the deployment of thousands of task-specific "expert" models on a single mobile device or edge server, effectively democratizing access to high-performance AI.

Conclusion and Future Work

TT-LoRA represents a significant step forward in parameter-efficient fine-tuning, offering massive compression ratios (up to 7000x over full fine-tuning) without sacrificing performance or increasing latency. Future work will focus on automating the selection of tensor shapes to reduce hyperparameter search costs and scaling the method to ultra-large models like LLaMA-3.1-405B.

Summary updated

Abstract

In recent years, Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide range of natural language processing (NLP) tasks, such as question-answering, sentiment analysis, text summarization, and machine translation. However, the ever-growing complexity of LLMs demands immense computational resources, hindering the broader research and application of these models. To address this, various parameter-efficient fine-tuning strategies, such as Low-Rank Approximation (LoRA) and Adapters, have been developed. Despite their potential, these methods often face limitations in compressibility. Specifically, LoRA struggles to scale effectively with the increasing number of trainable parameters in modern large scale LLMs. Additionally, Low-Rank Economic Tensor-Train Adaptation (LoRETTA), which utilizes tensor train decomposition, has not yet achieved the level of compression necessary for fine-tuning very large scale models with limited resources. This paper introduces Tensor Train Low-Rank Approximation (TT-LoRA), a novel parameter-efficient fine-tuning (PEFT) approach that extends LoRETTA with optimized tensor train (TT) decomposition integration. By eliminating Adapters and traditional LoRA-based structures, TT-LoRA achieves greater model compression without compromising downstream task performance, along with reduced inference latency and computational overhead. We conduct an exhaustive parameter search to establish benchmarks that highlight the trade-off between model compression and performance. Our results demonstrate significant compression of LLMs while maintaining comparable performance to larger models, facilitating their deployment on resource-constraint platforms.
BibTeX
@inproceedings{anjum2024tensor,
  author = {Afia Anjum and Maksim Ekin Eren and Ismael Boureima and Boian S. Alexandrov and Manish Bhattarai},
  title = {{Tensor Train Low-rank Approximation (TT-LoRA): Democratizing AI with Accelerated LLMs}},
  booktitle = {ICMLA},
  year = {2024},
  doi = {10.1109/icmla61862.2024.00085},
}
Figure