AdaRing: Towards Ultra-Light Vision-Language Adaptation via Cross-Layer Tensor Ring Decomposition
Abstract
Adapting large-scale Vision-Language Models (VLMs) like CLIP to downstream tasks typically involves parameter-efficient fine-tuning (PEFT) using adapters. However, existing methods often ignore cross-layer redundancy and struggle to balance task-specific discrimination with general knowledge preservation. We present AdaRing, a framework that utilizes Cross-Layer Tensor Ring Decomposition (TRD) to compress adapter weights by 90% while maintaining state-of-the-art performance. By decomposing stacked adapter weights into shared and layer-specific components, AdaRing effectively captures inter-layer correlations. Furthermore, a dual-path rank-driven adapter design ensures robust generalization to novel classes.
Introduction
The fine-tuning of massive VLMs is a significant bottleneck in deploying these models for specialized tasks. Standard PEFT methods, such as adapters, insert small trainable modules into a frozen backbone. However, these methods typically treat each layer's adapter independently, ignoring the high degree of feature similarity across adjacent layers. This "inter-layer" redundancy leads to inefficient parameter usage. Furthermore, existing adapters often struggle to balance the need for discriminative power on seen classes with the preservation of general knowledge for unseen classes.
Methodology: Cross-Layer Tensor Ring Decomposition
AdaRing addresses redundancy through a novel tensorization approach. Instead of independent weight matrices, adapter weights across all $L$ layers are stacked into a single high-order tensor $\mathcal{A} \in \mathbb{R}^{m \times L \times n}$.
As illustrated in Figure 1, this tensor is decomposed using Tensor Ring Decomposition (TRD). The decomposition yields a sequence of cores, categorized into layer-shared cores (capturing global network patterns) and layer-specific slices (handling local layer requirements). This mathematical formulation allows the model to represent complex cross-layer interactions with a fraction of the original parameters.
To further enhance performance, AdaRing employs a dual-path architecture consisting of a Fine-Grained Adapter (high-rank for task-specific features) and a Coarse-Grained Adapter (low-rank to preserve pre-trained knowledge). A learnable combinator dynamically merges these paths, while a generalization-aware regularization term ensures the fine-tuned embeddings do not drift too far from the original CLIP space.
Experimental Evaluation
AdaRing was evaluated on 11 diverse image classification datasets, including ImageNet, EuroSAT, and OxfordPets, using a "base-to-novel" transfer learning setup.
As shown in Figure 2, AdaRing demonstrates exceptional efficiency. While competing methods often experience a performance "collapse" when the parameter budget is severely restricted (e.g., $r=1$), AdaRing remains robust. On average, AdaRing reduces trainable parameters by 90% (from 0.67M to 0.06M) compared to leading methods like MMA, while achieving higher accuracy. Notably, on the EuroSAT satellite dataset, AdaRing improved base class accuracy by 12.21% and novel class accuracy by 3.53% over previous benchmarks.
Discussion and Conclusion
The success of AdaRing stems from the empirical observation that adjacent layers in VLMs are highly correlated, often exceeding 80% similarity. By treating the network as a holistic tensor rather than a collection of independent layers, AdaRing effectively eliminates redundant learning. The rank-driven dual-path design further solves the "jack of all trades" problem, allowing for both specialization and generalization. Future work will explore the scalability of this tensor-based approach to larger model architectures and more complex multi-modal tasks beyond classification.
Summary updated