Adaptive Principal Components Allocation with the ℓ2,g-regularized Gaussian Graphical Model for Efficient Fine-Tuning Large Models

Jingjing Zheng, Yankai CaoarXiv 2024 · DOI · arXiv

Introduction

The rapid advancement of Large Language Models (LLMs) has led to a significant increase in the number of parameters, making full fine-tuning computationally prohibitive for most researchers and practitioners. As shown in Figure 1, the scale of models has grown exponentially, necessitating the development of Parameter-Efficient Fine-Tuning (PEFT) techniques.

The exponential growth of LLM parameters from BERT to PaLM underscores the necessity for efficient adaptation methods.

While methods like LoRA and AdaLoRA have made strides by utilizing low-rank approximations, they often rely on local parameter importance or fixed rank allocations. This paper introduces a novel approach that leverages Gaussian Graphical Models (GGMs) to capture global dependencies between principal components of weight matrices, enabling more precise and efficient parameter allocation.

Methodology: Adaptive Principal Components Allocation

The core of the proposed method is the identification of critical "nodes" within the model's architecture using a combination of Singular Value Decomposition (SVD) and graph-based modeling.

Node Definition and Importance Scoring

Rather than modeling individual weights, the authors apply SVD to weight matrices to extract their top $r$ principal components. These components, alongside layer biases, serve as nodes in a GGM. An importance score, derived from gradients and weight magnitudes, is assigned to each node to quantify its contribution to the model's performance on a specific downstream task.

Sparse GGM with ℓ2,g-Regularization

To identify the most significant interactions between these nodes, the authors construct a GGM and seek a sparse precision matrix. Sparsity is enforced through a specialized non-convex $\ell_{2,g}$-regularization. This approach allows for "node-level" selection, effectively freezing less critical principal components while maintaining the most impactful global dependencies.

Various surrogate functions g(x) evaluated for the ℓ2,g-regularization to achieve optimal structural sparsity.

The resulting non-convex optimization problem is solved using a Block Coordinate Descent (BCD) algorithm, ensuring efficient convergence to a sparse solution.

Experimental Results

The method was evaluated on the RoBERTa-Base model across several tasks from the GLUE benchmark, including SST-2, MRPC, CoLA, and STS-B.

Performance and Efficiency

The proposed approach, denoted as "Ours-2", demonstrated superior efficiency and performance:

  • Parameter Reduction: It required only 0.084M trainable parameters—a 72% reduction compared to the 0.3M parameters used by LoRA and AdaLoRA.
  • Benchmark Scores: Despite the smaller parameter footprint, it achieved an average GLUE score of 85.1, outperforming Full Fine-tuning (85.0), LoRA (84.9), and AdaLoRA (83.9).
  • Task-Specific Gains: Notable improvements were observed in the CoLA task (64.4 MCC) and MRPC (90.2 Accuracy).

Conclusion and Future Directions

This work represents the first application of Gaussian Graphical Models to the PEFT domain, shifting the focus from local weight magnitude to global structural dependencies. By utilizing $\ell_{2,g}$-regularization, the method achieves state-of-the-art performance with unprecedented parameter efficiency. Future research will focus on scaling this approach to billion-parameter models and exploring its applicability to generative and multi-modal tasks.

Summary updated

Abstract

In this work, we propose a novel Parameter-Efficient Fine-Tuning (PEFT) approach based on Gaussian Graphical Models (GGMs), marking the first application of GGMs to PEFT tasks, to the best of our knowledge. The proposed method utilizes the $\ell_{2,g}$-norm to effectively select critical parameters and capture global dependencies. The resulting non-convex optimization problem is efficiently solved using a Block Coordinate Descent (BCD) algorithm. Experimental results on the GLUE benchmark [24] for fine-tuning RoBERTa-Base [18] demonstrate the effectiveness of the proposed approach, achieving competitive performance with significantly fewer trainable parameters. The code for this work is available at: https://github.com/jzheng20/Course projects.git.
BibTeX
@misc{zheng2024adaptive,
  author = {Jingjing Zheng and Yankai Cao},
  title = {{Adaptive Principal Components Allocation with the ℓ2,g-regularized Gaussian Graphical Model for Efficient Fine-Tuning Large Models}},
  year = {2024},
  eprint = {2412.08592},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.08592},
}
Figure