Parameter Efficient Quasi-Orthogonal Fine-Tuning via Givens Rotation

Xinyu Ma, Xu Chu, Zhibang Yang, Yang Lin, Xin Gao, Junfeng Zhao

Abstract

With the increasingly powerful performances and enormous scales of pretrained models, promoting parameter efficiency in fine-tuning has become a crucial need for effective and efficient adaptation to various downstream tasks. One representative line of fine-tuning methods is Orthogonal Fine-tuning (OFT), which rigorously preserves the angular distances within the parameter space to preserve the pretrained knowledge. Despite the empirical effectiveness, OFT still suffers low parameter efficiency at $O(d^2)$ and limited capability of downstream adaptation. Inspired by Givens rotation, in this paper, we proposed quasi-Givens Orthogonal Fine-Tuning (qGOFT) to address the problems. We first use $O(d)$ Givens rotations to accomplish arbitrary orthogonal transformation in $SO(d)$ with provable equivalence, reducing parameter complexity from $O(d^2)$ to $O(d)$. Then we introduce flexible norm and relative angular adjustments under soft orthogonality regularization to enhance the adaptation capability of downstream semantic deviations. Extensive experiments on various tasks and pretrained models validate the effectiveness of our methods.

Introduction: The Shift to Orthogonal Adaptation

Fine-tuning massive pretrained models like LLaMA or ViT is computationally expensive. While standard Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA rely on low-rank additive updates, a newer paradigm called Orthogonal Fine-Tuning (OFT) has emerged. OFT applies a multiplicative orthogonal transformation to the model's weights, preserving the "angular distances" between neurons. This preservation is key to maintaining the model's pretrained "common sense" during task-specific adaptation.

Comparison between LoRA (additive) and OFT (multiplicative) fine-tuning paradigms.

However, traditional OFT faces two significant hurdles. First, representing a general orthogonal matrix requires $O(d^2)$ parameters, which becomes a massive overhead for large hidden dimensions $d$. Second, strict orthogonality is often too rigid; real-world downstream tasks frequently involve "semantic shifts" that require subtle adjustments to vector norms and relative angles.

Methodology: GOFT and qGOFT

To overcome these limitations, we introduce Givens Orthogonal Fine-Tuning (GOFT) and its flexible extension, qGOFT.

Givens Orthogonal Fine-Tuning (GOFT)

Instead of learning a dense $d \times d$ matrix, GOFT leverages Givens Rotations. A Givens rotation is a fundamental linear algebra operation that rotates a vector within a single 2D plane. We prove that any rotation in $d$-dimensional space can be decomposed into just $d-1$ specific Givens rotations. This mathematical insight allows us to reduce parameter complexity from $O(d^2)$ to $O(d)$.

To ensure computational efficiency, we designed a parallel rotation strategy. By using sparse matrix multiplications, we can execute these rotations in $O(\log d)$ steps rather than $O(d)$ sequential operations, significantly mitigating the training bottleneck.

Quasi-Givens OFT (qGOFT)

To address the "rigidity" of strict orthogonality, we evolved GOFT into qGOFT. Instead of a strict rotation matrix with fixed trigonometric constraints, qGOFT utilizes a flexible $2 \times 2$ matrix for each Givens pair. This allows the model to adjust vector norms and relative angles. To prevent excessive drift from the pretrained knowledge, we apply a soft orthogonality regularizer, encouraging the transformation to remain "mostly" orthogonal while providing the flexibility needed for semantic adaptation.

Overview of GOFT and qGOFT: Sequential vs. Parallel Givens rotations and the transition to soft quasi-Givens rotation.

Experimental Evaluation

We validated qGOFT across a diverse set of benchmarks:

  • NLP (GLUE & SQuAD): Using DeBERTaV3-base, qGOFT outperformed LoRA, AdaLoRA, and original OFT.
  • Instruction Tuning (LLaMA-2-7B): Fine-tuned on Alpaca-cleaned, GOFT utilized only 0.007% of the model's parameters to beat LoRA (0.25%) and OFT on the Vicuna-Eval benchmark.
  • Computer Vision (VTAB-1K): On 19 diverse tasks using ViT-B/16, qGOFT achieved an average accuracy of 79.4%, significantly surpassing Full Fine-tuning (68.9%) and LoRA (74.5%).

Key Insights

  1. Efficiency without Sacrifice: A chain of $d$ rotations is mathematically sufficient to reach any point on a hypersphere, proving that $O(d^2)$ parameters are not necessary for effective orthogonal adaptation.
  2. The Power of Soft Priors: Replacing hard mathematical constraints (like the Cayley transform) with soft regularization allows the model to capture subtle semantic shifts that strict OFT misses.
  3. Zero Inference Overhead: As a reparameterization method, the learned weights can be merged back into the original model, ensuring no additional latency during deployment.

Limitations and Future Outlook

While highly parameter-efficient, qGOFT currently experiences slower training times (approx. 3.3x–3.5x per batch) due to multiple sparse matrix multiplications. Future work will focus on optimizing these operations via Taylor expansion approximations or specialized CUDA kernels. Additionally, the strength of the orthogonality regularizer remains a task-dependent hyperparameter that requires careful tuning.

Summary updated

Abstract

With the increasingly powerful performances and enormous scales of pretrained models, promoting parameter efficiency in fine-tuning has become a crucial need for effective and efficient adaptation to various downstream tasks. One representative line of fine-tuning methods is Orthogonal Fine-tuning (OFT), which rigorously preserves the angular distances within the parameter space to preserve the pretrained knowledge. Despite the empirical effectiveness, OFT still suffers low parameter efficiency at $\mathcal{O}(d^2)$ and limited capability of downstream adaptation. Inspired by Givens rotation, in this paper, we proposed quasi-Givens Orthogonal Fine-Tuning (qGOFT) to address the problems. We first use $\mathcal{O}(d)$ Givens rotations to accomplish arbitrary orthogonal transformation in $SO(d)$ with provable equivalence, reducing parameter complexity from $\mathcal{O}(d^2)$ to $\mathcal{O}(d)$. Then we introduce flexible norm and relative angular adjustments under soft orthogonality regularization to enhance the adaptation capability of downstream semantic deviations. Extensive experiments on various tasks and pretrained models validate the effectiveness of our methods.
BibTeX
@inproceedings{ma2024parameter,
  author = {Xinyu Ma and Xu Chu and Zhibang Yang and Yang Lin and Xin Gao and Junfeng Zhao},
  title = {{Parameter Efficient Quasi-Orthogonal Fine-Tuning via Givens Rotation}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=1zFkjbTgwC},
}
Figure