When Is Rank-1 Enough? Geometry-Guided Initialization for Parameter-Efficient Fine-Tuning

Haoran Zhao, Soyeon Caren Han, Eduard Hovy

Abstract

Parameter-efficient fine-tuning (PEFT) is a standard way to adapt multimodal large language models, yet extremely low-rank settings -- especially rank-1 LoRA -- are often unstable. We show that this instability is not solely due to limited capacity: in the rank-1 regime, optimization is highly sensitive to the update direction. Concretely, pretrained vision and text features form mismatched anisotropic regions, yielding a dominant "gap" direction that acts like a translation component and disproportionately steers early gradients under rank-1 constraints. Analyzing pretrained representations, we identify a modality-gap axis that dominates early gradient flow, while a random rank-1 initialization is unlikely to align with it, leading to weak gradients and training collapse. We propose Gap-Init, a geometry-aware initialization that aligns the rank-1 LoRA direction with an estimated modality-gap vector from a small calibration set, while keeping the initial LoRA update zero. Across multiple vision-language tasks and backbones, Gap-Init consistently stabilizes rank-1 training and can match or outperform strong rank-8 baselines. Our results suggest that at the extreme low-rank limit, initial alignment can matter as much as rank itself.

Introduction

Adapting Multimodal Large Language Models (MLLMs) to specific downstream tasks is a computationally intensive endeavor. Parameter-Efficient Fine-Tuning (PEFT) methods, such as LoRA (Low-Rank Adaptation), mitigate this by updating only a small subset of weights. However, a persistent challenge remains: when pushing the limits of efficiency with rank-1 adapters—the smallest possible update—models often suffer from instability or total training collapse.

Conventional wisdom suggests that rank-1 lacks the "expressive capacity" required for complex multimodal tasks. This research challenges that view, proposing that the failure is primarily geometric. In high-dimensional spaces, a randomly initialized rank-1 vector is almost certain to be orthogonal to the critical direction needed to bridge the gap between vision and text representations. This "orthogonality catastrophe" suppresses initial gradients, causing optimization to fail before learning begins.

The Geometry of Modality Gaps

Pretrained vision and text features do not occupy the embedding space uniformly; instead, they cluster into narrow, anisotropic "cones." The mismatch between these modalities is dominated by a simple translation or mean shift, known as the modality gap.

Geometric alignment intuition: Random rank-1 initialization often results in an orthogonal gradient that stalls training, whereas Gap-Init aligns the update with the modality gap vector.

As illustrated above, a random initialization ($A_{rand}$) is likely to be orthogonal to the modality gap vector ($\vec{g}$), leading to near-zero gradients and a "stuck" model. Gap-Init ensures the update direction ($B_{gap}$) is parallel to the modality gap, enabling optimal gradient flow.

Methodology: Gap-Init

The authors propose Gap-Init, a geometry-aware initialization strategy that requires no additional training and minimal data. The process involves:

  1. Modality Gap Estimation: Using a small calibration set (typically 256 image-text pairs), the average difference between visual and textual hidden states is calculated at each layer to identify the "modality gap" vector.
  2. Geometric Alignment: The first column of the LoRA $B$ matrix is initialized to align with this estimated gap vector.
  3. Zero-Update Start: The $A$ matrix is initialized to zero, ensuring the model starts from its pretrained state while immediately steering gradients toward closing the modality gap.
  4. SPOT-Gap: A robust scoring metric (Semantic Preserving Optimal Transport–Gap) is introduced to identify the layers with the most significant misalignment, optimizing layer selection for adaptation.

Evaluation and Results

The researchers evaluated Gap-Init across standard multimodal benchmarks, including COCO Captioning, VQAv2, OK-VQA, and Flickr30k, using backbones like BLIP-2, Qwen2-VL, and Gemma3.

Performance comparison: Gap-Init (r=1) stabilizes training and outperforms random rank-1 and rank-8 baselines on COCO captioning and VQA.

Key results include:

  • Superior Performance: On COCO Captioning, Rank-1 Gap-Init achieved a 140.59 CIDEr score, outperforming standard Rank-8 LoRA (138.49) and matching full fine-tuning (140.4).
  • Stabilizing VQA: In VQAv2, where standard rank-1 LoRA often collapses (15.58% accuracy), Gap-Init achieved 57.23% accuracy.
  • Efficiency and Reliability: Gap-Init drastically reduces training variance (up to 5x) and enables rank-1 models to outperform rank-8 baselines while using 8x fewer trainable parameters.
  • Hallucination Reduction: The method achieved higher POPE F1 scores (92.4), indicating more faithful model outputs compared to other PEFT techniques.

Conclusion and Future Work

Gap-Init demonstrates that at the extreme low-rank limit, the initial alignment of the update direction is as critical as the rank itself. By leveraging the global geometric features of the modality gap, researchers can achieve high performance with unprecedented parameter efficiency.

Future work may explore the domain specificity of the modality gap vector, as estimates from natural images may not generalize perfectly to specialized fields like medical imaging. Additionally, the "activation-space" perspective introduced here could potentially be extended to other domains, such as model merging or cross-lingual adaptation.

Summary updated

Abstract

Parameter-efficient fine-tuning (PEFT) is a standard way to adapt multimodal large language models, yet extremely low-rank settings -- especially rank-1 LoRA -- are often unstable. We show that this instability is not solely due to limited capacity: in the rank-1 regime, optimization is highly sensitive to the update direction. Concretely, pretrained vision and text features form mismatched anisotropic regions, yielding a dominant "gap" direction that acts like a translation component and disproportionately steers early gradients under rank-1 constraints. Analyzing pretrained representations, we identify a modality-gap axis that dominates early gradient flow, while a random rank-1 initialization is unlikely to align with it, leading to weak gradients and training collapse. We propose Gap-Init, a geometry-aware initialization that aligns the rank-1 LoRA direction with an estimated modality-gap vector from a small calibration set, while keeping the initial LoRA update zero. Across multiple vision-language tasks and backbones, Gap-Init consistently stabilizes rank-1 training and can match or outperform strong rank-8 baselines. Our results suggest that at the extreme low-rank limit, initial alignment can matter as much as rank itself.
BibTeX
@inproceedings{zhao2026when,
  author = {Haoran Zhao and Soyeon Caren Han and Eduard Hovy},
  title = {{When Is Rank-1 Enough? Geometry-Guided Initialization for Parameter-Efficient Fine-Tuning}},
  booktitle = {International Conference on Machine Learning},
  year = {2026},
  url = {https://openreview.net/forum?id=Umu6IsAUbS},
}
Figure