PoLAR: Polar-Decomposed Low-Rank Adapter Representation

Kai Lion, Liang Zhang, Bingcong Li, Niao He

Introduction

Low-Rank Adaptation (LoRA) has become the standard for fine-tuning Large Language Models (LLMs) due to its parameter efficiency. However, recent research reveals a significant inefficiency: rank underutilization. Even when a high rank is allocated to a LoRA adapter, the actual "stable rank" of the learned updates often collapses to near 1.0. This phenomenon, termed diversity collapse (DC), means that updates for different neurons align along a single direction, leaving much of the model's potential expressiveness unused.

The Diversity Collapse Problem

In standard LoRA, the weight update is parameterized as $\Delta W = A \times B$. While this is mathematically low-rank, the unconstrained nature of $A$ and $B$ during gradient descent often leads to a solution where the columns of the update matrix are highly correlated.

Visualizing Diversity Collapse: (a) conceptual vector alignment, (b) empirical evidence in LoRA, and (c) PoLAR's solution.

As shown in the figure above, LoRA's updates exhibit a blocky structure in their correlation heatmaps, indicating that the model is effectively learning only a single direction despite having the capacity for more. PoLAR (Polar-Decomposed Low-Rank Adapter Representation) is designed to solve this by explicitly decoupling the direction and magnitude of updates.

The PoLAR Method

PoLAR rethinks the matrix factorization of the update $\Delta W$ using polar decomposition:

$$\Delta W = X \Theta Y^\top$$

In this formulation:

  • $X$ and $Y$ are "direction matrices" constrained to the Stiefel manifold, meaning their columns are orthonormal. This forces the model to learn $r$ distinct, orthogonal directions, preventing collapse.
  • $\Theta$ is an unconstrained $r \times r$ "scale matrix" that captures the magnitude and interactions within the low-rank subspace.

Riemannian Optimization via "Landing"

Training on the Stiefel manifold typically requires expensive operations like Singular Value Decomposition (SVD). To maintain computational efficiency, PoLAR utilizes a landing algorithm. This approach uses an infeasibility penalty to pull matrices toward orthogonality during standard backpropagation. This "co-design" allows PoLAR to benefit from Riemannian geometry while remaining highly compatible with modern GPU parallelism.

Results and Discussion

PoLAR was evaluated across a wide range of benchmarks, including commonsense reasoning (BoolQ, ARC), mathematics (GSM8K), and general language understanding (GLUE), using models like Llama-2-7B and Gemma-3.

Stable Rank and Scaling

The most significant empirical finding is the restoration of stable rank. While LoRA and DoRA (Weight-Decomposed LoRA) show stable ranks that plateau near 1.0 regardless of the nominal rank provided, PoLAR's stable rank scales effectively with the allocated capacity.

Comparison of stable rank (||ΔW||²_F / ||ΔW||²_2) across different tasks for PoLAR, LoRA, and DoRA.

Performance and Efficiency

PoLAR consistently outperforms both LoRA and DoRA. For instance, on Llama-2-7B commonsense tasks, PoLAR (rank 16) achieved a mean accuracy of 74.00%, compared to LoRA's 72.76%. Furthermore, the landing method is significantly faster than traditional manifold optimization techniques (3x to 18x faster on an H100), with only a ~15% step-time overhead compared to vanilla LoRA. Because the matrices can be merged back into the base weights after training, PoLAR introduces zero overhead during inference.

Conclusion

PoLAR addresses a fundamental limitation in low-rank adaptation by ensuring that allocated rank translates into actual model expressiveness. By combining polar decomposition with efficient Riemannian optimization, it provides a robust, scalable alternative to LoRA that achieves faster theoretical convergence and superior downstream performance across diverse NLP tasks.

Summary updated

BibTeX
@inproceedings{lion2025polar,
  author = {Kai Lion and Liang Zhang and Bingcong Li and Niao He},
  title = {{PoLAR: Polar-Decomposed Low-Rank Adapter Representation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=jDxFD45kkc},
}
Figure