Introduction
Low-Rank Adaptation (LoRA) has become the standard for fine-tuning Large Language Models (LLMs) due to its parameter efficiency. However, recent research reveals a significant inefficiency: rank underutilization. Even when a high rank is allocated to a LoRA adapter, the actual "stable rank" of the learned updates often collapses to near 1.0. This phenomenon, termed diversity collapse (DC), means that updates for different neurons align along a single direction, leaving much of the model's potential expressiveness unused.
The Diversity Collapse Problem
In standard LoRA, the weight update is parameterized as $\Delta W = A \times B$. While this is mathematically low-rank, the unconstrained nature of $A$ and $B$ during gradient descent often leads to a solution where the columns of the update matrix are highly correlated.
As shown in the figure above, LoRA's updates exhibit a blocky structure in their correlation heatmaps, indicating that the model is effectively learning only a single direction despite having the capacity for more. PoLAR (Polar-Decomposed Low-Rank Adapter Representation) is designed to solve this by explicitly decoupling the direction and magnitude of updates.
The PoLAR Method
PoLAR rethinks the matrix factorization of the update $\Delta W$ using polar decomposition:
$$\Delta W = X \Theta Y^\top$$
In this formulation:
- $X$ and $Y$ are "direction matrices" constrained to the Stiefel manifold, meaning their columns are orthonormal. This forces the model to learn $r$ distinct, orthogonal directions, preventing collapse.
- $\Theta$ is an unconstrained $r \times r$ "scale matrix" that captures the magnitude and interactions within the low-rank subspace.
Riemannian Optimization via "Landing"
Training on the Stiefel manifold typically requires expensive operations like Singular Value Decomposition (SVD). To maintain computational efficiency, PoLAR utilizes a landing algorithm. This approach uses an infeasibility penalty to pull matrices toward orthogonality during standard backpropagation. This "co-design" allows PoLAR to benefit from Riemannian geometry while remaining highly compatible with modern GPU parallelism.
Results and Discussion
PoLAR was evaluated across a wide range of benchmarks, including commonsense reasoning (BoolQ, ARC), mathematics (GSM8K), and general language understanding (GLUE), using models like Llama-2-7B and Gemma-3.
Stable Rank and Scaling
The most significant empirical finding is the restoration of stable rank. While LoRA and DoRA (Weight-Decomposed LoRA) show stable ranks that plateau near 1.0 regardless of the nominal rank provided, PoLAR's stable rank scales effectively with the allocated capacity.
Performance and Efficiency
PoLAR consistently outperforms both LoRA and DoRA. For instance, on Llama-2-7B commonsense tasks, PoLAR (rank 16) achieved a mean accuracy of 74.00%, compared to LoRA's 72.76%. Furthermore, the landing method is significantly faster than traditional manifold optimization techniques (3x to 18x faster on an H100), with only a ~15% step-time overhead compared to vanilla LoRA. Because the matrices can be merged back into the base weights after training, PoLAR introduces zero overhead during inference.
Conclusion
PoLAR addresses a fundamental limitation in low-rank adaptation by ensuring that allocated rank translates into actual model expressiveness. By combining polar decomposition with efficient Riemannian optimization, it provides a robust, scalable alternative to LoRA that achieves faster theoretical convergence and superior downstream performance across diverse NLP tasks.
Summary updated