TL;DR
ProLand is a novel, retraction-free optimizer for LoRA that constrains basis directions to the Stiefel manifold using a "Landing" approach. By coordinating factor updates in the product space and incorporating Muon-style spectral directions, it achieves state-of-the-art fine-tuning performance across various LLMs while avoiding the heavy computational cost of per-step manifold retractions.
What Problem Does This Paper Address?
Low-Rank Adaptation (LoRA) is the go-to method for parameter-efficient fine-tuning, but it has a subtle efficiency problem: basis redundancy. Standard AdamW fine-tuning often produces highly correlated columns in the low-rank matrices, meaning the model isn't making the most of its limited "rank budget."
To solve this, researchers have turned to manifold optimization, specifically constraining the basis factor to the Stiefel manifold (ensuring columns are orthonormal, i.e., $B^\top B = I$). However, traditional manifold methods require a "retraction" (like a QR decomposition) at every single step to stay on the manifold. This is not only computationally expensive but also restrictive—sometimes the best path to a solution involves temporarily stepping "off" the manifold to explore the surrounding space.
Methods
The authors introduce ProLand (Product-space Landing), which combines two key ideas:
- Landing Methods: Instead of forcing the model back to the manifold at every step, ProLand allows "temporarily infeasible" iterates. It uses an explicit "Landing correction" term that progressively restores orthogonality over time.
- Coordinated Product-Space Updates: ProLand doesn't just update the $A$ and $B$ matrices independently. It designs the joint update in the space of the represented weight increment ($\Delta W = BA$).
Crucially, they derive a specific condition for the updates: $B^\top D_B = 0$. This ensures that the optimization step doesn't disturb the orthogonality of $B$ to first order. This "separation of concerns" allows the model to learn the task and restore feasibility simultaneously without them interfering with each other.
Furthermore, ProLand incorporates Muon-style spectral directions. It computes the polar factor of the momentum in the product space, providing a more effective geometry for optimization than standard gradients. To keep this practical for massive models, the authors developed an exact low-rank implementation that avoids ever forming dense, weight-sized matrices.
Datasets
The authors evaluated ProLand across a wide array of models and benchmarks:
- Encoder Models: DeBERTa-v3-base evaluated on four GLUE tasks (RTE, MRPC, STS-B, CoLA).
- Decoder Models: Qwen2.5-3B, Llama-3.2-3B, and the Muon-pretrained Moonlight-16B-A3B.
- Language Benchmarks: Seven tasks including MMLU, PIQA, WinoGrande, OpenBookQA, HellaSwag, BoolQ, and ARC-Challenge.
- Code Generation: HumanEval and MBPP.
Key Insights and Ideas
- The Benefit of Being "Infeasible": A major "aha" moment is illustrated in Figure 1. By allowing the optimizer to take a path that temporarily violates the $B^\top B = I$ constraint, ProLand can reach a better stationary solution than methods that strictly enforce the constraint at every step.
- Decoupling Learning from Feasibility: The mathematical choice to enforce $B^\top D_B = 0$ is vital. It means the "learning" part of the update is Frobenius-orthogonal to the "feasibility" (Landing) part. This prevents the gradient descent from "fighting" the orthogonality restoration.
- Spectral Updates in LoRA: While spectral optimizers like Muon have excelled in pre-training, ProLand demonstrates they are equally powerful for fine-tuning when correctly adapted to the product-space geometry of LoRA.
Key Results
ProLand consistently outperformed established baselines like AdamW, LoRA-Muon, sMuon, and Riemannion:
- Universal Superiority: ProLand achieved the highest average score in every tested encoder and decoder model-rank configuration across 11 tasks.
- GLUE Performance: On DeBERTa-v3, ProLand reached a mean score of 84.46, surpassing the strongest baseline (sMuon) by 0.57 points.
- Decoder Gains: On the 3B models, ProLand led by margins of 0.46 to 1.10 percentage points over the best baselines.
- Effective Constraint Control: Ablation studies showed that the Landing correction is highly effective, reducing the final median constraint violation by 98.1% compared to a version without it.
Limitations and Future Work
- Initialization Sensitivity: The theoretical analysis assumes the model has already entered a "regular" phase after a finite initialization period. The behavior during the very first steps of zero-initialized LoRA is not fully characterized.
- One-Sided Constraint: Currently, ProLand only constrains the $B$ factor. Future work could explore two-sided Stiefel constraints or other manifold geometries.
- Hyperparameter Tuning: While ProLand uses a fixed Landing strength ($\rho = 0.05$) in most experiments, the optimal balance between optimization and feasibility restoration might vary for extremely long training runs or different architectures.
Summary updated