Overcoming Factorization Redundancy in Low Rank Optimization

25 concepts23 papers1 proposal

When we optimize a high-dimensional matrix under a low-rank constraint, our standard engineering instinct is to factorize it. Instead of optimizing a massive matrix $W \in \mathbb{R}^{n \times m}$ directly, we decompose it into the product of two much smaller rectangular matrices, $W = AB$, where $A \in \mathbb{R}^{n \times r}$ and $B \in \mathbb{R}^{r \times m}$ for a small rank $r$. This simple trick underpins everything from classical matrix completion to modern parameter-efficient fine-tuning (PEFT) methods like Low-Rank Adaptation (LoRA).

However, this factorization introduces a fundamental geometric flaw: the decomposition is highly non-unique.

If we insert any invertible $r \times r$ transformation matrix $R$ between the factors, the underlying product remains completely unchanged:

$$W = A B = (AR)(R^{-1}B)$$

This simple algebraic identity creates an infinite continuous family of factor pairs $(A, B)$ that represent the exact same physical weight matrix or transformation. In optimization terms, the search space contains entire "fibers" or directions of complete redundancy. Standard Euclidean optimizers (like SGD or Adam) are blind to this geometry, treating $A$ and $B$ as independent entities. This mismatch introduces severe optimization bottlenecks, numerical instabilities, and training inefficiencies.

Why Factorization Redundancy Breaks Optimizers

To understand why this redundancy harms optimization, consider how standard gradient descent operates in a flat Euclidean space.

1. Transformation Variance and Ill-Conditioning

Because standard optimizers update $A$ and $B$ independently, they are sensitive to arbitrary internal scaling and rotations—a phenomenon termed "transformation variance" by Jui-Nan Yen et al. in LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization. If $A$ happens to scale up while $B$ scales down, the product $W$ might remain stable, but the gradients with respect to $A$ and $B$ will become severely unbalanced. This ill-conditioning forces practitioners to perform tedious hyperparameter tuning, such as setting different learning rates for different factors.

2. Destabilized Momentum and Acceleration

Traditional momentum-based acceleration methods rely on a stable, consistent history of gradients to build velocity toward an optimum. However, as shown by Huan Li et al. in Provable accelerated gradient method for nonconvex low rank optimization, the presence of rotational invariance means the optimizer can constantly change its trajectory within the redundant factor space without making any actual progress in improving the product matrix $W$. This prevents classical acceleration from maintaining a stable path.

3. Geometric Traps and Divergence

Even worse, optimization trajectories can fall into geometric traps. As identified by Joe Kileel 0001 et al. in Finding stationary points on bounded-rank matrices: a geometric hurdle and a smooth remedy, factorization redundancy can cause the individual factors $A$ and $B$ to drift toward infinity in opposite directions even while their product $W$ remains perfectly bounded. These "apocalypses" create artificial non-smoothness and false stationary points where algorithms prematurely stall.

The Geometric Remedy: Quotient Manifolds

To fix this, we must alter how we view the search space. Instead of optimizing in the flat Euclidean space of individual factors, we can define a geometry where all equivalent factorizations are collapsed into a single point. This is the core idea behind Riemannian optimization on quotient manifolds.

Euclidean Space (A, B)              Quotient Manifold
   |                                      |
   |  (AR, R⁻¹B)  <- Redundant Fiber      |
   |     *                                |
   |    /                                 |
   |   *  (A, B)                          |   [W]  <- Combined into a
   |  /                                   |          single unique point
   | *                                    |

In this framework, the optimization space is treated as a Riemannian quotient manifold, where each point is an equivalence class $[W]$ containing all possible factor pairs that yield that matrix product.

Early foundational work by Gilles Meyer et al. in Linear Regression under Fixed-Rank Constraints: A Riemannian Approach demonstrated that by computing gradients along the horizontal space of this manifold, algorithms completely ignore the redundant directions. Similarly, Francis Bach et al. in Low-Rank Optimization on the Cone of Positive Semidefinite Matrices applied this to symmetric low-rank matrices factorized as $Y Y^T$, where the redundancy is purely rotational ($Y \to YQ$ for an orthogonal matrix $Q$).

By respecting this underlying geometry, second-order optimization methods can achieve superlinear or even quadratic convergence rates. For example, the algorithms developed by Nicolas Boumal et al. in RTRMC: A Riemannian trust-region method for low-rank matrix completion and Bamdev Mishra et al. in Low-rank optimization with trace norm penalty and R3MC: A Riemannian three-factor algorithm for low-rank matrix completion navigate large-scale matrix completion problems efficiently because they never waste computational effort moving along equivalent representations.

This geometric awareness extends naturally to higher-dimensional tensors. When recovering tensors using the Tucker decomposition, standard gradient descent is highly sensitive to non-unique core and factor matrix parameterizations. To resolve this, Tian Tong et al. developed ScaledGD in Scaling and Scalability: Provable Nonconvex Low-Rank Tensor Completion, which uses a tailored preconditioning step to make the gradient updates invariant to these internal symmetries, ensuring convergence rates independent of the tensor's condition number. This geometric perspective has also proven vital in practical engineering domains; for instance, Yuanming Shi et al. leveraged Riemannian pursuit in Low-Rank Matrix Completion for Topological Interference Management by Riemannian Pursuit to bypass factorization ambiguities when optimizing dense wireless network topologies.

Modern Applications in Deep Learning and LoRA

While quotient manifold optimization was originally popularized in classical numerical linear algebra, factorization redundancy has re-emerged as a major bottleneck in modern deep learning layers and parameter-efficient fine-tuning.

Rotational Invariance in Neural Layers

Neural network layers parameterized with low-rank constraints inherently suffer from rotational invariance. As noted by Anton Obukhov et al. in Spectral Tensor Train Parameterization of Deep Learning Layers, this redundancy complicates training stability, particularly in sensitive architectures like Generative Adversarial Networks (GANs). Similarly, in deep metric learning, Soumava Kumar Roy et al. showed in Geometry Aware Constrained Optimization Techniques for Deep Learning that standard Euclidean optimization struggles with rotationally invariant distance metrics, which can be resolved by introducing geometry-aware Riemannian optimizers.

The One-to-Many Mapping Problem in Weight Generation

The ambiguity of low-rank representations creates unique challenges when trying to predict or generate neural network weights. In LoRAGen: Structure-Aware Weight Space Learning for LoRA Generation, Hao Huang et al. highlighted that because a single weight update maps to infinitely many decomposed matrix pairs, generative models face an ill-posed one-to-many regression target. Learning a stable weight-generation space requires explicitly accounting for this non-uniqueness.

Fixing LoRA via Refactoring and Preconditioning

In the context of fine-tuning foundation models, several recent works have introduced explicit geometric corrections to fix LoRA's optimization path:

Bypassing Factorization Entirely

An alternative, radical solution to factorization redundancy is to eliminate the factorization itself while preserving the low-rank constraint.

Instead of updating $A$ and $B$ as separate matrices, Fengmiao Bian et al. introduced RAdaGrad and RAdamW in Finding Low-Rank Matrix Weights in DNNs via Riemannian Optimization: RAdaGrad and RAdamW. These optimizers maintain and update the weight matrix directly on the Riemannian manifold of fixed-rank matrices. By completely bypassing the $W=AB$ decomposition, they entirely circumvent the convergence bottlenecks and ambiguities of factorization redundancy, offering a clean, direct path for low-rank neural network optimization.

Key Foundational Work

Linear Regression under Fixed-Rank Constraints: A Riemannian Approach (2011) — contributed 1 concept elements to this direction.
Low-rank optimization with trace norm penalty (2011) — contributed 1 concept elements to this direction.
RTRMC: A Riemannian trust-region method for low-rank matrix completion (2011) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_factorization_redundancy_15bf9398,
  title = {factorization redundancy},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/factorization-redundancy/rdr_15bf939812054420ab05187dbb5ee3c7},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_factorization_redundancy_15bf9398_v1,
  title = {factorization redundancy},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/factorization-redundancy/rdr_15bf939812054420ab05187dbb5ee3c7/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake