Reading Retraction-Free Optimizers at the Right Point

Many models need some weight matrices to stay orthogonal, meaning their columns are orthonormal. Examples include orthogonal attention and convolution layers, principal component analysis, and orthogonal alignment problems. The classical way to enforce this is a retraction: after each gradient step, a matrix factorization such as QR or SVD snaps the weights back onto the set of orthogonal matrices. That step is slow on GPUs. A family of "landing" optimizers skips it. They let the weights drift slightly off the constraint and add a pull term that brings them back over time, using only matrix multiplications. The speedups are real and are now widely reported. They are also measured on the drifting weights themselves. The same holds for the cheap evaluations that practitioners use to choose these optimizers' settings: short pilot runs, early stopping, and successive-halving searches. All of them score a model that does not yet satisfy the constraint.

This note proposes to test whether that measurement choice distorts conclusions, and to fix it cheaply. The fix is to score every checkpoint at its nearest orthogonal matrix, obtained with one SVD at evaluation time only. The hypothesis is that raw scores of drifting iterates are biased in a predictable direction, and that the bias is largest early in training, which is exactly when pilot studies are read. If so, short-horizon hyperparameter screens for landing methods will systematically favor settings that drift more, and some published speedup margins will shrink when time-to-target is measured on feasible models. If the hypothesis fails, the field gains a certificate that its cheap evaluations can be trusted.

Core Claim

Let $X$ be an $n \times p$ iterate, $P(X)$ its polar projection onto the orthogonal set (the closest matrix with orthonormal columns), and $d(X) = \|X - P(X)\|_F$ the distance between them. If the objective $f$ has $L$-Lipschitz gradients, a Taylor expansion around $P(X)$ gives

$$ \big| f(X) - f(P(X)) \big| \;\le\; \|\nabla f(P(X))\|_F \, d(X) + \tfrac{L}{2}\, d(X)^2 . $$

Near the manifold, $d(X)$ is roughly $\tfrac12\|X^\top X - I\|_F$, the constraint violation that landing papers already log. The first-order term does not have a random sign. The landing update moves partly along the ambient descent direction, so drifting iterates tend to sit where the unconstrained loss is lower than the constrained loss at the nearest feasible point. Raw scores are therefore optimistic, and more optimistic for configurations that drift more. In landing methods those are the ones with a weak pull strength $\lambda$ or a large step size. The testable prediction is that pilot rankings computed on raw iterates will prefer smaller $\lambda$ and larger steps than the full-length optimum does, and that scoring at $P(X)$ removes most of this preference.

There is a clean negative control. In LoRA fine-tuning, the adapter update is $W_0 + BA$, where $W_0$ is the frozen pretrained weight and $B$ is the matrix constrained to be orthonormal. Writing $B = QR$ and replacing $(B, A)$ by $(Q, RA)$ leaves the product unchanged. Projection therefore does not change the adapter's function, and the prediction is zero evaluation bias. Any ranking distortion found there must come from training dynamics rather than from where the model is scored. This separates the two effects.

Prior Work and Positioning

The landing line of work motivates the problem. Fast and accurate optimization on the orthogonal manifold without retraction introduced the landing update and reports an "8x speedup on GPUs" over Cayley and exponential maps, plus CIFAR-10 target accuracy reached "roughly 50% faster than retraction-based SGD." It also notes that the pull weight $\lambda$ must be calibrated against the gradient size, which is exactly the setting a pilot screen would choose. Local Linear Convergence of Infeasible Optimization with Orthogonal Constraints proves local linear convergence without per-step feasibility and reports landing "approximately twice as fast in terms of CPU time" than Riemannian gradient descent on synthetic PCA. Geometric design of the tangent term in landing algorithms for orthogonality constraints and Optimization without Retraction on the Random Generalized Stiefel Manifold widen the family of such methods. Distributed Retraction-Free and Communication-Efficient Optimization on the Stiefel Manifold adds gradient compression. All of these establish guarantees at convergence. None asks how their intermediate iterates should be scored when those scores drive decisions. That question is what this note addresses. It does not propose a new optimizer.

Two papers take opposite positions on feasibility and together supply the LoRA control. Riemannian Optimization for LoRA on the Stiefel Manifold enforces exact QR retractions and argues that this keeps $B$ at full effective rank. Retraction-Free Optimization over the Stiefel Manifold for the LoRA Fine-Tuning uses landing with a fixed penalty $\mu = 1/3$. It reports reaching vanilla LoRA's training loss "in roughly half the number of epochs," and MATH accuracy of 23.8% versus 19.3% for AdamW LoRA on LLaMA-3-8B. Because the LoRA function is invariant under the $(Q, RA)$ swap, we predict their comparison is unaffected by where checkpoints are scored. Confirming that prediction is part of the test.

Structure-preserving and submanifold methods act as feasible-by-construction references. Momentum Stiefel Optimizer, with Applications to Suitably-Orthogonal Attention, and Optimal Transport keeps iterates exactly orthogonal through a splitting scheme and reports CIFAR-10 vision-transformer test error of 8.32%, versus 9.75% for SGD and 12.90% for a Cayley optimizer. Efficient Optimization with Orthogonality Constraint: a Randomized Riemannian Submanifold Method (RSDM) cuts retraction cost by updating random low-dimensional pieces. Its iterates are always feasible, so its raw scores need no correction. That makes it an unbiased comparison point for landing methods.

The pilot-methodology literature supplies the lens. Lithography Solvent Discovery using Neuro-Symbolic Search Agent shows that imperfect proxy scores can invert rankings when candidates are later tested for real. Correcting Human Labels for Rater Effects in AI Evaluation: An Item Response Theory Approach corrects a systematic, modelable bias in cheap evaluations before they feed leaderboards. We borrow two ideas from these papers. The first is to treat the cheap evaluation as a biased instrument. The second is to correct it with a model of the bias rather than by adding samples. Here the bias model is the inequality above.

Research Program

The work builds a small evaluation wrapper for landing optimizers in PyTorch. At each evaluation checkpoint it computes three quantities. The first is the raw score $f(X)$. The second is the projected score $f(P(X))$: one thin SVD per constrained matrix, performed at evaluation only, with its cost reported separately from training cost. The third is a first-order corrected estimate that subtracts $\langle \nabla f, X - P(X)\rangle$ from the raw score using a single held-out batch. The corrected estimate lets us check whether a projection-free proxy suffices. For networks, the scores are validation loss and accuracy with every orthogonal layer replaced by its projection. For LoRA, the wrapper applies the function-preserving $(Q, RA)$ swap and, separately, a naive projection of $B$ alone. The naive version shows what goes wrong when the compensation into $A$ is omitted.

With this wrapper we run two studies. The first re-measures speed claims as time-to-target on feasible models. Time is counted until $f(P(X))$ reaches a target, instead of until $f(X)$ does. The second is a screening-fidelity study. We define a grid over landing step size, pull weight $\lambda$ and, for networks, learning-rate schedule. Each configuration is run to full budget over several seeds to obtain ground-truth final performance at the projected point. Short-horizon rankings at 2%, 5%, 10% and 25% of the budget are then compared against that truth under the raw, projected and corrected proxies. We record which configurations each proxy wrongly eliminates (false negatives) and which it wrongly promotes. We then fit the observed bias against $d(X)\,\|\nabla f\|$ to test whether the inequality predicts its size, not just its sign.

Baselines

The naive reference is the current practice of scoring raw landing iterates. We implement it by reading the same checkpoints as the other proxies, so data, seeds and compute are identical by construction. The only difference between proxies is the evaluation function applied to a shared set of training runs. This removes any tuning-budget asymmetry.

The strongest existing competitor for the speed question is the fastest feasible method on each task. That is the Momentum Stiefel Optimizer from the konglk1203/VariationalStiefelOptimizer code on the orthogonal vision transformer, whose reported CIFAR-10 test error is 8.32% (Momentum Stiefel Optimizer, with Applications to Suitably-Orthogonal Attention, and Optimal Transport). It is RSDM on large-scale PCA and Procrustes. No exact final-error figure is reported on Lacuna for RSDM on these tasks. Its paper states only that RSDM reached high precision while landing was still early in optimization. Each method receives the same grid size for tuning and the same GPU, and wall-clock time includes all factorizations. For the orthogonal WRN-28-10 on CIFAR-100, no reported accuracy was found on Lacuna. Local Linear Convergence of Infeasible Optimization with Orthogonal Constraints reports only that its landing variants matched or exceeded Cayley methods. For this study, "beating" the baseline does not mean a better optimizer. It means showing that a published landing speedup shrinks by more than seed-to-seed variation once measured on feasible models, or else confirming it holds within that variation.

For the screening question, the strongest competitor is successive halving on raw scores, the standard multi-fidelity search. The proposal counts as better if projected or corrected scoring raises Kendall's $\tau$ against the full-budget ranking at 5% budget by more than the spread across seeds. It also counts as better if it recovers the full-budget best configuration in more seeds at equal total compute.

The key ablation compares the projected proxy with the first-order corrected proxy, and in LoRA compares the function-preserving swap with naive projection of $B$. This isolates whether the gain comes from scoring a feasible point at all, or specifically from scoring a function-equivalent one.

Why not just…?

Why not just report constraint violation next to the loss, as landing papers already do? Two numbers cannot rank candidates. A screen must reduce each candidate to one score, and practitioners reduce to the loss. The inequality says how violation converts into loss bias, and the screening study measures whether that conversion changes decisions. If rankings by raw loss and by projected loss agree whenever violation is small, reporting both is enough, and we will say so.

Why not just run pilots longer or with more seeds? More seeds reduce variance but not a systematic bias. Longer pilots reduce bias only by approaching full cost, which defeats the purpose of a pilot. We test this directly by tracking raw-proxy $\tau$ as a function of budget. If raw $\tau$ reaches projected $\tau$ by 10% of budget, the correction matters only for very aggressive screens. That would be a real and reportable limitation.

Why not just screen with a feasible optimizer such as RSDM or QR retraction and switch to landing afterward? The settings being chosen, the pull weight and the landing step size, do not exist in those optimizers. Hyperparameters tuned under one optimizer also do not transfer cleanly to another. This alternative is still the right comparison for the speed study, and it is included there.

Why not just add a soft penalty $\gamma\|X^\top X - I\|_F^2$ and use ordinary AdamW? This changes the training method and leaves the measurement question untouched. The penalty-trained model also drifts and needs the same scoring decision. The landing papers (Fast and accurate optimization on the orthogonal manifold without retraction) argue that such penalties only land when $\gamma$ is large, and large $\gamma$ is poorly conditioned. We include soft-penalty runs in the grid as one more drifting method to score.

Decisive Experiments

The first check uses synthetic PCA and orthogonal Procrustes, where the constrained optimum is known exactly. Settings match those reported: $d = 500$, $r = 20$ for PCA, and $n = 3000$, $p = 2500$ for the large regime. We measure the raw-minus-projected gap along landing trajectories and test whether it is positive and scales with $d(X)\|\nabla f\|$. A gap that is not predominantly positive refutes the directional claim.

The second check is the screening-fidelity study on the orthogonal vision transformer on CIFAR-10 and the orthogonal WRN-28-10 on CIFAR-100. The measurements are Kendall's $\tau$, top-1 recovery and the false-elimination rate at each pilot budget, under all three proxies. The decisive signature is twofold: raw screens prefer smaller $\lambda$ than the full-budget optimum, and projected screens do not.

The third check re-measures time-to-target at feasible checkpoints for landing against the Momentum Stiefel Optimizer and a Cayley baseline on CIFAR-10. It also checks against RSDM and Riemannian gradient descent on PCA and Procrustes, to see whether the reported 8x, 2x and 50% margins survive.

The fourth check is the LoRA control. We fine-tune LLaMA-3.2-1B on GLUE, where the reported Manifold-LoRA average is 88.63% (Oblique variant), and on GSM8K and MATH. The predictions are that the function-preserving proxy shows no evaluation bias, that naive projection of $B$ does show one, and that any remaining ranking distortion is due to dynamics. If function-preserving projection still changes rankings in LoRA, the evaluation-bias account is incomplete, and the remaining distortion must be attributed to training dynamics rather than scoring.