Robot policies are usually trained at length in simulation and then fine-tuned on a small amount of real or realistic data. A practical problem shows up at that second stage. When a policy sees several past camera frames, not just the current one, fine-tuning can repair a perception error in two ways. It can learn to see the new domain correctly, or it can learn to lean harder on what it already did in earlier frames. The second option is cheap and looks fine on training clips, but it fails as soon as the real scene departs from the history. We propose that an optimizer quantity largely ignored in robotics decides which option wins: how fast each layer's weights can still rotate.
In networks with normalization layers, many weight matrices can be rescaled without changing the output. For those layers, a gradient step changes the function only by turning the weight vector, and the size of that turn shrinks as the weight's length grows. This is the "effective learning rate" (ELR). After long simulation training, layers can end up with very different weight lengths. Their ELRs then differ too, so at the start of fine-tuning some layers can barely move while others move freely. We hypothesize that in multi-frame policies the current-frame perception layers are often the ones left nearly frozen, so the adaptation budget flows into the temporal-fusion and action layers, which then learn the history shortcut. The proposal measures this per-layer rotation budget, tests whether it predicts history-shortcut behavior and forgetting, and reallocates it using weight rescaling. For these layers rescaling leaves the model's outputs exactly unchanged, so the reallocation forgets nothing at the moment it is applied.
Core Claim
For a weight vector $w$ in a layer whose output does not depend on $\|w\|$, define its angular update rate over $k$ optimizer steps as
$$ \omega_k(t) = \frac{1}{k}\arccos\frac{\langle w_t, w_{t+k}\rangle}{\|w_t\|\,\|w_{t+k}\|}, $$
which is the average angle the weight turns per step. Under plain gradient descent this rate scales like $\eta/\|w\|^2$, where $\eta$ is the nominal learning rate. That is the relation shown by Theoretical Analysis of Auto Rate-Tuning by Batch Normalization. Under Adam-type optimizers the per-coordinate step is roughly independent of gradient scale, so the rate scales closer to $\eta/\|w\|$. In both cases a layer's length sets its rotation budget.
Our claim has three parts. First, after simulation pretraining, multi-frame policies show a measurable imbalance in $\omega$ between current-frame perception layers and history or action layers. Second, during sim-to-real fine-tuning this imbalance predicts a rise in the policy's reliance on past frames over the current frame. It also predicts a loss of performance on the original domain. Third, rescaling layer lengths before fine-tuning so that perception layers get a larger share of rotation reduces both effects at equal compute. Rescaling changes no prediction at time zero.
Our honest prior is that the third part may turn out to be equivalent to a well-chosen per-layer learning rate. In that case the contribution is a diagnostic that picks the layers and ratios without a hyperparameter search, and we say so in the experiments.
Where This Sits
The mechanism comes from work on normalized networks. Neural Mechanics describes weight decay as an inward force that balances the outward growth caused by finite learning rates. Spherical Motion Dynamics and Rotational Equilibrium show that this push and pull settles into an equilibrium angular speed. Kosson et al. also argue that layers out of equilibrium learn at unbalanced rates. An Exponential Learning Rate Schedule for Deep Learning shows that weight decay and learning-rate schedules are interchangeable in such networks. That equivalence is why we treat rescaling and per-layer learning rates as rivals, not as different things.
These papers supply the method and the measurement. They study stationary supervised training, not a sudden domain switch at the start of fine-tuning.
The motivating evidence that collapsing ELR causes lost adaptability comes from reinforcement learning. Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalization fixes weight lengths with weight normalization and keeps learning stable up to an update-to-data ratio of 20. Hyperspherical Normalization for Scalable Deep Reinforcement Learning (SimbaV2) projects all weights and features onto a unit sphere. Both work only on low-dimensional state vectors, and both name vision as untested. Both also require building the constraint into the architecture from the start. That rules them out for already-pretrained vision-language-action policies (VLAs, robot policies built on a pretrained vision-language model that output motor actions). Our intervention is applied after pretraining and keeps the architecture unchanged.
The application regime comes from multi-frame policies and sim-to-real augmentation. CronusVLA adds a cache of past-frame features and a cross-frame action decoder to a single-frame OpenVLA backbone. It places a stop-gradient on cached history so the backbone does not become "lazy." That design choice is direct evidence that a history shortcut exists. However, the stop-gradient protects only the backbone; the cross-frame decoder can still learn to copy history. Our work asks whether the shortcut has an optimizer cause and treats it at that level.
Seeing Realism from Simulation restyles simulation videos into realistic ones with a depth-conditioned video diffusion model. It reports gains on LIBERO-Plus and Robotwin 2.0 but a "slight performance dip" on clean LIBERO. We read that dip as a possible symptom of uncontrolled adaptation, not of the data itself, and we test that reading. ReBot and AnyCamVLA constrain the scope. ReBot shows that unconstrained generative video flickers. AnyCamVLA avoids fine-tuning altogether because direct fine-tuning on new viewpoints caused forgetting. Our proposal is a bet that fine-tuning can be made to forget less, not a replacement for those data pipelines. LIBERO supplies the evaluation logic: it measures forward transfer and forgetting separately and finds that the two trade off.
Research Program
We work at two scales. The small scale is a ViT-based multi-frame diffusion policy trained from scratch on LIBERO simulation tasks. At this scale every intervention, including weight normalization from scratch, is affordable, and we can run many seeds. The large scale is a CronusVLA-style multi-frame policy on an OpenVLA backbone. Here we only fine-tune, which matches how practitioners use these models.
In both cases the source domain is clean simulation. The target domain is one of three shifts, in increasing realism. The first is LIBERO-Plus perturbations. The second is videos restyled with the pipeline of Seeing Realism from Simulation. The third is SimplerEnv's real-to-sim evaluation for the WidowX and Google Robot setups, with a small set of real AgileX or Franka trials if hardware time allows.
The first deliverable is a measurement study. We log $\omega$ and $\|w\|$ for every layer that is approximately scale-invariant: linear maps feeding a LayerNorm, and query and key projections. We also log the same angle for layers that are not scale-invariant, since the angle is defined for any weight.
We track behavior alongside the optimizer quantities. History reliance is the change in predicted action when the current frame is swapped for one from a different episode, divided by the change when the past frames are swapped instead. Source retention is success on clean LIBERO after fine-tuning. Target success is the usual task success rate. Representation change per layer is the CKA similarity between features before and after fine-tuning, a standard index of how much a layer's features change.
The second deliverable is the intervention. Before fine-tuning, each scale-invariant perception layer is rescaled by a factor $c_\ell$. We choose $c_\ell$ so that its predicted initial $\omega$ matches a target ratio relative to the temporal and action layers. The target ratio is set once on a held-out shift and then frozen. Because rescaling is output-preserving for these layers, the fine-tuning start point is functionally identical to the baseline's. Weight decay then pulls the norms back toward equilibrium, so the reallocation fades naturally.
We also test a sustained variant that rescales to the target norm every $N$ steps. This variant is close to Kosson et al.'s rotational optimizer applied selectively.
Baselines
The naive reference is full fine-tuning with AdamW, a cosine schedule, and the learning rate and weight decay that the base policy's own recipe uses. This is what CronusVLA and OpenVLA users do by default. Beating it means higher target success and higher source retention at the same number of steps.
The strongest augmentation-side competitor is the video-transfer method of Seeing Realism from Simulation. That paper reports a 5.1% absolute gain for $\pi_0$ on LIBERO-Plus, an 8% gain for RDT-1B on Robotwin 2.0, and a "slight performance dip" on clean LIBERO. We will run their augmented data through our policies with the naive optimizer and with ours. The question is whether our method removes the dip without giving up the LIBERO-Plus gain.
For the multi-frame architecture, the strongest reported policy is CronusVLA, with a 70.9% success rate on SimplerEnv and a 26.8% improvement over OpenVLA on LIBERO-Long. Our large-scale runs start from this architecture with its stop-gradient intact. Any gain is therefore on top of the existing laziness fix, not a replacement for it.
The decisive optimizer competitor is per-layer learning rates tuned by search. We will give it the same budget of tuning runs that our method spends choosing its single target ratio, plus a generous multiple, on the same data and seeds. If tuned per-layer learning rates match our method only when given much more search, our claim reduces to "a free diagnostic." If they match it at equal search, we report that the rescaling itself adds nothing.
We will also run the following, each matched on adapter or step budget:
- LoRA with the vision encoder frozen, and LoRA with it unfrozen.
- A uniform learning-rate warm restart.
- At the small scale, weight normalization from scratch in the style of Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalization.
Two ablations isolate the key ingredient. The first gives the extra rotation to the temporal and action layers instead of perception, which should increase history reliance if the hypothesis holds. The second rescales layers randomly with the same total norm change.
Why not just…?
Why not just tune weight decay and the learning-rate schedule? Tuning will find a working setting for one shift. Palenicek et al. report that soft regularization needs fragile per-task tuning, while fixing norms works across tasks. A global schedule also cannot fix an imbalance between layers, because it moves every layer's rotation rate by the same factor. Our tuned per-layer learning-rate baseline is the strong version of this objection. It is a real threat, and the equal-search comparison settles it.
Why not just freeze the backbone and train LoRA adapters? Freezing avoids weight-length drift but also prevents the perception layers from adapting, which is exactly what the domain shift requires. The OpenVLA authors found that fine-tuning the vision encoder is critical for manipulation. Our LoRA arms test whether that finding holds under sim-to-real shift for multi-frame policies. If frozen-backbone LoRA matches our target success while retaining source performance, we would conclude the rotation story is unnecessary for practitioners.
Why not just augment harder, or use a single-frame policy? More visual augmentation is the route of Seeing Realism from Simulation, and its clean-LIBERO dip shows augmentation alone does not prevent harmful drift. Single-frame policies remove the history shortcut but lose what history buys. CronusVLA's 26.8% LIBERO-Long improvement over single-frame OpenVLA, and Diffusion Forcing's 80% versus 0% on an occluded real fruit-swapping task (Diffusion Forcing), show that history matters. We include a single-frame policy fine-tuned identically. If the multi-frame advantage disappears after sim-to-real fine-tuning under the naive optimizer but survives under ours, that is direct support for the claim.
Why not just use hyperspherical normalization everywhere, as in SimbaV2? That requires changing and retraining the architecture, which is not available for pretrained VLAs. We test it at the small scale only, as an upper reference for what full norm control achieves.
Proposed Experiments and Decisive Checks
The first check can falsify the premise. If per-layer $\omega$ at the start of fine-tuning is roughly uniform across perception, temporal, and action layers, the hypothesis is wrong and we stop. We will report this measurement for both scales regardless of outcome.
The second check is predictive. Across the three target shifts and several seeds, the initial perception-to-history rotation ratio should correlate negatively with the rise in history reliance and with the loss of clean-LIBERO success under naive fine-tuning. It should also predict which layers show the largest CKA change.
The third check is causal and uses the intervention and its reversed ablation. Giving perception layers more rotation should lower history reliance and raise source retention. Giving the temporal layers more rotation should do the opposite.
The final check is practical. On SimplerEnv WidowX and Google Robot, LIBERO-Plus, and clean LIBERO, our method should beat naive fine-tuning on the combination of target success and source retention. It must also tie or beat equal-search per-layer learning rates for the method to count as more than a diagnostic.
No reported result was found on Lacuna for history-reliance or per-layer rotation measurements in any VLA. Those baselines are measured fresh in this work.
References
- Theoretical Analysis of Auto Rate-Tuning by Batch Normalization
- Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics
- Spherical Motion Dynamics: Learning Dynamics of Normalized Neural Network using SGD and Weight Decay
- Rotational Equilibrium: How Weight Decay Balances Learning Across Neural Networks
- An Exponential Learning Rate Schedule for Deep Learning
- Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalization
- Hyperspherical Normalization for Scalable Deep Reinforcement Learning
- CronusVLA: Transferring Latent Motion Across Time for Multi-Frame Prediction in Manipulation
- OpenVLA: An Open-Source Vision-Language-Action Model
- Seeing Realism from Simulation: Efficient Video Transfer for Vision-Language-Action Data Augmentation
- ReBot: Scaling Robot Learning with Real-to-Sim-to-Real Robotic Video Synthesis
- AnyCamVLA: Zero-Shot Camera Adaptation for Viewpoint Robust Vision-Language-Action Models
- LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning
- Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion