A robot policy trained with its camera in one place often fails when the camera sits somewhere slightly different at deployment. This happens when a simulated camera does not match the physical rig, when a bracket is bumped, or when a new robot copies an old setup imperfectly. The current fix synthesizes a new image from the training viewpoint using a large rendering network. That network needs the camera's exact pose and lens parameters, and it re-renders every pixel. We propose something much smaller. Some recent policies already read the image at a handful of learned, continuous locations, much as a person glances at a few spots. For these policies, we leave the image alone and instead move the glance points. At the start of each episode, a small geometric warp is fitted by gradient descent with no calibration. It maps where the policy thinks it is looking (in training-camera coordinates) to where that point actually is in the deployed camera's image. The policy then receives features from the right physical locations, still labeled with the coordinates it was trained on.
This works because of differentiable spatial sampling: reading a feature map at fractional coordinates by blending the nearest grid cells (bilinear interpolation), so that the read-out value has a gradient with respect to the coordinates. Spatial Transformer Networks introduced this to let a network learn its own image warps. GridS used it to make a vision-language-action policy, meaning a robot controller that maps camera images and a language instruction to motor commands, read only $K$ continuously placed tokens instead of a full patch grid. Our claim is that the same mechanism can absorb the camera-mismatch part of the sim-to-real gap. The warp is fitted per deployment from a self-supervised feature-matching signal anchored on the robot's own body. It needs no camera intrinsics or extrinsics, uses a few hundred parameters, and runs at the cost of $K$ point reads.
Core Claim
For moderate camera displacements, the kind produced by rig mismatch, bumps and re-mounting, a frozen sparse-sampling policy recovers most of its canonical-view success rate if its sampling coordinates are passed through a fitted warp. The warp is a plane-induced homography, the projective map that exactly relates two views of a flat surface, plus a small smooth residual. It is fitted by gradient descent through the bilinear sampler so that the frozen encoder's features at a home-pose frame match stored canonical features. We make three testable predictions. First, this alignment should approach the calibrated, 171M-parameter rendering approach of AnyCamVLA inside a characterizable range of viewpoint shift, and fall off beyond it. Second, fitting in the frozen encoder's feature space should outperform classical keypoint-plus-RANSAC homography estimation when viewpoint shift is combined with appearance shift such as lighting or rendered-versus-real texture. Third, warping sampling coordinates should beat applying the same warp to the full image, because it avoids resampling blur and empty borders over the whole frame.
The homography assumption is the honest weak point. It exactly aligns only the table plane, and objects with height show parallax, meaning they shift by different amounts than the plane when the camera moves. Our bet is that manipulation mostly depends on contact points near the support surface. Where that bet fails, the rendering approach should win, and we intend to measure where the crossover happens rather than hide it.
How It Relates to Prior Work
The motivating evidence comes from two sources. AnyCamVLA shows that camera viewpoint is a dominant deployment failure: a base policy averages 67.9% success on LIBERO under unseen viewpoints, and 94.5% when a feed-forward novel-view-synthesis model re-renders test images into the training view. That paper also argues that fine-tuning the policy on diverse viewpoints trades away base skills. Its method requires calibrated camera parameters and a large per-frame renderer, and we remove both requirements at the cost of a weaker geometric model.
GridS supplies the substrate. A small network predicts $K$ continuous coordinates, features are bilinearly sampled there, and coordinate embeddings are attached. The paper reports a 76% reduction in floating-point operations with a 1.6-point success gain on LIBERO, and a 28.6% gain in real-robot out-of-distribution settings on an SO100 arm. Its robustness comes from compression alone; nothing in it aligns the deployed geometry to the training geometry. We add exactly that alignment, which is possible because GridS keeps "where to read" separate from "what coordinate label the policy sees."
Affine Transport for Sim-to-Real Domain Adaptation constrains the design. It fits closed-form global affine maps between simulated and real state distributions and reduces a puck-prediction error from 0.73 to 0.08. It also notes that global low-dimensional maps break down on high-dimensional, nonlinear shifts. We therefore restrict the fitted warp to the one factor with known low-dimensional structure, camera geometry, and do not ask it to fix appearance.
Differentiable Kernel Evolution constrains the optimizer. It shows that bilinear coordinate gradients are biased and vanish on flat feature regions. This is why we fit coarse-to-fine over a feature pyramid from several initializations, rather than trusting single-scale gradient descent.
Research Program
We would build three components. The first is an alignment module that applies a homography $H$ (8 parameters) plus a residual displacement field $r$ on a coarse $4\times4$ control grid, smoothly interpolated. Each canonical sampling coordinate $x_k$ predicted by the frozen GridS head is mapped to $\tilde x_k = H(x_k) + r(x_k)$. Features are read at $\tilde x_k$, and the canonical embedding of $x_k$ is kept.
The second is a fitting objective. We use the frozen visual encoder's feature map $\Phi$ on a test frame $I$ captured with the robot commanded to its home pose, and a stored canonical feature map $\bar\Phi$ averaged over training-episode first frames. Both are compared over a dense grid of points $u$ in a mask $M$ covering the robot base, arm and table. The fitting problem is
$$\min_{H, r} \; \sum_{u \in M} \big\| \Phi(I)[H(u) + r(u)] - \bar\Phi[u] \big\|^2 + \lambda \|\nabla r\|^2,$$
where brackets denote bilinear sampling and $\lambda$ penalizes non-smooth residuals. Anchoring on the robot at a known joint configuration gives the calibration target for free: the arm's appearance at home pose is the same across training episodes, while task objects are not. The mask $M$ is defined once in canonical coordinates. During an episode, we optionally re-fit every few seconds, warm-started from the previous solution, against the training frame whose recorded joint configuration is nearest to the current one. This handles vibration or drift.
The third component is a dense-policy variant. The same fitted warp is applied to the whole image before a conventional patch-based policy, which makes the method usable beyond GridS and gives the image-versus-sampler ablation.
On the analysis side, we would chart success rate against a single viewpoint-shift magnitude, combining rotation angle and translation normalized by camera-to-table distance. We would also report homography estimation error against ground truth in simulation, and diagnose failures by object height to test the parallax explanation directly.
Baselines
The naive reference point is the frozen policy with no correction, run both as the dense base policy and as the GridS policy on the shifted cameras. GridS's compression may already buy some viewpoint robustness, so the GridS no-alignment condition is the reference our method must beat. On LIBERO's unseen-viewpoint setting, the dense counterpart is the 67.9% base-policy figure reported by AnyCamVLA. For GridS under controlled camera shifts on LIBERO-PLUS, we found no reported per-perturbation result on Lacuna, so we will measure it ourselves.
The strongest existing method is AnyCamVLA. It reports 94.5% average success across unseen LIBERO viewpoints with a 171M-parameter renderer at 30Hz in front of a frozen 7B policy. We would run it on identical viewpoint sets and policies in two conditions: with true calibration, as it assumes, and with calibration estimated from a checkerboard-free procedure, to show how it degrades when the premise weakens. Beating it means matching its success within a few points over the moderate-shift range, without calibration and at under 1% of its parameters and latency. We do not expect to beat it at large shifts and will report that range as a loss.
The classical competitor is a keypoint homography. Local keypoints are matched between the home-pose test frame and the canonical frame, a homography is estimated robustly with RANSAC, and it is applied either to the image or to the sampler coordinates. It gets the same mask, frames and tuning budget as our fitter. We found no reported result on Lacuna for this baseline on LIBERO or SO100. Beating it means a clear success-rate gap under combined viewpoint and lighting or texture shift; a tie under pure viewpoint shift would be expected and acceptable.
Two further baselines isolate the key ingredients. The first applies the identical fitted warp to the full image instead of the sampling coordinates. The second fits the warp with a generic whole-image feature match instead of the robot-anchored mask. The fine-tuning baseline trains the policy with viewpoint augmentation under a matched compute budget. We score it on both shifted views and canonical views, so any forgetting of base skills that AnyCamVLA describes shows up.
Why not just…?
Why not just run AnyCamVLA? It works and is the upper reference. However, it needs metric camera calibration and a 171M-parameter renderer per frame, which is often impractical on low-cost arms like the SO100, where cameras get bumped mid-task. If our method fails to approach it even at small shifts, the proposal's central claim is false. The success-versus-shift curve settles this.
Why not just estimate a homography with keypoints and RANSAC? This is the most serious threat, because geometrically it solves the same problem without any differentiable machinery. Its known weakness is matching across appearance change. Rendered-versus-real texture, lighting shifts and textureless robot links produce few reliable keypoints, while features from a policy encoder trained for the task may still match. If keypoint RANSAC equals our fitter under combined appearance and viewpoint shift, we would drop the feature-space fitting claim. The remaining contribution would then be applying any estimated warp to sampler coordinates, and the image-versus-sampler ablation would decide whether even that holds up.
Why not just fine-tune or augment the policy with randomized viewpoints? AnyCamVLA reports that viewpoint fine-tuning trades away base skills, and augmentation cannot cover a rig mismatch not anticipated at training time. Our matched-compute fine-tuning baseline, scored on both canonical and shifted views, tests this directly rather than taking it on citation.
Why not just rely on GridS's compression? GridS already reports real-robot out-of-distribution gains, including under camera shifts. If the no-alignment GridS policy already stays flat as shift grows, alignment adds nothing and we will say so. The prediction is that it degrades with shift magnitude, because its coordinate embeddings encode training-camera geometry.
Proposed Experiments
In simulation, we use LIBERO task suites from LIBERO and the camera-perturbation and lighting splits of LIBERO-PLUS, both used in the GridS evaluation. We render cameras at graded rotations and translations so the true homography of the table plane is known. We also cross viewpoint shifts with lighting perturbations as a controlled proxy for appearance mismatch between rendered and real images. On hardware, we deploy on an SO100 arm with the camera deliberately re-mounted at measured offsets and bumped during episodes, with a Franka Panda setup as a secondary platform if available. Metrics are task success rate, warp error in pixels against ground truth (simulation only), fitting time, per-step overhead in milliseconds, and parameter count. Given the GridS finding that 76% fewer operations gave only a 1.2× wall-clock speedup, we report measured latency, not operation counts.
Four results would be decisive. The first is a success-versus-shift curve locating where our method stays within a few points of calibrated AnyCamVLA and where it falls away. The second is a failure breakdown by object height; if failures do not concentrate on tall objects at large shifts, the parallax story is wrong. The third is the combined appearance-and-viewpoint condition, where feature-space fitting must beat keypoint RANSAC to justify itself. The fourth is the sampler-versus-image ablation at a matched warp, which shows whether re-aiming the glance points is itself the useful idea.
References
- Spatial Transformer Networks
- See What Matters: Differentiable Grid Sample Pruning for Generalizable Vision-Language-Action Model
- AnyCamVLA: Zero-Shot Camera Adaptation for Viewpoint Robust Vision-Language-Action Models
- Affine Transport for Sim-to-Real Domain Adaptation
- Differentiable Kernel Evolution
- LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning