Robot policies trained from camera images break when the camera moves. The strongest current fix is to synthesize the training demonstrations from new camera angles with a diffusion model (a generator that turns random noise into an image by many small denoising steps) and keep the original action labels. Because each image takes tens of sequential network evaluations, this is always done once, offline, with camera poses drawn at random before training starts. We propose to make the view synthesizer fast enough, one network evaluation per image, that it can run inside the policy training loop. That lets the training loop choose which camera poses to synthesize based on where the current policy is failing. The claim is about the second step: once generation is cheap, synthesized views can be chosen to target the policy's current blind spots, and this should beat a random offline cache of the same compute.
In plain terms, the idea is to distill a single-image novel-view generator into a one-step model, then use it like an adversary during imitation learning. It renders the demonstration from camera poses where the policy's predicted actions currently disagree most with the expert's, and a cheap round-trip check discards renders that are broken rather than hard. Latency reduction is the enabling mechanism here, not the end goal. The research question is whether policy-adaptive viewpoint selection, which only online generation allows, buys robustness that a bigger offline cache cannot.
What exists and where the gap is
Two papers establish that generative viewpoint augmentation works for manipulation, and both are offline. VISTA renders single-view demonstrations from new viewpoints with ZeroNVS, keeps the action labels, and filters bad frames by perceptual distance (LPIPS). It raises Threading success under camera shift from about 10% to 32%, and real-robot reaching success from 35% to 88%. RoVi-Aug chains segmentation, inpainting, ControlNet and ZeroNVS to augment both embodiment and viewpoint. Both depend on ZeroNVS, a camera-conditioned diffusion model sampled with many-step DDIM. That cost is why their poses are fixed in advance and why VISTA's rejected frames are expensive. These papers motivate the proposal: they show that generative priors help with large viewpoint shifts, and that their cost is what keeps them offline.
Two other papers move viewpoint handling online, but without a generative prior or without touching training. VR-Drive predicts 3D Gaussians feed-forward from sparse driving cameras and renders new views during training. It cuts nuScenes average L2 error under unseen viewpoints from 1.17 m to 0.68 m. Its own limitations note that feed-forward splatting struggles where unobserved regions must be hallucinated, and its poses are still sampled from a fixed distribution. AnyCamVLA runs a feed-forward view synthesizer (LVSM) at test time to warp the new camera back to the training view. It lifts LIBERO average success under unseen viewpoints from 67.9% to 94.5%, but it needs calibrated poses at deployment and leaves the policy untrained against synthesis artifacts. Both constrain our design: online rendering is already shown to be useful, so our contribution must be the adaptive selection plus a generative prior, not "online" alone.
The method comes from one-step generation. Mean Flows trains a network to predict the average velocity over a time interval rather than the instantaneous one. This allows one-evaluation sampling, reaching ImageNet 256 FID 3.43 at one step against 2.0–2.3 for 250-step diffusion, at roughly 16% extra training compute for Jacobian-vector products. 3D MeanFlow shows the same idea works for conditional geometric completion at about 5 ms per shape, with a shape-level consistency loss to stop drift. It also reports that one-step sampling loses diversity, which matters less for strongly conditioned view synthesis than for unconditional generation. One-Shot Refiner already does one-step diffusion for novel views on RE10K and DL3DV, but only for static scenes and only scored on image metrics. It is never used as an augmentation engine. It is the closest existing system and the natural off-the-shelf alternative generator.
Core Claim
With a one-step camera-conditioned generator, viewpoint augmentation can be steered toward the camera poses where the current policy's action loss is highest. Two conditions apply: the comparison is at equal generator compute, and the steering is gated by a cheap fidelity check. Under those conditions, this produces policies more robust to held-out camera placements than random-pose offline augmentation with a multi-step generator.
Research Program
The first component is the generator. We start from ZeroNVS weights, fine-tune on DROID as VISTA did so grippers render correctly, and then train it as a MeanFlow model conditioned on the source image and relative camera pose $c$. Let $z_t$ be the noisy latent at time $t$ and $u_\theta(z_t, r, t \mid x, c)$ the predicted average velocity from time $t$ back to $r$. The training target follows the MeanFlow identity
$$u_\theta(z_t,r,t) \approx v(z_t,t) - (t-r)\,\frac{d}{dt}u_\theta(z_t,r,t),$$
where $v$ is the instantaneous flow velocity. One-step sampling sets $r=0, t=1$. The multi-step ZeroNVS teacher supplies $v$, so this is distillation-assisted MeanFlow rather than training from scratch. We also train a two-step variant to see where the speed/quality knee lies. Diffusion Policy conditions on a short observation history, so all frames in a history window are rendered at the same pose with shared noise to avoid flicker across frames.
The second component is pose selection. We discretize the camera-perturbation space (azimuth, elevation, translation, field of view) into bins. For each bin $k$ we keep a running estimate $L_k$ of the policy's action loss on views rendered from it. Each minibatch draws its poses with probability proportional to $\exp(L_k/\tau)$, mixed with a uniform floor $\epsilon$ so the policy never stops seeing easy views. Here $\tau$ is a temperature. This works like prioritized replay, but over camera poses.
The third component is the fidelity gate. Adversarial selection has an obvious failure mode: it will happily pick poses where the generator produces garbage, because garbage also yields high action loss. To prevent this, each render is sent back to the source pose with one more generator evaluation, and the round-trip LPIPS against the original frame is computed. Renders above a threshold are dropped, and their loss does not update $L_k$. Because generation is one step, this check costs one extra evaluation, not a second full diffusion chain.
The fourth component is the policy itself. It is a standard image-based Diffusion Policy with a ResNet encoder, trained on mixed real and synthesized frames. We deliberately leave the policy's own action-sampling latency alone so that viewpoint effects are not confounded with action-sampler changes.
Baselines
The naive reference point is Diffusion Policy trained on the single original camera with standard crop and color augmentation. On Threading under camera shift, VISTA reports about 10% success for its single-view baseline, and we expect a similar figure. Beating it is necessary but uninteresting.
The strongest existing training-time method is VISTA-style offline augmentation: multi-step ZeroNVS fine-tuned on DROID, random poses, and LPIPS filtering. Its reported result is 32% Threading success under camera shift (VISTA). We rerun it ourselves on the same demonstrations, the same held-out camera poses and the same policy architecture. We give it exactly the generator GPU-hours our method consumes, so it produces fewer but higher-quality images. Beating it means higher held-out-camera success at equal compute, outside seed variance across five seeds. A second criterion is reaching VISTA-level success with substantially less generation compute.
The baseline that isolates our key ingredient is our own one-step generator used offline with random poses, with the same number of images our online method generates in total. If adaptive online selection does not beat this, the contribution reduces to "a faster ZeroNVS," and we will report it as such. A second ablation keeps adaptive selection but removes the round-trip gate, which tests whether the adversary exploits generator failures.
The online feed-forward competitor is a VR-Drive-style setup (VR-Drive): feed-forward rendering with no generative prior, inside the training loop, with random poses. We swap in One-Shot Refiner's splatting-plus-one-step-refinement where feasible. It is run with the same pose bins, both with and without our adaptive sampler, so that we separate "generative prior" from "adaptive selection."
The test-time competitor follows AnyCamVLA: our one-step generator warps test views back to the training camera in front of the naive policy. Its reported LIBERO number, 94.5% average success under unseen viewpoints with a large VLA, is a reference rather than a like-for-like target, since our policy is smaller. We rerun the idea on our policy so the comparison is fair, and we also report deployment latency per control step.
Why not just…?
Why not just precompute a bigger offline cache? This is the most serious threat. If uniform random poses over a wide range already cover the hard cases, a large cache from the one-step generator will match adaptive selection, and online generation adds nothing except convenience. The equal-image offline ablation above settles this directly. We also test the mechanism: if adaptivity matters, the gap should grow with the size of the pose space (adding elevation and field of view) and shrink when the pose range is narrow. Neither VISTA nor RoVi-Aug reports compute or storage figures; no reported result found on Lacuna for their generation cost, so we will measure it.
Why not just use feed-forward splatting online, as VR-Drive does, with no diffusion prior? It is fast and geometrically grounded. But manipulation scenes from a single wrist or shoulder camera have large unseen regions, and VR-Drive's own analysis says feed-forward splatting fails where hallucination is needed. Our feed-forward baseline, run with and without adaptive selection, will show whether the prior matters. We expect any advantage to show up mainly at large translations such as the 25 cm / 35° shifts where RoVi-Aug reports standard policies collapsing to 0%. If the feed-forward renderer matches us there, the generative prior is unnecessary and we would adopt it.
Why not just normalize the camera at test time like AnyCamVLA? That approach needs calibrated extrinsics at deployment and adds a synthesis call to every control step, and the frozen policy has never seen synthesis artifacts. We evaluate it under perturbed pose metadata, adding pose noise to the "known" extrinsics, and report its per-step latency. If it wins even with noisy poses, training-time augmentation is the wrong lever for this setting.
Why not just use point clouds as in 3D Diffusion Policy? DP3 gains viewpoint robustness by discarding color, but it requires depth sensors and loses tasks distinguished by appearance. We restrict our claim to RGB-only policies and include one color-dependent LIBERO task where a point-cloud policy cannot succeed by construction.
Proposed Experiments
Simulation uses robomimic/MimicGen Threading, Hammer Cleanup and Coffee, matching VISTA's tasks so its 32% figure is a direct anchor. We also use a LIBERO suite with held-out camera placements, following AnyCamVLA's protocol. Training cameras come from one pose. Test cameras are drawn from a held-out grid that includes poses outside the augmentation bins, which checks that the policy has not merely memorized the targeted bins. Metrics are task success, generator GPU-hours, wall-clock training time, fraction of renders rejected by the gate, and generator fidelity as LPIPS against ground-truth simulator renders, where simulation gives us the true view for free. A real-robot check repeats VISTA's cup-on-saucer task with five out-of-distribution camera poses.
The decisive experiment is a single comparison at matched image count: adaptive-plus-gate against random poses, using the same one-step generator. A second check plots success against generator compute for the multi-step offline pipeline and for ours, to locate where one-step quality loss stops mattering. A third verifies that the gate is doing its job: without it, selected bins should concentrate where simulator-measured render error is high, and with it they should not. If the first comparison shows no gain, we will report that viewpoint augmentation is limited by pose coverage and render fidelity, and that adaptivity does not help. That would still tell the field whether to keep paying for multi-step generation.
References
- View-Invariant Policy Learning via Zero-Shot Novel View Synthesis
- RoVi-Aug: Robot and Viewpoint Augmentation for Cross-Embodiment Robot Learning
- ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Image
- VR-Drive: Viewpoint-Robust End-to-End Driving with Feed-Forward 3D Gaussian Splatting
- AnyCamVLA: Zero-Shot Camera Adaptation for Viewpoint Robust Vision-Language-Action Models
- Mean Flows for One-step Generative Modeling
- 3D MeanFlow: One-Step Point Cloud Completion and Generation via Average-Velocity Transport
- One-Shot Refiner: Boosting Feed-forward Novel View Synthesis via One-Step Diffusion
- Diffusion Policy: Visuomotor Policy Learning via Action Diffusion
- 3D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations