When a robot or a video pipeline has to show a scene from a camera it was not recorded with, the standard move is to reproject each pixel using estimated depth. Two failures appear together. Some target pixels have no source pixel at all, because the surface behind them was hidden in the original view. These are disocclusion holes. Other target pixels do have a source, but the new camera is closer, zoomed in, or higher resolution, so a handful of source pixels get stretched across many target pixels. How much they are stretched varies with depth across the image. Today these two problems are handled by separate systems: inpainting models fill the holes and super-resolution models, which increase image detail, sharpen the stretched regions. The inpainting models are usually trained on synthetic renderings because real multi-view data with ground truth is scarce, and they inherit a synthetic-to-real gap.
We propose a single self-supervised recipe that trains one repair model for both failures, using only real monocular images and videos. The idea generalizes a known trick. Geometric Reciprocity showed that under depth-based warping, the holes a real reprojection would create can be reproduced exactly by warping a real frame to a virtual view and back again. That gives free, real training pairs for hole filling. We extend the round trip to virtual cameras with arbitrary, continuous changes in focal length and resolution. Each pixel of the round-trip image then carries two labels. One says whether it is a hole. The other is a continuous local magnification factor, read off the geometry of the warp. A model conditioned on this per-pixel scale map learns to inpaint and to super-resolve at any spatially varying scale in one pass. It is trained on real pixels, with no simulator and no bicubic-downsampling dataset.
Core Claim
Reprojection to a new camera is a single problem with a per-pixel "sampling density" field. At density zero it is inpainting. At density between zero and one it is super-resolution by a locally varying factor. Near depth edges, where density jumps, it is the hard mixed case in which foreground color bleeds into background.
To make this precise, let $w$ map target pixel coordinates $p$ to source coordinates through depth and the two camera models. Let $J(p)$ be the Jacobian of $w$, the matrix of partial derivatives describing how a small patch around $p$ is stretched when mapped into the source. The local magnification is
$$\sigma(p) = |\det J(p)|^{-1/2}.$$
We use $\sigma(p)=\infty$ to mark pixels that fail the visibility test, meaning they are holes. The square root converts the area ratio into a linear zoom factor, so $\sigma$ plays the role of the "×s" scale used by arbitrary-scale super-resolution methods.
The claim has two parts. First, a round trip from a real frame $I$ to a virtual lower-resolution or wider camera $V$ and back to $I$ produces exactly the hole mask and the $\sigma$ field that a genuine reprojection from $V$ to $I$ would. This is the continuous analogue of reciprocity, and it holds by construction of the warp. Second, a model trained on these real round-trip pairs will beat the same architecture trained on synthetic multi-view renderings with exact ground truth, when both are tested on real novel views. The advantage should be concentrated at depth edges and at large $\sigma$.
Positioning
Three strands of prior work define the gap.
The disocclusion literature supplies the self-supervision idea and constrains it. Geometric Reciprocity trains stereo inpainting on ImageNet and Kinetics round trips. It reaches 35.52 dB PSNR on DAVIS-GRT against roughly 28–29 dB for StereoCrafter and StereoDiffusion, which is direct evidence that real self-supervision can beat synthetic-trained repair. However, it assumes a fixed pixel grid with nearest-neighbor warping and identical intrinsics, so the scale change that accompanies zooming or resolution mismatch is outside its model. FreeFix repairs extrapolated 3D Gaussian Splatting views with a frozen diffusion model guided by a Fisher-information confidence mask. It operates per scene through slow 2D–3D iterations at fixed resolution. Our model is feed-forward and scene-agnostic. FreeOrbit4D handles very large camera orbits by cascading several foundation models. That is a different operating point, better suited to 180° redirection than to the small, frequent camera mismatches we target.
The arbitrary-scale strand supplies the scale machinery. ContinuousSR predicts a 2D Gaussian field from a low-resolution image and renders any scale in about 1 ms per scale. Mip-Splatting and Arbi-3DGSR make 3D Gaussian rendering alias-free across scales. Arbi-3DGSR reports a 6.59 dB gain over 3DGS at 5.7× zoom, at about 57 minutes of optimization per static scene. All of these assume one global scale per image, or in the 3D methods a well-observed scene. None handles holes. The 2D methods are trained on synthetic downsampling rather than on the degradation a real warp produces.
The sim-to-real strand supplies both the motivation and the deployment test. AnyCamVLA shows that re-rendering a shifted test camera back to the training camera rescues a frozen robot policy. It raises LIBERO success from 67.9% to 94.5% across unseen viewpoints. Its view-synthesis model runs at a fixed resolution, and the file notes that it cannot perform continuous scale-aware reconstruction under zoom or resolution changes. PixelDA and ASM close sim-to-real gaps by making synthetic data look real, and they leave large residual gaps (22.8% versus a 55.5% oracle mIoU for ASM). We take the opposite route and avoid synthetic training data for the repair model entirely.
Research Program
The data engine takes any real frame together with monocular depth. It samples a virtual camera $V$ with a random small pose change and a random continuous change of focal length and pixel count, spanning magnifications of 1× to 8× and non-integer values. It renders $V$ from $I$ by area-integrated splatting rather than nearest-neighbor sampling, so that $V$ carries the correct band-limited content, following the Mip-Splatting reasoning. It then warps $V$ back to $I$'s camera, recording the hole mask and $\sigma(p)$ from the analytic Jacobian. The training pair is the round-trip image plus its $\sigma$ map as input, and the original $I$ as target. We run this on ImageNet-scale images and Kinetics-scale video, matching the Geometric Reciprocity training corpora so that the comparison with it is clean.
The repair model is an encoder over the round-trip image and the $\sigma$ map. It is decoded, in the manner of ContinuousSR, into a 2D Gaussian field whose kernel sizes are conditioned on local $\sigma$, which lets one forward pass render the output at any target resolution. A lightweight generative head fills pixels with $\sigma=\infty$. Losses are a pixel reconstruction loss, a perceptual loss, and an edge-weighted term on pixels within a few pixels of a depth discontinuity.
The analysis program asks where the benefit comes from. It stratifies error by $\sigma$ band, by distance to the nearest hole, and by depth-edge proximity. It also tests sensitivity to depth-estimation noise, since Geometric Reciprocity names upstream depth quality as its limiting factor.
One honest caveat shapes the design. In our round trip, the low-resolution $V$ is itself resampled from a real image, so its blur kernel is our area integral rather than a real lens-and-sensor kernel. The geometry of the degradation is exact, but its optics are not. We therefore randomize the virtual camera's point-spread function and noise during rendering. The real-camera test described below is designed specifically to reveal whether this residual gap matters.
Baselines
The naive reference is plain depth-based reprojection with bilinear interpolation and no repair. Holes are left unfilled for image metrics and filled by nearest-background propagation for robot-policy tests. It bounds how much repair is worth at all.
The strongest disocclusion competitor is Geometric Reciprocity's image model, which reports 35.52 dB PSNR on DAVIS-GRT. We will retrain it on the same corpora with the same compute and augment its inputs with our scale-changing virtual cameras, so it sees identical data but without $\sigma$ conditioning or a continuous-scale decoder. We will also run its released fixed-scale form. Beating it means higher PSNR and lower LPIPS on real test views with focal-length changes, with the gain concentrated in bands where $\sigma > 1$. On pure-translation DAVIS-GRT, we require parity within noise, so that we do not buy scale handling at the cost of hole filling.
The strongest scale competitor is ContinuousSR. For it, no absolute PSNR figure appears in the file; it reports only a 0.18 dB margin over GSASR on Manga109. We run it in the two orders a practitioner would choose, described under the alternatives below. We give it the same tuning budget and, in one variant, retrain it on our round-trip data so that the data and the architecture can be separated.
For the robot setting, the strongest system is AnyCamVLA, at 94.5% average LIBERO success across unseen viewpoints. We keep its frozen policy and test harness fixed and swap only the view-correction module. Beating it means a higher success rate on LIBERO perturbations that include intrinsic changes: zoom, lower-resolution test cameras, and focal-length mismatch. At the same time, we must match its success on pure extrinsic shifts at comparable latency.
The key ablation isolates the ingredient we claim. We train the identical architecture on synthetic multi-view renderings with exact ground truth, drawn from Blender scenes and from the procedurally generated 3D-IRAIS retail scenes of the IRAIS work, which provide depth and camera parameters. This version gets the same number of training pairs and the same steps. We then compare on real test views. A second ablation removes the $\sigma$ map input while keeping the round-trip data, to test whether explicit scale conditioning matters beyond data diversity.
Why not just…?
Why not just inpaint with Geometric Reciprocity and then run an arbitrary-scale super-resolution model on the result? A single super-resolution pass applies one scale to the whole image. After reprojection, however, magnification varies with depth, so near objects may need 4× while the far background needs 1.2×. Also, the hallucinated hole content gets sharpened as if it were observed detail. We will run this cascade, including a tiled variant that applies ContinuousSR per region at the local median $\sigma$. The stratified error analysis will show whether its losses are confined to mixed-scale and depth-edge regions, as we predict. If it matches us everywhere, the contribution shrinks to the data recipe, and we will report it as such.
Why not just super-resolve the source image first and then warp and inpaint? This is the most serious threat, because it avoids resampling artifacts at their origin. Its known weaknesses are two. First, it must super-resolve to the maximum scale needed anywhere in the image, which costs compute. Second, sharpened foreground edges get smeared across background at depth discontinuities, the classic ghosting of depth-based rendering. More importantly, off-the-shelf super-resolution is trained on synthetic downsampling, which is itself a sim-to-real gap. We will therefore also run this pipeline with ContinuousSR retrained on our round-trip degradations. If that variant ties the joint model, we will conclude that the real-data recipe, not joint modeling, carries the gain, and the experiment settles the question either way.
Why not just use a feed-forward view-synthesis model like AnyCamVLA's, conditioned on intrinsics? Such models learn scale change implicitly from whatever real multi-view data they saw, and they run at a fixed output resolution. AnyCamVLA is documented as lacking continuous scale-aware reconstruction and as producing texture distortions under large disocclusions. The LIBERO intrinsic-perturbation comparison tests this directly, at matched latency.
Why not just optimize a per-scene alias-free Gaussian model with diffusion repair, in the style of Arbi-3DGSR or FreeFix? Those methods need many views and tens of minutes per static scene; Arbi-3DGSR reports about 57 minutes. Our setting is a single live camera stream. We include them only on Mip-NeRF 360 scenes, as an upper-quality reference, not as a deployable competitor.
Proposed Experiments
The decisive real-data test uses real multi-view captures in which target views have true ground truth: Mip-NeRF 360, Tanks and Temples, and LLFF. We warp a downsampled source view into a held-out camera whose image is cropped and resampled to emulate focal lengths 1.3× to 6× longer, and we report PSNR, SSIM, and LPIPS stratified by $\sigma$ band and depth-edge distance. This is also where the synthetic-versus-real training ablation is judged. The central prediction is that the real-round-trip model beats the synthetic-trained model most strongly at depth edges. If the synthetic-trained model wins or ties, the sim-to-real motivation fails and we will say so.
A second check is DAVIS-GRT under its original protocol, to confirm that we keep hole-filling quality relative to 35.52 dB. The deployment check is LIBERO with the AnyCamVLA harness, extended with zoom and resolution perturbations, reporting task success and module latency. For each, we compare against the naive reprojection, both super-resolution/inpainting cascades, Geometric Reciprocity, and AnyCamVLA.
A final diagnostic addresses the optics caveat. We measure how the gap between our model and the synthetic-trained model changes when test sources come from a genuinely lower-resolution camera, such as Waymo frames used as sources for views of the same scene, rather than from downsampled high-resolution images. This tells us whether the residual blur-kernel mismatch in our recipe matters in practice.
References
- Geometric Reciprocity: Unlocking Self-Supervision for Stereoscopic Video Generation
- FreeFix: Boosting 3D Gaussian Splatting via Fine-Tuning-Free Diffusion Models
- FreeOrbit4D: Training-Free Arbitrary Camera Redirection for Monocular Videos via Geometry-Complete 4D Reconstruction
- Pixel to Gaussian: Ultra-Fast Continuous Super-Resolution with 2D Gaussian Modeling
- Mip-Splatting: Alias-free 3D Gaussian Splatting
- Arbitrary-Scale 3D Gaussian Super-Resolution
- AnyCamVLA: Zero-Shot Camera Adaptation for Viewpoint Robust Vision-Language-Action Models
- Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks
- Adversarially Masking Synthetic to Mimic Real: Adaptive Noise Injection for Point Cloud Segmentation Adaptation
- Amodal Instance Segmentation with IRAIS Dataset for Sim-to-Real Transfer