Robots trained from a few demonstrations break when the camera sits somewhere it never sat during training. A recent fix renders new viewpoints of the demonstration scene from a 3D reconstruction and trains on those renders. The gains are large, but nobody has measured why they happen. Two explanations fit. The first is that the policy benefits from correct motion parallax, meaning near objects shift more than far ones when the camera moves, so the rendered views carry true depth structure. The second is that the policy benefits only from seeing more viewpoints, and a cheap flat image warp with no depth would work just as well. These explanations lead to very different engineering choices. The first justifies expensive 3D reconstruction. The second says a homography warp, a 2D projective transform of the image, is enough.
We propose a controlled experiment that separates the two. We build a family of interchangeable renderers that differ only in how much depth-correct parallax they produce. The family runs from a single flat depth layer with no parallax, through layered-depth renderers with increasing numbers of layers, up to full 3D Gaussian splatting (a scene model made of many small translucent 3D blobs) and finally the simulator's own exact re-render. Each renderer is packaged as a sealed container behind one shared "render this pose" interface. Every arm therefore sees the same demonstrations, the same augmented camera trajectories, the same policy training code and the same compute. Only the depth structure of the pixels changes. The reproducible-execution machinery is the measuring instrument, and motion parallax is the variable being measured.
Core Claim
The claim is empirical and falsifiable. When a view-augmented visuomotor policy is tested on camera shifts that include translation, its success should rise monotonically with the geometric correctness of the parallax in its training renders. Deliberately wrong parallax, consistent across frames but with the depth scale off, should hurt more than no parallax at all. If instead a zero-parallax warp matches Gaussian splatting, the field has been paying for 3D reconstruction it does not need. Either outcome is worth knowing, and no paper in our file has measured it.
The mechanism is easy to state. Under a small sideways camera translation $t$, a point at depth $d$ moves across the image by roughly $f\,t/d$ pixels, where $f$ is the focal length. A single-plane warp replaces every $d$ with one constant $d_0$. It therefore reproduces viewpoint change exactly for rotation and gets translation wrong for everything off that plane. A $K$-layer renderer quantizes $d$ into $K$ bins. Varying $K$ gives a dose of parallax correctness, from zero at $K=1$ upward. Scaling the true depth by a factor $s$ gives a dose of wrongness. At $s=1$ the parallax is correct. Other values keep occlusion ordering and artifact statistics but get shift magnitudes wrong.
Relation to Prior Work
The work that motivates this proposal is One Demo is Worth a Thousand Trajectories. It reconstructs a scene from a single eye-in-hand fisheye demonstration with fisheye Gaussian splatting, plans new collision-free trajectories, and renders matching views. It reports large improvements over no augmentation. It does not compare against a depth-free augmentation that uses the same trajectories. That comparison is what we add.
The representation we reuse for the middle of the parallax scale comes from MatryODShka. That paper converts stereo 360° video into concentric color-plus-transparency spheres, known as multi-sphere images, and shows that the number of layers controls how much translation is supported before artifacts appear. We use the layered-image idea as a tunable knob rather than as a display pipeline.
Perception work constrains our predictions. Gao et al. (2020) and the stereoscopic distortion analysis in Stereoscopic 3D geometric distortions analyzed from the viewer's point of view show that missing parallax makes human viewers see scenes shear and "follow" the head. This is evidence about human observers, not learned policies, so it does not settle our question.
Execution-environment work supplies the method. Bencher puts benchmarks with incompatible dependencies in separate containers behind a gRPC interface. It supports only scalar objectives, and we extend the pattern to image-returning renderers. Establishing Best Practices for Building Rigorous Agentic Benchmarks shows that trivial agents can score well on flawed benchmarks; for example, empty responses reached 38% on τ-bench. We borrow that "trivial baseline must fail" discipline and apply it to the rendering side: a parallax-free renderer is the "do-nothing" of 3D augmentation. Habchi et al. show that hidden environmental non-determinism corrupts measurements. We treat GPU-dependent non-determinism in splatting as a confound that must be measured before renderer differences can be interpreted. Finally, the EAIR bug study finds that simulation is the most bug-prone module in embodied AI projects. That is a practical reason to hash-check renderer outputs rather than trust them.
Research Program
We will build five renderer containers with an identical render(scene, pose) → image, depth interface. Each container pins its own CUDA stack.
- Oracle: the simulator re-rendering the true scene at the new pose.
- Gaussian splatting: fisheye Gaussian splatting fit to the single demonstration, following Pan et al.
- Layered depth, K layers: built from the splat model's depth, for K in {2, 4, 8, 32}.
- Single plane (K=1): a homography warp at the median scene depth.
- Scaled depth: the oracle geometry with depth multiplied by s in {0.5, 0.75, 1.33, 2}.
A single augmentation planner, identical across arms, generates the new camera and action trajectories once and stores them. Every renderer consumes that same stored list. Policies are trained with the same architecture, seeds and step budget in each arm.
Before any policy comparison we run a determinism audit. Each renderer renders the same poses twice on each of two GPU types, and we record pixel-hash agreement and maximum per-pixel error. We then train policies on renders from each GPU. This lets us report how much variation in success comes from renderer non-determinism alone. Differences between arms smaller than that floor will not be claimed.
Success is judged by the final simulator state (task completed), as RoboMimic already does. We do not score by matching the expert trajectory, and we do not claim novelty for outcome-based scoring. The container harness, trajectory lists and hashes will be released so any arm can be re-run exactly.
Baselines
No augmentation. The naive reference point is a policy trained only on the original demonstration, with no rendered views. Every other arm must beat it on shifted-camera tests, or the setup has not reproduced the basic effect.
Gaussian-splatting augmentation (strongest existing method). This is the method of Pan et al., and it is the strongest competitor because it is the only method in our file shown to fix out-of-distribution camera failures from one demonstration. The file gives only relative numbers. Free-space augmentation achieved "an average task success rate improvement of 56% over policies trained without augmentation" in simulation. It improved real-world robustness to out-of-distribution camera views by 55%. It reached 100% versus 0% success on real obstacle scenarios. The paper also found 50° of rotation offset to be the best trade-off between diversity and render quality. No absolute RoboMimic square success rate is reported in the file. We therefore re-run this arm ourselves inside the shared harness, with its published settings including the 50° bound, and treat our reproduction as the reference. Reproducing a clearly positive gain over no augmentation is a precondition for the study.
Single-plane warp (key ablation). This is the ablation that isolates the proposal's key ingredient. It gets the same trajectories, the same rotations, and nearly the same image statistics as the splatting arm, but no parallax. Beating it means the splatting arm must show a success margin, on translation-heavy test shifts, larger than the determinism floor plus seed variance.
Scaled-depth arms (wrong-parallax control). These keep occlusion ordering but make parallax magnitudes wrong. They separate "correct depth matters" from "any 3D-looking artifacts help."
Oracle. The oracle is the ceiling. It is what collecting many real views would achieve in simulation.
Why not just…?
Why not just use 2D warps, crops and color jitter, which are cheaper? This alternative is a real threat, and the study is built to test it rather than argue it away. The file's evidence against 2D warps concerns human perception (Gao et al.), not policy learning. Pan et al. compare only against no augmentation. If the single-plane arm matches splatting under translation, we will report that plainly and recommend the cheap method. The test split separates rotation-only shifts from translation shifts for this reason. Warps should match splatting on rotation-only shifts, and any gap should appear on translation.
Why not just pin a conda environment and fix seeds instead of using containers? For one lab on one machine, that might be enough. The five renderers need mutually incompatible CUDA and graphics stacks, however. Bencher documents exactly this failure of package managers with conflicting low-level libraries. The cross-GPU determinism audit also needs identical software on different hardware. If the audit shows bit-identical renders and no difference in success across hardware, we will say the containers were a convenience, not a necessity.
Why not just rank renderers by image-quality metrics such as PSNR or LPIPS? If image metrics predicted policy success, policy training would be unnecessary. We compute PSNR, SSIM and LPIPS against the oracle for every arm and report their rank correlation with policy success. The scaled-depth arms are a pointed test. At moderate s they can score well on per-image metrics, because textures stay sharp, while the parallax is wrong. A strong correlation would mean our harness is overkill for renderer selection.
Why not just collect more demonstrations from more viewpoints? In simulation the oracle arm is exactly that, so the gap between oracle and splatting measures what reconstruction costs relative to real data. The practical reason not to rely on collection is cost in the real world, which is Pan et al.'s motivation.
Proposed Experiments
The primary benchmark is the RoboMimic square task with an eye-in-hand camera. Test-time camera offsets are split into rotation-only, translation-only (5–15 cm) and combined shifts. Each arm is trained with five seeds per GPU type. The metric is task success by final simulator state.
The first decisive check is the parallax dose curve. Success should increase with K from 1 to 32 on translation shifts and stay flat on rotation-only shifts. The second is the wrong-parallax check: scaled-depth arms should fall at or below the single-plane arm. The third is the metric-validity check, the rank correlation described above. The fourth is the determinism floor, the success variance attributable to cross-GPU render differences.
If time allows, a secondary replication uses the UMI real demonstrations with the single-plane and splatting arms only. No reported absolute result was found in the file for these, so any comparison will rest on our own reproduction.
The result that would most change practice is a flat dose curve, meaning parallax does not matter. The result that would most support 3D reconstruction is a monotone curve together with a wrong-parallax penalty.
References
- One Demo is Worth a Thousand Trajectories: Action-View Augmentation for Visuomotor Policies
- MatryODShka: Real-time 6DoF Video View Synthesis Using Multi-sphere Images
- Stereoscopic 3D geometric distortions analyzed from the viewer’s point of view
- Bencher: Simple and Reproducible Benchmarking for Black-Box Optimization
- Establishing Best Practices for Building Rigorous Agentic Benchmarks
- A Qualitative Study on the Sources, Impacts, and Mitigation Strategies of Flaky Tests
- An Empirical Study on Embodied Artificial Intelligence Robot (EAIR) Software Bugs