Renderer-Swap Replay: Separating Agent Failure from Scale Aliasing in Gaussian-Splat Simulators

Embodied AI benchmarks are moving onto 3D Gaussian Splatting, a scene representation built from millions of small, colored, semi-transparent 3D blobs that are projected onto the image. It renders photorealistic rooms quickly. The problem is that standard splatting is tuned for the camera distance it was trained at. A robot that walks up to a cabinet sees thin, spiky "needle" artifacts. When the same robot looks across a hall, it sees flicker and blown-out brightness. Graphics researchers call this multi-scale aliasing and have built renderers that remove much of it. The new navigation and manipulation benchmarks still render with the plain rasterizer. When an agent fails there, nobody can say whether its reasoning failed or the pictures it was shown were broken.

We propose a measurement protocol, renderer-swap replay. Each benchmark episode is recorded inside a hermetically sealed, version-pinned container as a sequence of camera poses and simulator states. The episode is then replayed with only the rendering module changed: the vanilla rasterizer, an anti-aliased one, or an expensive supersampled reference. We count how often the agent's actions and episode outcomes flip. That flip rate, broken down by how large each pixel's footprint is relative to the splats it covers, gives a direct estimate of how much of a benchmark score reflects rendering rather than the agent. It can be reported as a validity check alongside the score. The reproducible-environment machinery is what makes the estimate trustworthy: without bit-level control over everything except the renderer, a flip cannot be attributed to aliasing.

Core Claim

In Gaussian-splat embodied benchmarks, a measurable share of agent failures is caused by scale aliasing rather than by perception or planning. That share is concentrated in frames whose camera-to-content scale falls far from the training scale, and it can be isolated by counterfactual renderer replay in a hermetic harness. A second, falsifiable claim follows. If the flip rate is high, swapping in a stateless anti-aliased renderer should recover part of the gap the benchmarks themselves report. If the flip rate is near the noise floor, the community should stop attributing these gaps to rendering. Both outcomes are publishable and useful.

Where This Sits

Two boundary papers motivate the work. Towards Physically Executable 3D Gaussian for Embodied Navigation (SAGE-3D) attaches invisible collision meshes to splatted scenes and releases SAGE-Bench. It notes a "photorealism tax": navigation models need more training iterations because of high-frequency visual noise. RoboTidy builds a tidying benchmark on the same scene data, with wide navigation cameras and close-up wrist cameras. It reports NaVid-base success dropping from 0.22 on R2R to 0.16 in its scenes. Neither paper tests whether the renderer causes these effects. That is the gap we close. We do not claim either paper's drop is caused by aliasing. We build the instrument that decides.

The anti-aliasing literature supplies the renderers we swap in. Analytic-Splatting integrates each blob's contribution over the whole pixel area instead of sampling at the pixel center. It reports 35.03 dB PSNR on Blender Synthetic versus 29.77 dB for vanilla 3DGS and 34.56 dB for Mip-Splatting, at roughly 10% lower frame rate. Mipmap-GS instead lets blobs change shape for each viewing scale, supervised by super-resolved or downsampled pseudo-ground-truth images. That runtime change of scene geometry conflicts with an immutable, replayable environment. The file's own analysis flags this tension, so we use Mipmap-GS only with its per-scale deformations frozen into precomputed, hashed assets. We do not use it as a live renderer. The older radiance-field work, Mip-NeRF and Zip-NeRF, establishes that filtering over the pixel's cone of view can match brute-force supersampling at much lower cost. Mip-NeRF reports matching 128× supersampling while running 22× faster. This justifies treating supersampling as an offline reference rather than a deployed method.

The reproducibility literature supplies the harness design. Bencher isolates more than 80 conflicting benchmarks in containers behind a gRPC interface. It has only handled scalar objectives, never camera-pose-to-image rendering. OpenHands and MacArena show how to reset sandboxes and VMs to a pristine state per task. Establishing Best Practices for Building Rigorous Agentic Benchmarks shows that 70% of audited agent benchmarks violate task or outcome validity, including a do-nothing agent scoring 38% on τ-bench. Its checklist treats observations as text or fixed-resolution screenshots and has no criterion for rendering fidelity. Our flip-rate metric is a candidate addition to that checklist. It is a direct analogue of the do-nothing check, aimed at the visual channel.

Research Program

The first deliverable is a renderer service. It is a container image exposing one gRPC call that takes a scene identifier, camera pose, and intrinsics and returns an image, with the rasterizer selected by a pinned build tag. The vanilla, analytic-integration, and frozen-Mipmap-GS rasterizers ship as separate images with identical interfaces. The simulator (Isaac Sim for SAGE-Bench and RoboTidy) and the policy run in their own containers. This removes the CUDA, PyTorch, and Omniverse conflicts RoboTidy faces when co-hosting a 72B model, and it ensures a renderer swap touches nothing else.

The second deliverable is the replay protocol. An episode is first run live with the vanilla renderer, logging every pose, collision state, and action. The replay then feeds the same simulator states to the policy through a different renderer. We use two modes. Open-loop replay counts per-step action disagreements on fixed states. Closed-loop rerun lets the trajectory diverge and records outcome flips in success, Continuous Success Ratio, and Integrated Collision Penalty.

Before any flip is attributed to aliasing, we measure the noise floor. We rerun each episode with the same renderer on the same GPU and on a second GPU model, then record pixel drift and action flips caused by nondeterministic GPU arithmetic alone. Renderer-induced flips must exceed this floor.

Every frame is tagged with a scale ratio $r = s_{\text{pix}} / s_{\text{train}}$. Here $s_{\text{pix}}$ is the world-space footprint of a pixel at the depth of the dominant splats, and $s_{\text{train}}$ is the median footprint seen during scene reconstruction. Flips are reported as a function of $r$. If aliasing is the mechanism, flips should rise toward both small $r$ (close-ups, needle artifacts) and large $r$ (distant views, flicker) and stay flat near $r=1$. If flips are uniform in $r$, the mechanism is something else, for example a generic shift in appearance, and the aliasing hypothesis fails.

The third stage asks whether the finding matters for training. We train the same navigation policy on SAGE-Bench with each renderer at matched environment steps. We compare convergence speed, which is the photorealism tax measured directly, and final success. Every policy is evaluated under every renderer, so we can tell robustness apart from renderer-specific overfitting.

Baselines

The naive reference is the benchmark as published: the vanilla point-sampled rasterizer and the existing SAGE-Bench and RoboTidy evaluation code, run in our containers but otherwise untouched. For context, RoboTidy reports NaVid-base at 0.16 success in its scenes against 0.22 on R2R. SAGE-3D reports a 31% relative success gain on VLN-CE Unseen for SAGE-trained models. We found no reported result on Lacuna for any embodied benchmark evaluated under an anti-aliased Gaussian renderer. Beating the reference therefore means a statistically significant success gain under anti-aliasing, together with flips that concentrate at extreme $r$.

The strongest existing anti-aliasing method we deploy is Analytic-Splatting. It has the best image-quality number in the file, 35.03 dB on Blender Synthetic, and it is stateless, so it is compatible with deterministic replay. It runs on the same reconstructed scenes and the same GPU, with frame time reported so that its roughly 10% throughput cost is charged against it in equal-wall-clock training comparisons.

The quality ceiling is 8× supersampling: rendering at higher resolution and box-filtering down. It runs offline only. It defines "correct" pixels for the scale-binned error curves and the upper bound on recoverable flips. A deployed renderer counts as beating the vanilla reference if it closes a substantial fraction of the success gap to the supersampled ceiling at near-vanilla frame time.

The ablation isolating our key ingredient, hermetic attribution, repeats the swap comparison without container pinning and without the noise-floor subtraction. This shows how many apparent renderer effects are environment drift. A second ablation replaces the anti-aliased renderer with a fixed 2D blur of the vanilla output, matched in overall image sharpness. It separates scale-aware filtering from plain low-pass filtering.

Why not just…?

Why not just swap in Analytic-Splatting, retrain, and report the success gain? That conflates three things: rendering quality, a change in appearance distribution, and run-to-run drift. A higher score would not show that aliasing caused the original gap. Open-loop replay with noise-floor subtraction and binning by $r$ is what attributes the effect. The retraining experiment is still included as stage three.

Why not just supersample? Supersampling at 4× quadruples the rasterization load in simulators that SAGE-3D runs at 6.2 ms per frame across millions of trajectories. Mip-NeRF's 22× speedup over 128× supersampling shows that filtered rendering can match the brute-force result far more cheaply. We keep supersampling as the offline ceiling, not as the method.

Why not just train the policy with blur or noise augmentation so it becomes robust to artifacts? This is a real threat. If an augmented vanilla-trained policy shows a flip rate as low as an anti-aliasing-trained policy, the practical recommendation becomes "augment" rather than "fix the renderer." We run it as a full arm of stage three. One caveat applies: Analytic-Splatting's comparisons argue that 2D blur cannot restore geometry eroded into needles up close, so we expect augmentation to fail mainly at small $r$. The experiment decides.

Why not just pin a Dockerfile? Pinning is necessary but not sufficient. Identical images can still drift numerically across GPU models, as the file notes for nondeterministic CUDA atomics. The two-GPU noise-floor measurement quantifies this. Isolating the renderer as its own service, following Bencher, is what makes one-variable swaps possible.

Decisive Checks

The project succeeds or fails on three measurements on SAGE-Bench and RoboTidy navigation episodes.

The first is whether renderer-swap action flips exceed the same-renderer, cross-GPU noise floor by a clear margin.

The second is whether those flips form the predicted U-shape in $r$. RoboTidy's wrist camera tests the small-$r$ side and its navigation camera the large-$r$ side.

The third is whether training under analytic integration shortens convergence and closes part of the 0.22-to-0.16 gap more than blur augmentation does at matched wall-clock time.

A flat flip curve at the noise floor would be an equally clear negative result. It would show that the "photorealism tax" does not come from aliasing, and the protocol would still stand as a cheap validity audit for future splat-based benchmarks.