How Reliable Are Level-of-Detail Rendering Rankings? A Program-Committee Audit

City-scale 3D renderers keep large scenes fast with a level-of-detail hierarchy: a tree of progressively finer scene pieces, where the renderer draws a coarse summary of distant regions and only descends to fine detail near the camera. Methods in this family are compared almost entirely by one image-quality number (usually PSNR, peak signal-to-noise ratio) averaged over all test views, plus frames per second. Those test views are drawn from a single training run, at resolutions and downsampling factors that differ from paper to paper. The average hides the two things the hierarchy is built to control. The first is quality at each distance, which corresponds to each level of the tree. The second is the visible "popping" that happens when the renderer switches levels as the camera moves. Leaderboards may therefore be reporting differences that fall inside run-to-run noise, while missing the failures users actually see.

We propose to audit these rankings the way a careful program committee audits its own decisions. Design and Analysis of the NIPS 2016 Review Process showed that averaged cardinal scores were inflated and unreliable, and that a "messy middle" of near-random decisions covered 30–45% of papers. It also showed that ordinal comparisons, meaning relative rankings instead of absolute scores, were more stable. We bring the same tools to hierarchical renderers. Each method is re-run several times under controlled settings. Quality is measured separately for each distance band and at each level switch along continuous fly-throughs. Methods are then ranked ordinally across scenes. Finally, we measure what fraction of published pairwise orderings survive this treatment.

Core Claim

We claim, and will test, that a large share of pairwise orderings among current hierarchical renderers sit in a "messy middle." In these comparisons the gap in mean PSNR is smaller than the spread across random seeds and preprocessing choices. We also claim that distance-stratified and transition-aware measurements reorder methods in ways that human viewers of fly-through videos agree with better than they agree with the averaged score.

The deliverable is a small, reusable protocol together with the measurement of how much it changes conclusions. It is not a new renderer. The protocol has three parts: controlled re-runs, a per-distance-band fidelity profile, and a level-switch popping score. If stratification and replication turn out not to change any conclusion, that negative result is itself worth publishing. Reproducibility: The New Frontier in AI Governance argues for exactly this kind of reporting. It proposes preregistration, adequate statistical power and negative-result reporting, and notes that the presence of code alone is a weak signal of reproducibility.

Why this gap is real

The material supplies clear motivation.

  • The methods under test measure no per-level quantity. Octree-GS places Gaussian "anchors" (small blobs that are the drawing primitives) at octree levels chosen by pixel footprint plus a learned bias. Horizon-GS trains coarse aerial structure first, freezes it, and then adds street detail. HUG partitions scenes into blocks and repairs block seams with vote-based refiltering. All three are built around levels, transitions and seams, yet all report scene-averaged PSNR, SSIM and LPIPS.
  • The methods are fragile in ways that suggest seed and setting variance matters. Octree-GS is known to be sensitive to tree depth, anchor density and splitting thresholds, which must be tuned per scene. HUG evaluates Mega-NeRF scenes downsampled by a factor of four and MatrixCity at reduced resolution. Octree-GS reports speed at 2K. Figures across these papers are therefore not directly comparable.
  • Scene averages already hide large systematic effects. MatrixCity shows that the same model (MipNeRF-360) scores about 25.7 PSNR on aerial views but 23.2 on street views. The same paper documents "dirty air," floating density above buildings, which an average barely penalizes.
  • Task-grounded metrics can outperform image averages. Pseudo-Simulation for Autonomous Driving introduced a driving metric for flat Gaussian splatting that correlates with closed-loop driving scores at $R^2 = 0.8$, versus 0.7 for open-loop metrics. That shows image-level evaluation can be improved on. It does not touch level-of-detail hierarchies or transitions, which is where our protocol works.

Research Program

Controlled re-runs. For each method we fix three things: the input images, the resolution (full MatrixCity resolution plus the downsampling factor used by the original paper, so both settings are visible), and a common hyperparameter search budget. We then train at least five seeds. Methods whose official code we cannot run are excluded rather than reimplemented from scratch, so that we audit the published systems. We preregister the protocol, the scenes, the metrics and the analysis before running anything, following the Reproducibility paper's recommendation.

Distance-banded fidelity profile. MatrixCity provides ground-truth depth for every frame. We bin test pixels by true depth into log-spaced bands and report PSNR and LPIPS per band. We also record which octree level each method actually drew for those pixels. Together these give a curve of fidelity against distance, in place of a single number. Dirty air appears as error in pixels whose true depth is "sky."

Level-switch popping score. We render dense, continuous camera paths from MatrixCity's synchronized trajectories, descending from aerial to street height. Because MatrixCity is rendered from a game-engine city, every frame has ground truth. A method can have perfect per-frame PSNR and still flicker when its chosen level changes, so we score the temporal difference of the rendered video against the temporal difference of the ground truth, restricted to frames $t$ where the set of drawn anchors changes:

$$P = \frac{1}{|S|}\sum_{t \in S} \left\| (\hat I_{t+1} - \hat I_t) - (I_{t+1} - I_t) \right\|_1$$

Here $\hat I_t$ is the rendered frame, $I_t$ the ground-truth frame, and $S$ the set of level-switch frames. For block-based methods like HUG we compute the same quantity on frames where the camera crosses a block boundary, which directly measures seam artifacts.

Program-committee statistics. We treat each (scene, band, seed) as a "reviewer" of each method. A pairwise ordering between two methods is counted as decided only if the seed-level confidence interval of their difference excludes zero. Otherwise it falls in the messy middle. We report the messy-middle fraction of all published pairwise claims we can reconstruct. Across scenes we aggregate ordinally, by average rank, instead of averaging PSNR. PSNR scales differ widely across scenes: 30.59 versus 23.80 on Horizon-GS's own aerial and street splits. One scene can dominate an average, much as inflated reviewers dominate a mean score.

Human ground truth for validity. Because our protocol only matters if it tracks what viewers see, we collect pairwise preferences from participants who watch side-by-side fly-through videos of the same path. Ordinal pairwise judgments are used because Shah et al. found them more reliable than cardinal scales.

Baselines

Naive reference: the published leaderboard. For each method we use its reported scene-averaged PSNR/SSIM/LPIPS from a single run. The strongest reported figures in the material are Horizon-GS's 30.59 aerial and 23.80 street PSNR on its synthetic data, against 21.81 street and 25.44 aerial for flat 3D Gaussian splatting, at about 51 FPS. Octree-GS reports speed above 30 FPS at 2K and up to 10x faster rendering than non-hierarchical methods. For HUG and CityGaussian on MatrixCity, no reported number was found on Lacuna, only qualitative claims of improvement. The protocol "beats" this reference if its rankings on held-out paths and scenes agree with human pairwise preferences more strongly than the leaderboard's rankings do, measured by Kendall's tau with bootstrap intervals. It also beats the reference if it identifies published orderings that fail to replicate.

Strongest existing evaluation practice: seed-replicated averages. This is the protocol a careful reviewer would already request: the standard averaged metrics, re-run over the same five seeds, at matched resolution and with matched tuning budget. It is the real competitor, because it fixes reproducibility without any stratification. Our protocol must beat it on agreement with human preferences for the stratified and popping components to be justified.

Video-quality competitor. We score the whole fly-through with a generic per-frame perceptual metric (LPIPS) plus a generic temporal-consistency term computed over all frames, not only level-switch frames. This tests whether restricting attention to switch frames adds anything.

Rendering methods under test. The subjects are flat 3D Gaussian splatting, Octree-GS, Horizon-GS, HUG and CityGaussian. Each gets the same data, the same GPU type, the same number of tuning trials, and the same wall-clock cap per run.

Ablations. Each ablation removes one ingredient so its contribution can be isolated. We run distance bands without seed replication, replication without bands, averaged versus ordinal cross-scene aggregation, and the popping score computed on all frames versus switch frames only.

Why not just…?

Why not just report mean and standard deviation over seeds? This is the most serious threat, and it may be enough for the reproducibility half of the problem. The seed-replicated baseline above is exactly this. If its rankings agree with human preferences as well as ours do, we will report that stratification and popping measurement add nothing and recommend the simpler fix. We expect it to fall short for one reason. MatrixCity shows the aerial–street gap is systematic rather than random: MipNeRF-360 drops from about 25.7 to 23.2. Seed variance cannot reveal a method that trades street fidelity for aerial fidelity, and averaging can hide that trade.

Why not just use an existing video quality metric? Generic temporal metrics weight every frame equally. Popping, by contrast, occurs in the small fraction of frames where the drawn level changes. The whole-video comparison and the switch-frame ablation directly measure whether this focus matters. If whole-video LPIPS tracks human preference equally well, the popping score is unnecessary.

Why not just evaluate on a downstream task, as pseudo-simulation does? Pseudo-Simulation is the right model when there is a task. Its driving benchmark, however, is built on flat splatting over nuScenes street scenes, with no aerial-to-street descent. Many uses of city-scale renderers are viewing and fly-through rather than planning, so human perception is the appropriate reference for them. We therefore treat downstream metrics as complementary. A natural extension, outside this proposal's scope, is to replace flat splatting in NAVSIM with a hierarchical renderer.

Why not just release everything openly and let replication sort it out? Open code for these methods helps. The Reproducibility paper notes that code availability is rising while replicability remains unmeasured. Releases alone also do not fix mismatched downsampling factors or single-seed claims. Our protocol is what one would run on those released artifacts.

Decisive Experiments

Main study. The core experiment uses MatrixCity's Small City, with held-out continuous aerial-to-street paths. We add real drone scenes from the Mega-NeRF collection, Rubble and Building, at both full and quarter resolution. Real scenes lack dense per-frame ground truth for continuous paths, so on them we report distance-banded fidelity on held-out views only, using depth estimated from the reconstruction. Popping on real scenes is judged only by human preference.

First check: messy-middle fraction. If more than a quarter of published pairwise orderings fall inside seed-level confidence intervals, current single-run leaderboards are unreliable for this family of methods. The threshold is preregistered.

Second check: ranking reversals. We test whether distance-banded rankings reverse aggregate rankings for at least one method pair, in a way consistent across seeds.

Third check: validity. This is the one that decides whether the protocol is worth adopting. Protocol rankings must agree with human pairwise preferences on held-out paths significantly better than the seed-replicated baseline does.

Fourth check: transfer. Rankings produced on MatrixCity must predict human preferences on the real Mega-NeRF scenes at least as well as rankings produced on the real scenes' own averaged metrics. This speaks to whether open synthetic benchmarks can stand in for data that cannot be released.

Cost. Five seeds for five methods on three scenes is a substantial GPU budget. If it proves too large, we will cut the method count before cutting seeds, since replication is the ingredient the whole argument rests on.