Parallax, Not Meters: Correcting Monocular Depth for Texture Alignment

Many 3D reconstruction pipelines build a scene by predicting depth from single images and then "painting" each photograph onto the resulting surface. When two photographs disagree about where a surface lies, the painted result shows ghosting (an edge appears twice), blur, or torn seams. The usual diagnosis blames monocular scale ambiguity: a single image cannot tell a small near object from a large far one. The usual remedy is to make the depth model output physical meters. This note argues that the diagnosis is partly wrong and the remedy is misdirected. Texture alignment does not care about meters. It cares about whether each view's depth agrees with the scale of the camera poses, and about errors that vary across the image. These errors are amplified by parallax, meaning the apparent shift of a point between two views. A depth model that is more metrically accurate can still produce worse textures.

We propose to make this precise and act on it. First, a controlled study measures how each component of depth error (a global scale offset relative to the poses, a global shift, a smooth spatial warp) turns into texture misalignment, and tests a simple first-order prediction of that relationship. Second, a correction method fits a low-dimensional, spatially smooth scale field for each view's monocular depth. The fitting objective is agreement of high-frequency texture across views, not agreement with ground-truth depth. The corrected depth then initializes textured Gaussian primitives. The decisive question is whether this texture-driven, locally varying correction beats two cheaper fixes on held-out views: per-view global scale-and-shift alignment, and metric foundation-model depth.

Core Claim

Take a point seen at depth $Z$ in a source view, with depth error $\delta Z$. Re-render it in a second camera displaced by baseline $b$ perpendicular to the viewing ray, with focal length $f$ in pixels. The projected texel lands off target by approximately

$$\Delta x \approx \frac{f\,b}{Z}\cdot\frac{\delta Z}{Z},$$

which is the point's disparity between the two views times its relative depth error. This is the first-order derivative of the projection $x = fX/Z$ quoted in the research file.

Three consequences follow, and none of them is acknowledged in the scale-ambiguity literature. First, if depth and camera translations are scaled by the same factor, $\delta Z/Z$ is matched by a scaled $b$ and nothing moves on screen. Absolute metric scale is therefore invisible to texture alignment. Second, metric depth paired with up-to-scale poses causes uniform misalignment. Such poses come from structure-from-motion, as in RealEstate10K-style walkthroughs. Here a constant $\delta Z/Z$ produces a disparity-proportional error everywhere, so plugging a "better" metric model into an existing pipeline can introduce ghosting. Third, the error that remains after the best global fit is a spatially varying relative error, and only near, high-parallax, high-frequency regions punish it. Aggregate depth metrics such as absolute relative error (Abs Rel) average it away.

The proposal's claim is that the correct target for monocular depth correction in texturing is this parallax-weighted residual. The cheapest signal that measures it is cross-view disagreement of high-frequency image content.

Relation to Prior Work

Two papers motivate the problem by showing that texture fidelity is hostage to geometry. Less Gaussians, Texture More: 4K Feed-Forward Textured Splatting (LGTM) attaches a small texture map to each Gaussian primitive, a soft 3D blob used for fast rendering, and fills it by projecting high-resolution source pixels onto the primitive's plane. Its authors state that this projection distorts when primitives sit at the wrong depth, and that sparse-view textures break at wide novel angles. DiffTex optimizes per-texel blending weights over source pixels with photometric, grazing-angle and smoothness losses, and prunes inconsistent observations. It assumes known proxy geometry and calibrated cameras. We reuse LGTM's textured primitives and DiffTex's idea of consensus across views. We differ from both by treating the geometry itself as the quantity to correct, under an error model derived from parallax.

Three lines of work resolve monocular scale, and all three supply components or constraints. Metric3D v2 normalizes focal length into a canonical camera to output metric depth and normals. It needs focal-length metadata. ScaleDepth and RSA predict a single scene-level scale (and, for RSA, shift) from CLIP language embeddings. All three are optimized and evaluated on depth error against sensors. All three use a global correction, which by our error model cannot remove the residual that matters. We use them as initializations and baselines, not competitors in kind. LiDARTouch anchors scale with a 4-beam LiDAR. It sets a hardware reference point that we treat as out of scope but discuss below.

The closest existing work is Mode-GS. It initializes Gaussians from monocular depth and optimizes a per-view scale online to make views consistent, using ordinary single-color Gaussians. It is the strongest existing instance of "make monocular depth consistent across views." Our proposal differs in three ways: the correction is a spatial field rather than one scalar, the objective is high-frequency cross-view agreement rather than overall rendering loss, and the downstream representation is textured primitives, whose fidelity depends far more on depth. YoNoSplat normalizes scene scale by the maximum distance between cameras. It is an implicit precedent for our first consequence: consistency rather than meters is what rendering needs. It does not study texture alignment.

Research Program

The first component is a perturbation study on ScanNet++, chosen because it pairs high-resolution images with scanned geometry. We start from ground-truth depth, which is an oracle texturing condition. We then inject controlled errors: a global scale mismatch between depth and pose, a global shift, smooth low-frequency warps of varying amplitude, and the actual residual patterns of Metric3D v2 and Depth Anything after optimal global alignment. For each condition we texture with an LGTM-style textured-Gaussian renderer and with DiffTex-style blending. We then measure held-out-view error both in full and in a high-pass band, along with the measured pixel offset of matched edges. This tests whether the displayed equation predicts the observed offset. It also tests whether "relative error × disparity" predicts texture degradation better than Abs Rel does.

The second component is the correction method. For each view $i$ we keep the monocular depth $D_i(u)$ at pixel $u$ fixed and learn a log-scale field $g_i(u)$. This field is a bicubic spline over a coarse control grid, from 1×1, which is pure global scale, up to 8×8. Corrected depth is $\exp(g_i(u))\,D_i(u) + c_i$, where $c_i$ is a per-view shift. The fields are fit by minimizing

$$\mathcal{L} = \sum_{(i,j)} \sum_{u} w_{ij}(u)\,\big\| H(I_i)(u) - H(I_j)(\pi_{j}(u; \exp(g_i)D_i + c_i)) \big\|_1 + \lambda \sum_i \|\nabla g_i\|^2 .$$

Here $H$ is a high-pass filter on image $I$, and $\pi_j$ reprojects pixel $u$ of view $i$ into view $j$ using the corrected depth and known up-to-scale poses. The weight $w_{ij}$ downweights grazing angles and occlusions, following DiffTex's perspective reasoning. The coefficient $\lambda$ sets smoothness. The high-pass filter matters because low-frequency photometric agreement is nearly flat in depth for smooth surfaces. Edges and fine texture are where depth errors become visible, so they also carry the gradient. The corrected depths then seed textured Gaussians, which are optimized normally. In one variant, primitive positions are frozen to the corrected depth. In the other, they are free to move.

Baselines

The naive reference point is textured Gaussians initialized from raw relative depth (Depth Anything) fitted to the poses only by structure-from-motion sparse points, with no further correction. It shows how much misalignment exists before anyone tries to fix it.

The metric preprocessor baseline runs Metric3D v2 with true intrinsics and feeds its depth unchanged into the same textured-Gaussian pipeline. It is the strongest available monocular geometry and the most likely reviewer suggestion. Metric3D v2 reports Abs Rel of 0.047 on NYUv2 and 0.044 on KITTI zero-shot. We will run it at the same resolution and with the same downstream optimization budget as our method. A variant aligns its depth to the pose scale with one least-squares scalar, which separates "metric" from "pose-consistent." RSA (Abs Rel 0.097 on NYUv2 versus a 0.094 linear-regression oracle, per RSA), run from its official code, represents language-based global scaling.

The strongest existing method for this question is Mode-GS-style per-view scale calibration. We reimplement it inside our renderer: one scale per view, optimized jointly with the Gaussians under the rendering loss. Mode-GS reports PSNR 17.27 versus 15.84 for standard 3DGS on the R³LIVE "Main Building" scene. We will reproduce that scene first to validate the reimplementation. Because our method's 1×1-grid setting with the high-pass loss differs from this baseline only in its objective, a two-factor ablation is possible: field resolution (1×1 versus 4×4 versus 8×8) crossed with objective (full rendering loss versus high-pass agreement). This isolates the key ingredient.

A further baseline is DiffTex-style blending and pruning applied on top of uncorrected depth. It tests whether texture-side robustness alone suffices.

All methods share images, poses, view counts, primitive budget, texture-grid size and iteration count. Hyperparameters for each are tuned on the same held-out validation scenes. Beating Mode-GS-style calibration means a consistent gain in held-out high-pass PSNR and LPIPS across scenes and view counts. It also requires a reduction in measured edge offset, with confidence intervals across scenes excluding zero. A gain only in full-band PSNR would not count.

Why not just…?

Why not just run Metric3D v2 and texture on its output? Our error model predicts that metric depth combined with up-to-scale poses produces a global mismatch, and that the remaining local residual is untouched even after rescaling. The perturbation study measures both directly. The file reports no texture-quality number for Metric3D v2. Its low Abs Rel does not imply alignment, and we will report the rank correlation between Abs Rel and held-out high-pass error across depth models to settle it.

Why not just fit one scale and shift per view, as Mode-GS does? This is the real threat. If depth residuals after global alignment are small relative to $Z^2/(fb)$ in the scenes that matter, a 1×1 field will match ours. The field-resolution ablation decides this. We commit in advance to reporting it plainly: if 4×4 and 8×8 fields give no gain over 1×1 under the high-pass objective, the contribution reduces to the objective and the error analysis, and we will say so.

Why not just let Gaussian positions optimize freely and absorb depth errors? With dense views this probably works. LGTM's authors, however, report that sparse-view textures look correct from input views and break at novel ones, which is the signature of geometry overfitting to appearance. We sweep the number of input views (2, 4, 8, 16) with free versus field-constrained positions. We expect the constrained version to win only at low view counts.

Why not just add sparse LiDAR? LiDARTouch shows that 4 beams bring KITTI RMSE to 2.504 m against 2.112 m for 64 beams, and it also notes that thin structures between beams stay ungrounded. Those are exactly the high-parallax, high-frequency regions our objective targets. Sensors also change the problem: we address camera-only and web-capture settings. On KITTI we will include a LiDAR-anchored depth as an upper-reference condition, not a competitor.

Proposed Experiments

The perturbation study on ScanNet++ is the first decisive check. The error model is supported if the predicted offset $fb\,\delta Z/Z^2$ explains most of the variance in measured edge displacement, and if the parallax-weighted residual predicts held-out high-pass error better than Abs Rel. If Abs Rel predicts texture quality just as well, the core claim fails.

The main comparison runs on ScanNet++ and DL3DV held-out views, plus RealEstate10K, whose poses are up to scale and therefore directly test the mismatch prediction. R³LIVE tests drift-prone robot trajectories. The file reports no LGTM numbers on RealEstate10K or DL3DV, and no reported result found on Lacuna for textured splatting with monocular-depth initialization. Every number in this comparison will therefore come from our own controlled reruns. Metrics are full-band and high-pass PSNR, SSIM, LPIPS, measured edge offset in pixels, and seam-discontinuity error at primitive boundaries.

The second decisive check is the cross-depth-model ranking. If Metric3D v2 has lower Abs Rel than an aligned relative model yet loses on texture alignment under SfM poses, the "parallax, not meters" thesis is confirmed in the setting where practitioners would most naturally go wrong.