AffineGS: Unlocking Accurate Geometry in 3DGS via Spatially Varying Affine Rectification

Tze Ho Elden Tse, Jizong Peng, Richard Chen, Angela YaoarXiv 2026 · arXiv

TL;DR

AffineGS addresses a fundamental flaw in how 3D Gaussian Splatting (3DGS) uses AI-generated depth priors: the mistaken assumption that these priors can be aligned to a scene using a single global transform. By introducing a spatially varying affine rectification model, AffineGS corrects complex, region-dependent distortions in monocular depth, enabling the reconstruction of highly accurate 3D geometry from RGB images alone—without the need for expensive depth sensors.

What Problem Does This Paper Address?

3D Gaussian Splatting (3DGS) has revolutionized real-time photorealistic rendering, but it has a "dirty secret": its underlying geometry is often a mess. Because 3DGS focuses on matching pixels from specific viewpoints, it often creates "floaters," multi-layered surfaces, and jagged geometry that looks fine from the training views but falls apart when you try to extract a 3D mesh for applications like robotics or VR.

To fix this, researchers often use "monocular depth priors"—depth maps predicted by foundation AI models like Depth-Anything. Standard practice is to align these predictions to the 3DGS scene using a global "scale-and-shift" (affine) transform.

The authors of AffineGS discovered that this assumption is fundamentally broken. Modern depth models exhibit piecewise-linear distortions: they might get the scale right for a chair in the foreground but be completely off for the wall behind it. Applying one global transform to the whole image inevitably warps the geometry of some objects to satisfy others.

Figure 1: Standard 3DGS (left) overfits appearance, creating noisy geometry. AffineGS (right) identifies that monocular depth is fragmented into different "regimes" (d) and corrects them spatially (e) to produce clean meshes.

Methods

The core of AffineGS is a move from global constants to spatially varying fields. Instead of one $\alpha$ (scale) and $\beta$ (shift) for the whole image, the model estimates a unique $\alpha(u,v)$ and $\beta(u,v)$ for every pixel.

  1. Spatially Varying Affine Rectification: The authors model the relationship between predicted disparity and true metric depth locally. They solve for the local scale factor by comparing the spatial gradients of the disparity against the rendered 3DGS depth. This "gradient-domain" approach is clever because it isolates the scale from the "shift" (bias) ambiguities.
  2. Edge-Aware Guided Filtering: To ensure these correction fields are smooth within an object but can change abruptly at object boundaries, they use a Guided Filter. This prevents the math from "blurring" the geometry across sharp edges.
  3. Normal-Aware Regularization: They derive a new loss function that ties surface normals directly to depth gradients using projective camera geometry. This forces the individual Gaussian primitives to align their physical orientation with the actual slope of the scene's surfaces.
Figure 2: The AffineGS pipeline. (a) Rendering depth and normals; (b) Estimating and filtering the spatially varying affine fields; (c) Applying gradient and normal-aware supervision.

Datasets

The authors evaluated their method on three diverse datasets to prove its robustness:

  • MuSHRoom: A multi-sensor hybrid room dataset specifically designed for evaluating 3D reconstruction and novel view synthesis.
  • ScanNet++: A large-scale indoor dataset providing high-fidelity 3D ground truth and dense RGB sequences.
  • NeRF-LLFF: A standard dataset used here to test "sparse-view" scenarios (reconstructing a scene from as few as 2 to 5 images).

Key Insights and Ideas

  • The "Aha" Moment on Fragmentation: The paper statistically proves that monocular depth models "fracture" scenes into distinct geometric clusters. While a global transform leaves 40% of a scene misaligned, partitioning it into just 5 local regions jumps the alignment accuracy to over 91%.
  • Gradients Over Values: By supervising the change in depth (gradients) rather than absolute depth values, the model becomes much more robust to the systematic biases inherent in monocular depth estimators.
  • Coupling Normals and Depth: Most 3DGS methods treat normals and depth as separate problems. AffineGS uses the math of pinhole cameras to prove they are two sides of the same coin, ensuring the "tilt" of the Gaussians matches the "slope" of the depth map.

Key Results

AffineGS significantly bridges the gap between "sensor-free" methods and those that require expensive LiDAR data.

  • Geometric Accuracy: On the MuSHRoom dataset, AffineGS achieved an F-score (a measure of mesh quality) of 0.7097, nearly tripling the performance of the baseline DN-Splatter (0.2765) when both were denied access to real depth sensors.
  • Sparse-View Robustness: In extreme cases with only 2 training views, AffineGS maintained a mean PSNR of 18.79 dB, outperforming the next best baseline (DRGS) by nearly 1.5 dB and preventing the "structural collapse" common in other methods.
  • Qualitative Superiority: As seen in Figure 4 and 5, AffineGS recovers crisp, planar surfaces (like flat walls) and sharp object boundaries that previous methods turned into "rippled" or "melted" artifacts.
Figure 4: Qualitative comparison. Compared to globally aligned methods (a), AffineGS (c) recovers sharp planar surfaces and complex boundaries that match the quality of sensor-supervised methods (b).

Limitations and Future Work

  • Foundation Model Ceiling: The method's quality is still bounded by the underlying monocular depth model. If the AI model fails to see a thin structure or a mirror, AffineGS cannot "invent" that geometry.
  • Absolute Scale: While the geometry is internally consistent and accurate, it still lacks an absolute real-world scale (e.g., knowing exactly how many meters wide a room is) without external reference.
  • Hyperparameter Tuning: The authors noted that some scenes (like very dark or "floater"-prone environments) require scaling the loss weights, suggesting a need for more automated weight balancing in the future.

Summary updated

Abstract

3D Gaussian Splatting (3DGS) has become the leading approach for photorealistic novel view synthesis, yet its geometric accuracy lags behind its visual quality. Existing depth-guided methods attempt to resolve this by regularizing the optimization with monocular depth priors, typically using a single global scale-and-shift alignment. Crucially, we discover a fundamental flaw in this assumption: modern monocular depth networks systematically exhibit region-dependent, piecewise-linear distortions. This discovery invalidates standard global alignment strategies, which warp scene geometry by enforcing a single transform. To address this, we introduce a spatially varying affine rectification model that corrects monocular disparity into consistent metric depth using regionally smooth, edge-aware fields. We further derive a normal-aware regularizer that uses spatial gradients to couple the rectified disparity with 3DGS depth and normals. Across diverse experimental settings, our sensor-free method shows strong photometric quality of Gaussian Splatting while substantially improving geometric accuracy and outperforming state-of-the art geometry-focused methods.
BibTeX
@misc{tse2026affinegs,
  author = {Tze Ho Elden Tse and Jizong Peng and Richard Chen and Angela Yao},
  title = {{AffineGS: Unlocking Accurate Geometry in 3DGS via Spatially Varying Affine Rectification}},
  year = {2026},
  eprint = {2611.07709},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.07709},
}
Figure