TL;DR
AffineGS addresses a fundamental flaw in how 3D Gaussian Splatting (3DGS) uses AI-generated depth priors: the mistaken assumption that these priors can be aligned to a scene using a single global transform. By introducing a spatially varying affine rectification model, AffineGS corrects complex, region-dependent distortions in monocular depth, enabling the reconstruction of highly accurate 3D geometry from RGB images alone—without the need for expensive depth sensors.
What Problem Does This Paper Address?
3D Gaussian Splatting (3DGS) has revolutionized real-time photorealistic rendering, but it has a "dirty secret": its underlying geometry is often a mess. Because 3DGS focuses on matching pixels from specific viewpoints, it often creates "floaters," multi-layered surfaces, and jagged geometry that looks fine from the training views but falls apart when you try to extract a 3D mesh for applications like robotics or VR.
To fix this, researchers often use "monocular depth priors"—depth maps predicted by foundation AI models like Depth-Anything. Standard practice is to align these predictions to the 3DGS scene using a global "scale-and-shift" (affine) transform.
The authors of AffineGS discovered that this assumption is fundamentally broken. Modern depth models exhibit piecewise-linear distortions: they might get the scale right for a chair in the foreground but be completely off for the wall behind it. Applying one global transform to the whole image inevitably warps the geometry of some objects to satisfy others.
Methods
The core of AffineGS is a move from global constants to spatially varying fields. Instead of one $\alpha$ (scale) and $\beta$ (shift) for the whole image, the model estimates a unique $\alpha(u,v)$ and $\beta(u,v)$ for every pixel.
- Spatially Varying Affine Rectification: The authors model the relationship between predicted disparity and true metric depth locally. They solve for the local scale factor by comparing the spatial gradients of the disparity against the rendered 3DGS depth. This "gradient-domain" approach is clever because it isolates the scale from the "shift" (bias) ambiguities.
- Edge-Aware Guided Filtering: To ensure these correction fields are smooth within an object but can change abruptly at object boundaries, they use a Guided Filter. This prevents the math from "blurring" the geometry across sharp edges.
- Normal-Aware Regularization: They derive a new loss function that ties surface normals directly to depth gradients using projective camera geometry. This forces the individual Gaussian primitives to align their physical orientation with the actual slope of the scene's surfaces.
Datasets
The authors evaluated their method on three diverse datasets to prove its robustness:
- MuSHRoom: A multi-sensor hybrid room dataset specifically designed for evaluating 3D reconstruction and novel view synthesis.
- ScanNet++: A large-scale indoor dataset providing high-fidelity 3D ground truth and dense RGB sequences.
- NeRF-LLFF: A standard dataset used here to test "sparse-view" scenarios (reconstructing a scene from as few as 2 to 5 images).
Key Insights and Ideas
- The "Aha" Moment on Fragmentation: The paper statistically proves that monocular depth models "fracture" scenes into distinct geometric clusters. While a global transform leaves 40% of a scene misaligned, partitioning it into just 5 local regions jumps the alignment accuracy to over 91%.
- Gradients Over Values: By supervising the change in depth (gradients) rather than absolute depth values, the model becomes much more robust to the systematic biases inherent in monocular depth estimators.
- Coupling Normals and Depth: Most 3DGS methods treat normals and depth as separate problems. AffineGS uses the math of pinhole cameras to prove they are two sides of the same coin, ensuring the "tilt" of the Gaussians matches the "slope" of the depth map.
Key Results
AffineGS significantly bridges the gap between "sensor-free" methods and those that require expensive LiDAR data.
- Geometric Accuracy: On the MuSHRoom dataset, AffineGS achieved an F-score (a measure of mesh quality) of 0.7097, nearly tripling the performance of the baseline DN-Splatter (0.2765) when both were denied access to real depth sensors.
- Sparse-View Robustness: In extreme cases with only 2 training views, AffineGS maintained a mean PSNR of 18.79 dB, outperforming the next best baseline (DRGS) by nearly 1.5 dB and preventing the "structural collapse" common in other methods.
- Qualitative Superiority: As seen in Figure 4 and 5, AffineGS recovers crisp, planar surfaces (like flat walls) and sharp object boundaries that previous methods turned into "rippled" or "melted" artifacts.
Limitations and Future Work
- Foundation Model Ceiling: The method's quality is still bounded by the underlying monocular depth model. If the AI model fails to see a thin structure or a mirror, AffineGS cannot "invent" that geometry.
- Absolute Scale: While the geometry is internally consistent and accurate, it still lacks an absolute real-world scale (e.g., knowing exactly how many meters wide a room is) without external reference.
- Hyperparameter Tuning: The authors noted that some scenes (like very dark or "floater"-prone environments) require scaling the loss weights, suggesting a need for more automated weight balancing in the future.
Summary updated