Intrinsic Geometry-Appearance Consistency Optimization for Sparse-View Gaussian Splatting

Kaiqiang Xiong, Rui Peng, Jiahao Wu, Zhanke Wang, Jie Liang, Xiaoyun Zheng, Feng Gao, Ronggang Wang

TL;DR

ICO-GS (Intrinsic Geometry-Appearance Consistency Optimization) tackles the "overfitting" problem in sparse-view 3D Gaussian Splatting. By enforcing mutual reinforcement between geometry and appearance through robust multi-view feature matching and cycle-consistent virtual view synthesis, it achieves state-of-the-art reconstruction quality from as few as three images.

What Problem Does This Paper Address?

3D Gaussian Splatting (3DGS) has revolutionized real-time neural rendering, but it has a "sparse-view" Achilles' heel. When you only have a handful of images (e.g., 3 or 6) to reconstruct a scene, 3DGS suffers from a geometry-appearance discrepancy.

Because the standard optimization only cares about minimizing the difference between the rendered image and the training image, the model takes a "shortcut." It creates messy, noisy geometry (full of "floaters") and then manipulates the color and opacity of those Gaussians to make the training views look perfect. This is called appearance compensation. While the training views look great, any novel viewpoint looks like a blurry, artifact-ridden mess because the underlying 3D structure is fundamentally broken.

Methods

The authors introduce ICO-GS, a framework designed to force geometry and appearance to be "intrinsically consistent." Their approach consists of two main pillars:

  1. Robust Geometric Regularization:
  2. Feature-based Matching: Instead of comparing raw RGB pixels (which are sensitive to lighting changes), they use deep features from a pre-trained network to enforce multi-view consistency.
  3. Pixel-wise Top-k Selection: To handle occlusions (where a point is hidden in some views), the system only calculates consistency across the $k$ most similar views, effectively ignoring outliers and occluded regions.
  4. Edge-aware Smoothness: A loss term that encourages smooth depth in flat areas while preserving sharp boundaries at object edges.
  5. Geometry-guided Appearance Optimization:
  6. Cycle-Consistency Depth Filtering: To ensure they only use "good" geometry, they project depth from one view to another and back again. If the point doesn't land back where it started, the geometry is deemed unreliable and discarded.
  7. Virtual View Synthesis: They use the "reliable" depth to warp existing images into new, virtual viewpoints. This provides extra "pseudo-supervision" that forces the Gaussians to learn a view-consistent appearance rather than overfitting to a single camera.

The whole process is managed via Curriculum Learning, starting with basic 3DGS, then adding geometric constraints, and finally introducing the appearance-guided virtual views to ensure stable convergence.

Datasets

The researchers evaluated ICO-GS on three standard benchmarks:

  • LLFF: Real-world forward-facing scenes.
  • DTU: Object-centric captures, specifically chosen for its challenging "weakly-textured" surfaces (like smooth white porcelain) where geometry is hardest to recover.
  • Blender: Synthetic 360-degree scenes used to test full object reconstruction.

Key Insights and Ideas

  • The "Shortcut" is the Enemy: The most profound insight is that 3DGS is too flexible. Without enough views, the optimizer prefers "hallucinating" colors on bad geometry over finding the correct 3D shape.
  • Mutual Reinforcement: Geometry and appearance shouldn't be optimized in isolation. Accurate geometry provides the "map" for consistent appearance, and consistent appearance across views acts as a "validator" for the geometry.
  • Reliability over Quantity: Rather than using all available depth data for regularization (which might be noisy), the Cycle-Consistency Filtering ensures that only high-confidence geometric priors influence the appearance learning.

Key Results

ICO-GS consistently outperformed existing state-of-the-art sparse-view methods (like BinocularGS and FSGS):

  • Quantitative: On the 3-view DTU dataset, it achieved a +1.1 dB PSNR gain over previous methods. It also showed superior performance in SSIM (structural similarity) and LPIPS (perceptual quality).
  • Qualitative: Visual results show significantly cleaner depth maps with sharp boundaries. In regions where other methods produced "cloudy" artifacts or blurred textures (like the gaps between leaves), ICO-GS maintained crisp, faithful details.

Limitations and Future Work

  • View-Dependent Effects: The virtual view synthesis assumes a mostly "Lambertian" (matte) world. In scenes with heavy specular highlights or complex reflections, the warped appearance might provide slightly incorrect supervision.
  • Computational Overhead: Because of the extra warping and feature extraction steps, the training time is roughly 1.5x longer than the baseline BinocularGS.
  • Future Directions: The authors suggest exploring ways to better handle non-Lambertian surfaces and further optimizing the training speed to bring it closer to vanilla 3DGS.

Summary updated

Abstract

3D human reconstruction from a single image is a challenging problem and has been exclusively studied in the literature. Recently, some methods have resorted to diffusion models for guidance, optimizing a 3D representation via Score Distillation Sampling(SDS) or generating a back-view image for facilitating reconstruction. However, these methods tend to produce unsatisfactory artifacts (\textit{e.g.} flattened human structure or over-smoothing results caused by inconsistent priors from multiple views) and struggle with real-world generalization in the wild. In this work, we present \emph{MVD-HuGaS}, enabling free-view 3D human rendering from a single image via a multi-view human diffusion model. We first generate multi-view images from the single reference image with an enhanced multi-view diffusion model, which is well fine-tuned on high-quality 3D human datasets to incorporate 3D geometry priors and human structure priors. To infer accurate camera poses from the sparse generated multi-view images for reconstruction, an alignment module is introduced to facilitate joint optimization of 3D Gaussians and camera poses. Furthermore, we propose a depth-based Facial Distortion Mitigation module to refine the generated facial regions, thereby improving the overall fidelity of the reconstruction. Finally, leveraging the refined multi-view images, along with their accurate camera poses, MVD-HuGaS optimizes the 3D Gaussians of the target human for high-fidelity free-view renderings. Extensive experiments on Thuman2.0 and 2K2K datasets show that the proposed MVD-HuGaS achieves state-of-the-art performance on single-view 3D human rendering.
BibTeX
@misc{xiong2026intrinsic,
  author = {Kaiqiang Xiong and Rui Peng and Jiahao Wu and Zhanke Wang and Jie Liang and Xiaoyun Zheng and Feng Gao and Ronggang Wang},
  title = {{Intrinsic Geometry-Appearance Consistency Optimization for Sparse-View Gaussian Splatting}},
  year = {2026},
  eprint = {2603.02893},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.02893},
}
Figure