Introduction: The Evolution of 3D View Synthesis Artifacts
Novel View Synthesis (NVS) is a cornerstone of immersive technologies like Virtual Reality (VR) and Augmented Reality (AR). Historically, creating 3D views from 2D images resulted in glaring geometric errors such as "black holes" (missing data) or "stretching" artifacts where occluded areas were exposed.
Modern generative approaches, utilizing Generative Adversarial Networks (GANs) and sophisticated inpainting, have largely eliminated these obvious gaps by "hallucinating" missing pixels. However, this progress has introduced a new class of subtle distortions: blurry textures, structural inconsistencies, and semantically incorrect "uncanny" details. As shown in the figure below, these modern artifacts are often localized and difficult for traditional metrics to detect.
The Crisis in Quality Assessment
The research community currently faces a significant hurdle: our Image Quality Assessment (IQA) tools are optimized for a previous generation of technology. Existing 3D IQA databases, such as IVC (2011), IVY (2016), and IETR (2019), are dominated by obsolete distortions that modern state-of-the-art (SOTA) models no longer produce.
Furthermore, standard metrics like PSNR and SSIM—and even modern deep-learning metrics like LPIPS (Learned Perceptual Image Patch Similarity)—fail to align with human perception when judging these new generative artifacts. This discrepancy means researchers may be optimizing models based on metrics that do not reflect actual visual quality.
Methodology and Proposed Pipeline
To address this gap, the authors conducted a pilot study using 60 3D views generated by recent algorithms. They gathered subjective human quality ratings to establish a "ground truth" and evaluated how well automated metrics correlated with these judgments. Based on their findings, they propose a four-step roadmap for future research:
- Dataset Creation: Building a large-scale collection of authentic 3D views using diverse generative algorithms.
- Subjective Testing: Implementing large-scale human judgments using Elo rating systems to provide robust perceptual scores.
- Deep Learning Model: Developing a CNN-based metric designed specifically for 3D views, incorporating "domain-shift awareness" to handle various synthesis methods.
- Enhancement: Utilizing the new metric to drive 3D image restoration and synthesis improvement.
Key Results: The Failure of Current Metrics
The pilot study revealed a startling lack of correlation between popular metrics and human judgment. While LOGS (a 3D-specific metric) performed the best, its correlation remained moderate. Most notably, LPIPS—often considered the gold standard for perceptual similarity—almost entirely failed to rank the images correctly, showing a rank correlation (SRCC) near zero.
Conclusion
The transition from geometric projection to generative inpainting in 3D synthesis has rendered existing quality databases and metrics obsolete. The failure of even "perceptual" metrics like LPIPS suggests that 3D synthesis artifacts require a specialized understanding of scene semantics and structural consistency. There is an urgent need for a new, large-scale database and a dedicated metric to guide the next generation of 3D view synthesis research.
Summary updated