Do We Need a New Large-Scale Quality Assessment Database for Generative Inpainting Based 3D View Synthesis? (Student Abstract)

Sadbhawna, Vinit Jakhetiya, Badri N. Subudhi, Harshit Shakya, Deebha MumtazAAAI 2022 · DOI

Introduction: The Evolution of 3D View Synthesis Artifacts

Novel View Synthesis (NVS) is a cornerstone of immersive technologies like Virtual Reality (VR) and Augmented Reality (AR). Historically, creating 3D views from 2D images resulted in glaring geometric errors such as "black holes" (missing data) or "stretching" artifacts where occluded areas were exposed.

Modern generative approaches, utilizing Generative Adversarial Networks (GANs) and sophisticated inpainting, have largely eliminated these obvious gaps by "hallucinating" missing pixels. However, this progress has introduced a new class of subtle distortions: blurry textures, structural inconsistencies, and semantically incorrect "uncanny" details. As shown in the figure below, these modern artifacts are often localized and difficult for traditional metrics to detect.

Modern generative inpainting often introduces subtle structural inconsistencies and blurry textures (indicated by green arrows) that differ significantly from older geometric artifacts.

The Crisis in Quality Assessment

The research community currently faces a significant hurdle: our Image Quality Assessment (IQA) tools are optimized for a previous generation of technology. Existing 3D IQA databases, such as IVC (2011), IVY (2016), and IETR (2019), are dominated by obsolete distortions that modern state-of-the-art (SOTA) models no longer produce.

Furthermore, standard metrics like PSNR and SSIM—and even modern deep-learning metrics like LPIPS (Learned Perceptual Image Patch Similarity)—fail to align with human perception when judging these new generative artifacts. This discrepancy means researchers may be optimizing models based on metrics that do not reflect actual visual quality.

Methodology and Proposed Pipeline

To address this gap, the authors conducted a pilot study using 60 3D views generated by recent algorithms. They gathered subjective human quality ratings to establish a "ground truth" and evaluated how well automated metrics correlated with these judgments. Based on their findings, they propose a four-step roadmap for future research:

  1. Dataset Creation: Building a large-scale collection of authentic 3D views using diverse generative algorithms.
  2. Subjective Testing: Implementing large-scale human judgments using Elo rating systems to provide robust perceptual scores.
  3. Deep Learning Model: Developing a CNN-based metric designed specifically for 3D views, incorporating "domain-shift awareness" to handle various synthesis methods.
  4. Enhancement: Utilizing the new metric to drive 3D image restoration and synthesis improvement.

Key Results: The Failure of Current Metrics

The pilot study revealed a startling lack of correlation between popular metrics and human judgment. While LOGS (a 3D-specific metric) performed the best, its correlation remained moderate. Most notably, LPIPS—often considered the gold standard for perceptual similarity—almost entirely failed to rank the images correctly, showing a rank correlation (SRCC) near zero.

Quantitative comparison of metrics against human subjective scores. Note the extremely low SRCC for LPIPS and APT, highlighting their inability to rank generative 3D synthesis quality.

Conclusion

The transition from geometric projection to generative inpainting in 3D synthesis has rendered existing quality databases and metrics obsolete. The failure of even "perceptual" metrics like LPIPS suggests that 3D synthesis artifacts require a specialized understanding of scene semantics and structural consistency. There is an urgent need for a new, large-scale database and a dedicated metric to guide the next generation of 3D view synthesis research.

Summary updated

BibTeX
@inproceedings{sadbhawna2022do,
  author = {Sadbhawna and Vinit Jakhetiya and Badri N. Subudhi and Harshit Shakya and Deebha Mumtaz},
  title = {{Do We Need a New Large-Scale Quality Assessment Database for Generative Inpainting Based 3D View Synthesis? (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i11.21656},
}
Figure