GenWarp: Single Image to Novel Views with Semantic-Preserving Generative Warping

Junyoung Seo, Kazumi Fukuda, Takashi Shibuya, Takuya Narihira, Naoki Murata, Shoukang Hu, Chieh-Hsin Lai, Seungryong Kim, Yuki Mitsufuji

Abstract

Generating consistent novel views from a single 2D image is a fundamental challenge in computer vision due to the ill-posed nature of predicting unseen geometry and textures. Traditional methods often rely on a disjointed pipeline of depth estimation followed by inpainting, which frequently results in geometric artifacts and semantic drift. This paper introduces GenWarp, a novel framework that integrates geometric warping directly into the attention mechanisms of a diffusion model. By utilizing a two-stream architecture and warped coordinate embeddings, GenWarp preserves high-resolution semantic details while hallucinating occluded regions, outperforming existing baselines in both image quality and geometric fidelity.

1. Introduction

The ability to synthesize new perspectives of a scene from a single photograph—novel view synthesis (NVS)—has applications ranging from virtual reality to 3D content creation. However, this task is inherently difficult because it requires a model to understand the 3D structure of a scene and plausibly fill in "holes" created by camera movement.

Existing state-of-the-art approaches typically employ a two-step "warping-and-inpainting" strategy. First, a Monocular Depth Estimation (MDE) model predicts the depth of pixels to project them into a new camera coordinate system. Second, a generative model, such as Stable Diffusion, fills in the resulting occlusions. This pipeline suffers from two primary failure modes:

  1. Noisy Depth Sensitivity: Small errors in depth estimation lead to "splattered" pixels, which the inpainting model often fails to interpret correctly, resulting in blurry or distorted outputs.
  2. Semantic Loss: During the explicit warping process, fine-grained details (e.g., text, specific textures) are often mangled. The inpainting model, lacking a direct link to the original image's features, may generate plausible but contextually inaccurate replacements.

2. Methodology

GenWarp moves away from explicit pixel-level warping in favor of Semantic-Preserving Generative Warping, a framework that performs warping and generation within a unified latent space.

2.1 Two-Stream Architecture

The framework utilizes two parallel U-Nets: a Semantic Preserver and a Diffusion Model. The Semantic Preserver processes the source image to extract high-resolution features, while the Diffusion Model generates the target novel view. This dual-stream approach ensures that the generation process is constantly conditioned on the original image's features, preventing the loss of identity and texture common in traditional methods.

2.2 Warped Coordinate Embedding

To provide the model with a geometric prior, GenWarp uses 2D coordinate maps. A grid of coordinates from the source image is transformed into the target view's perspective using estimated depth. These warped coordinates are then fed into the model as a conditioning signal. This informs the diffusion process of the spatial relationship between the source and target, essentially mapping where source pixels should influence the target generation.

2.3 Augmented Self-Attention

The core innovation of GenWarp is its augmented attention mechanism. In standard diffusion models, self-attention allows pixels within a single image to interact. GenWarp extends this by concatenating features from the source image into the attention layers:

  • Self-attention facilitates the generation of new content (inpainting) by leveraging generative priors.
  • Cross-view attention enables the model to "pull" details directly from the source image (warping).

By performing these operations simultaneously, the model learns to balance the preservation of original details with the hallucination of new, consistent geometry.

3. Experiments and Results

3.1 Datasets and Training

GenWarp was trained on RealEstate10K, a large-scale dataset of indoor and outdoor walkthroughs. To test its generalization capabilities, the model was evaluated on ScanNet (indoor RGB-D scans) and ACID (outdoor scenes). Furthermore, the model was tested on "in-the-wild" images, including AI-generated content and internet photography, to demonstrate its robustness beyond specific training distributions.

3.2 Quantitative Performance

GenWarp demonstrates significant improvements over baselines such as GeoGPT and standard Stable Diffusion Inpainting. On the ScanNet out-of-domain benchmark, GenWarp achieved:

  • FID (Fréchet Inception Distance): 46.03 (compared to 52.20 for the leading competitor), indicating superior image realism.
  • PSNR (Peak Signal-to-Noise Ratio): 12.95, showing higher mathematical reconstruction accuracy relative to ground truth views.

3.3 Qualitative Analysis

Visual evaluations reveal that GenWarp excels at maintaining structural integrity. While baseline models often produce "melted" geometry or inconsistent lighting when the camera moves significantly, GenWarp maintains sharp textures and straight lines. It is particularly effective at preserving the "vibe" of a scene—ensuring that unique objects, like a specific chair or sign, remain recognizable across different viewpoints.

4. Discussion and Limitations

While GenWarp represents a significant step forward, it faces challenges in extreme scenarios. When the target viewpoint has zero overlap with the source image, the warping signal vanishes, and the model relies entirely on its generative prior, which can lead to generic results. Additionally, while the model can generate sequences of views, doing so sequentially can lead to error accumulation over time. Future work will likely focus on improving performance for long-range camera trajectories and leveraging higher-quality 3D datasets as they become available.

5. Conclusion

GenWarp demonstrates that integrating geometric awareness into the attention layers of a diffusion model is a powerful alternative to explicit warping. By treating warping as a semantic-preserving generative task, the framework achieves state-of-the-art results in single-image novel view synthesis, offering a robust solution for maintaining consistency in complex, real-world scenes.

Summary updated

BibTeX
@inproceedings{seo2024genwarp,
  author = {Junyoung Seo and Kazumi Fukuda and Takashi Shibuya and Takuya Narihira and Naoki Murata and Shoukang Hu and Chieh-Hsin Lai and Seungryong Kim and Yuki Mitsufuji},
  title = {{GenWarp: Single Image to Novel Views with Semantic-Preserving Generative Warping}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=rLJisJmMKw},
}
Figure