Novel View Synthesis using DDIM Inversion

A. SubramanyamOpenReview 2026 · OpenReview

TL;DR

This paper introduces a novel, lightweight framework for synthesizing novel views from a single input image. By leveraging DDIM inversion and a specialized translation U-Net (TUNet), it efficiently maps a single-view latent representation to a target view's latent, then fuses this with noise information to generate high-fidelity, geometrically consistent novel views. This approach avoids expensive fine-tuning of large diffusion models and offers improved generalization.

What Problem Does This Paper Address?

Generating realistic and geometrically consistent novel views from a single input image is a long-standing challenge in computer vision and graphics. Existing methods often struggle with occluded regions, maintaining geometric consistency across viewpoints, and require extensive computational resources. Many approaches necessitate fine-tuning large diffusion models on multi-view datasets or training them from scratch, which is prohibitively expensive. Furthermore, these methods can lead to blurry reconstructions and poor generalization to unseen scenes or object categories. This paper aims to bridge this gap by proposing an efficient, lightweight framework that leverages the generative power of pre-trained diffusion models without requiring extensive retraining.

Methods

The core of the proposed method lies in operating within the DDIM-inverted latent space.

  1. DDIM Inversion: Instead of inverting an image all the way to pure noise (t=T), the process is stopped at an intermediate timestep (t=600). This yields a latent representation that retains sufficient low-frequency structure for translation while also containing valuable noise/variance information.
  2. TUNet (Translation U-Net): A lightweight U-Net architecture is trained to translate the DDIM-inverted latent of the input image (specifically, its mean component) to the latent representation of the desired target view. This translation is conditioned on camera parameters (intrinsics, extrinsics) and class embeddings, ensuring geometric consistency.
  3. Fusion Strategy: To inject high-frequency details that are often lost in the coarse translation, a novel fusion strategy is employed. This strategy combines the translated latent mean from TUNet with the high-variance noise/variance component extracted from the original DDIM-inverted input latent. Two specific strategies are proposed:
  4. Strategy A (Variance Fusion): Explicitly extracts the noise/variance component and adds it to the translated latent mean.
  5. Strategy B (Direct Noise Addition): Directly adds the noise component from the full DDIM-inverted input latent.
  6. DDIM Sampling: The fused latent is then used as the initial condition for DDIM sampling with a pre-trained diffusion model (like Stable Diffusion) to synthesize the final high-fidelity novel view.

Datasets

  • MvImgNet: Used for training and evaluating novel view synthesis on a large-scale dataset with 6.5 million frames across 238 categories. Specific subsets were used for 3-class and 167-class experiments.
  • RealEstate10K: Used for evaluating novel view synthesis, particularly for longer-range viewpoint changes.
  • Unsplash: A collection of freely-licensed photographs used to create an out-of-domain test set for evaluating zero-shot generalization.
  • LOLv1, LOLv2, SICE: Datasets used for evaluating the extension of the method to low-light image enhancement tasks.

Key Insights and Ideas

  • Intermediate DDIM Inversion is Key: Stopping DDIM inversion at an intermediate timestep (t=600) is crucial. It provides a latent that is stable enough for translation while still retaining the high-frequency noise information needed for detail reconstruction.
  • Latent Space Translation for Efficiency: Performing view translation in the latent space, rather than pixel space, significantly reduces computational cost and complexity, allowing for a lightweight translation network (TUNet).
  • Noise/Variance as a Detail Injector: The inherent noise/variance component of DDIM inversion, often discarded or treated as a nuisance, is repurposed as a powerful source of high-frequency details to enrich the synthesized novel views.
  • Generalization Beyond Training Data: The method demonstrates strong zero-shot generalization capabilities, performing well on unseen classes and out-of-domain images, highlighting the robustness of the learned latent space translation.

Key Results

  • Quantitative Performance: The proposed method outperforms existing methods like GIBR and NViST on MvImgNet and RealEstate10K in terms of LPIPS, PSNR, and SSIM. For instance, on the 167-class model, it achieves better LPIPS, PSNR, SSIM, and FID compared to NViST. On RealEstate10K, it shows competitive LPIPS and improved PSNR/SSIM over baselines, especially for long-range pairs.
  • Efficiency: The method is significantly more efficient, using a lightweight TUNet (148M parameters) and achieving faster inference times compared to methods that train entire diffusion models or operate in pixel space.
  • Qualitative Results: Visual comparisons show that the method generates geometrically consistent novel views with faithful surface textures and preserves global scene layout, even in challenging scenarios like occluded regions or out-of-domain images. The fusion strategy effectively injects fine-grained details.
  • Low-Light Enhancement: The framework is successfully extended to low-light image enhancement, demonstrating its versatility and ability to generalize to different image-to-image translation tasks.

Limitations and Future Work

  • Failure Cases: The paper acknowledges failure cases where structure can be distorted, shapes are not preserved, or textures/counts are inconsistent. These are visualized in Figure 6.
  • Out-of-Domain Camera Parameters: For out-of-domain images, the method relies on identifying visually similar scenes in MvImgNet to proxy camera parameters, which might not always be accurate.
  • Future Work: The paper suggests exploring further improvements in handling complex scene structures and textures. The extension to low-light enhancement hints at broader applications of the latent translation and fusion framework. Further research could focus on more robust camera pose estimation for out-of-domain data or exploring different fusion strategies.

Summary updated

Abstract

Synthesizing novel views from a single input image is a challenging task. It requires extrapolating the 3D structure of a scene while inferring details in occluded regions, and maintaining geometric consistency across viewpoints. Many existing methods must fine-tune large diffusion backbones using multiple views or train a diffusion model from scratch, which is extremely expensive. Additionally, they suffer from blurry reconstruction and poor generalization. This gap presents the opportunity to explore an explicit lightweight view translation framework that can directly utilize the high-fidelity generative capabilities of a pretrained diffusion model while reconstructing a scene from a novel view. Given the DDIM-inverted latent of a single input image, we employ a camera pose-conditioned translation U-Net, TUNet, to predict the inverted latent corresponding to the desired target view. However, the image sampled using the predicted latent may result in a blurry reconstruction. To this end, we propose a novel fusion strategy that exploits the inherent noise correlation structure observed in DDIM inversion. The proposed fusion strategy helps preserve the texture and fine-grained details. To synthesize the novel view, we use the fused latent as the initial condition for DDIM sampling, leveraging the generative prior of the pretrained diffusion model. Extensive experiments on MVImgNet and RealEstate10K demonstrate that our method outperforms existing methods. The code is available at https://github.com/VisualConception-Group/ddim_nvs .
BibTeX
@misc{subramanyam2026novel,
  author = {A. Subramanyam},
  title = {{Novel View Synthesis using DDIM Inversion}},
  year = {2026},
  url = {https://openreview.net/forum?id=4udcih4qUA},
  howpublished = {OpenReview},
}
Figure