Worldsheet: Wrapping the World in a 3D Sheet for View Synthesis from a Single Image

Ronghang Hu, Nikhila Ravi, Alexander C. Berg, Deepak Pathak

Abstract

Worldsheet is a novel approach to single-image view synthesis that "shrink-wraps" a deformable mesh sheet onto an input image to capture its 3D structure. By combining a differentiable texture sampler with end-to-end training—requiring only 2D images and no 3D supervision—it generates high-resolution, photorealistic 3D "pop-ups" that significantly outperform previous point-cloud and multi-plane image methods.

Introduction

Generating a 3D "novel view" from a single 2D image is a fundamental challenge in computer vision. While humans intuitively infer 3D structure from perspective and prior knowledge, machines struggle to reconstruct depth and geometry from a single viewpoint. Existing approaches typically utilize Multi-Plane Images (MPIs), which fail under large rotations, or Point Clouds, which often result in sparse, "holey" renderings at high resolutions.

Input image of a courtyard used for single-image view synthesis.

Traditional mesh-based methods offer smooth surfaces but have historically been limited to specific object categories with consistent "mean shapes." Worldsheet addresses these limitations by treating the 3D world as a flexible sheet that can be deformed to fit arbitrary scene geometry without category-specific templates or ground-truth 3D data.

Methodology

The Worldsheet framework operates on the principle of "shrink-wrapping" a flat grid onto the scene's geometry.

The "Sheet" Warp

The model utilizes a ResNet-50 backbone to predict 2D offsets ($\Delta x, \Delta y$) and depth ($z$) for every vertex on a lattice grid. This allows the grid to adapt its topology to the objects present in the input image.

Differentiable Texture Sampler

A key contribution is the analytic texture sampler. Instead of generating textures from scratch, Worldsheet "splats" original pixels onto the mesh's UV map. This process is fully differentiable, enabling the model to learn geometry by minimizing the difference between a rendered view and a target view during training.

Inpainting and Multi-layering

To handle occlusions revealed by camera movement, a GAN-based inpainting network (pix2pixHD) fills in missing textures. For complex scenes with overlapping objects, the authors propose stacking multiple Worldsheets with predicted transparency channels, enhancing parallax and occlusion handling.

Results

Worldsheet was evaluated on benchmarks including Matterport3D, Replica, and RealEstate10K. It demonstrates significant improvements in both quantitative metrics and visual quality.

Synthesized novel view of the courtyard, showing the 3D "pop-up" effect.

On the RealEstate10K dataset, Worldsheet achieved a PSNR of 26.74, outperforming the previous state-of-the-art, SynSin (22.83). Notably, the method scales to high resolutions (up to 960x960), preserving fine details and maintaining geometric consistency where point-cloud methods often fail.

Conclusion

Worldsheet provides a robust, scene-agnostic method for 3D reconstruction from a single image. By leveraging the efficiency of meshes and the fidelity of analytic texture sampling, it produces high-quality novel views. While limitations remain regarding complex topologies and extreme rotations, the approach sets a new standard for single-image view synthesis. Future work may explore adaptive mesh resolutions to better capture fine structural details.

Summary updated

Abstract

We present Worldsheet, a method for novel view synthesis using just a single RGB image as input. The main insight is that simply shrink-wrapping a planar mesh sheet onto the input image, consistent with the learned intermediate depth, captures underlying geometry sufficient to generate photorealistic unseen views with large viewpoint changes. To operationalize this, we propose a novel differentiable texture sampler that allows our wrapped mesh sheet to be textured and rendered differentiably into an image from a target viewpoint. Our approach is category-agnostic, end-to-end trainable without using any 3D supervision, and requires a single image at test time. We also explore a simple extension by stacking multiple layers of Worldsheets to better handle occlusions. Worldsheet consistently outperforms prior state-of-the-art methods on single-image view synthesis across several datasets. Furthermore, this simple idea captures novel views surprisingly well on a wide range of high-resolution in-the-wild images, converting them into navigable 3D pop-ups. Video results and code are available at https://worldsheet.github.io.
BibTeX
@inproceedings{hu2020worldsheet,
  author = {Ronghang Hu and Nikhila Ravi and Alexander C. Berg and Deepak Pathak},
  title = {{Worldsheet: Wrapping the World in a 3D Sheet for View Synthesis from a Single Image}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2020},
  doi = {10.1109/iccv48922.2021.01230},
}
Figure