Abstract
Worldsheet is a novel approach to single-image view synthesis that "shrink-wraps" a deformable mesh sheet onto an input image to capture its 3D structure. By combining a differentiable texture sampler with end-to-end training—requiring only 2D images and no 3D supervision—it generates high-resolution, photorealistic 3D "pop-ups" that significantly outperform previous point-cloud and multi-plane image methods.
Introduction
Generating a 3D "novel view" from a single 2D image is a fundamental challenge in computer vision. While humans intuitively infer 3D structure from perspective and prior knowledge, machines struggle to reconstruct depth and geometry from a single viewpoint. Existing approaches typically utilize Multi-Plane Images (MPIs), which fail under large rotations, or Point Clouds, which often result in sparse, "holey" renderings at high resolutions.
Traditional mesh-based methods offer smooth surfaces but have historically been limited to specific object categories with consistent "mean shapes." Worldsheet addresses these limitations by treating the 3D world as a flexible sheet that can be deformed to fit arbitrary scene geometry without category-specific templates or ground-truth 3D data.
Methodology
The Worldsheet framework operates on the principle of "shrink-wrapping" a flat grid onto the scene's geometry.
The "Sheet" Warp
The model utilizes a ResNet-50 backbone to predict 2D offsets ($\Delta x, \Delta y$) and depth ($z$) for every vertex on a lattice grid. This allows the grid to adapt its topology to the objects present in the input image.
Differentiable Texture Sampler
A key contribution is the analytic texture sampler. Instead of generating textures from scratch, Worldsheet "splats" original pixels onto the mesh's UV map. This process is fully differentiable, enabling the model to learn geometry by minimizing the difference between a rendered view and a target view during training.
Inpainting and Multi-layering
To handle occlusions revealed by camera movement, a GAN-based inpainting network (pix2pixHD) fills in missing textures. For complex scenes with overlapping objects, the authors propose stacking multiple Worldsheets with predicted transparency channels, enhancing parallax and occlusion handling.
Results
Worldsheet was evaluated on benchmarks including Matterport3D, Replica, and RealEstate10K. It demonstrates significant improvements in both quantitative metrics and visual quality.
On the RealEstate10K dataset, Worldsheet achieved a PSNR of 26.74, outperforming the previous state-of-the-art, SynSin (22.83). Notably, the method scales to high resolutions (up to 960x960), preserving fine details and maintaining geometric consistency where point-cloud methods often fail.
Conclusion
Worldsheet provides a robust, scene-agnostic method for 3D reconstruction from a single image. By leveraging the efficiency of meshes and the fidelity of analytic texture sampling, it produces high-quality novel views. While limitations remain regarding complex topologies and extreme rotations, the approach sets a new standard for single-image view synthesis. Future work may explore adaptive mesh resolutions to better capture fine structural details.
Summary updated