Abstract
For many fundamental scene understanding tasks, it is difficult or impossible to obtain per-pixel ground truth labels from real images. We address this challenge by introducing Hypersim, a photorealistic synthetic dataset for holistic indoor scene understanding. To create our dataset, we leverage a large repository of synthetic scenes created by professional artists, and we generate 77,400 images of 461 indoor scenes with detailed per-pixel labels and corresponding ground truth geometry. Our dataset: (1) relies exclusively on publicly available 3D assets; (2) includes complete scene geometry, material information, and lighting information for every scene; (3) includes dense per-pixel semantic instance segmentations and complete camera information for every image; and (4) factors every image into diffuse reflectance, diffuse illumination, and a non-diffuse residual term that captures view-dependent lighting effects. We analyze our dataset at the level of scenes, objects, and pixels, and we analyze costs in terms of money, computation time, and annotation effort. Remarkably, we find that it is possible to generate our entire dataset from scratch, for roughly half the cost of training a popular open-source natural language processing model. We also evaluate sim-to-real transfer performance on two real-world scene understanding tasks – semantic segmentation and 3D shape prediction – where we find that pre-training on our dataset significantly improves performance on both tasks, and achieves state-of-the-art performance on the most challenging Pix3D test set.
Introduction
A core challenge in computer vision is the acquisition of high-quality ground truth data for complex scene understanding tasks. While human annotation can provide semantic labels, it is virtually impossible for humans to accurately estimate per-pixel depth, surface normals, or complex lighting components from a 2D image. Synthetic datasets offer a promising alternative, providing perfect ground truth by construction. However, many existing synthetic datasets suffer from a "sim-to-real" gap due to lack of photorealism, simplistic geometry, or missing physical information like material properties and lighting decomposition.
Hypersim addresses these limitations by providing a massive, photorealistic dataset of 77,400 images across 461 indoor scenes. By utilizing professional architectural assets and a novel rendering pipeline, Hypersim offers a high-fidelity training environment that bridges the gap between simulation and reality.
Dataset Generation and Methodology
The creation of Hypersim involved several key innovations designed to maximize both quality and utility for researchers.
Asset Sourcing and View Sampling
Rather than using procedurally generated environments, the researchers utilized the Evermotion Archinteriors collection—high-end 3D scenes created by professional artists. To capture meaningful viewpoints within these complex scenes, a saliency model based on triangle density was employed. This heuristic assumes that detailed objects (like furniture) have higher geometric complexity than flat surfaces (like walls), allowing the system to automatically identify interesting camera positions.
Rendering and Factorization
Using a custom cloud-based V-Ray pipeline, each image is factored into three distinct components:
- Diffuse Reflectance: The intrinsic color of the material.
- Diffuse Illumination: How light interacts with the scene geometry.
- Non-diffuse Residual: Captures view-dependent effects like glossy reflections and specular highlights.
Interactive Annotation and Cost
To provide semantic labels, a custom tool allowed humans to annotate the 3D meshes directly. This approach is highly efficient: once a 3D object is labeled, that label is automatically propagated to every 2D image in which the object appears. The authors noted that generating the entire dataset cost approximately $57,000—roughly half the cost of training a state-of-the-art Natural Language Processing (NLP) model like Megatron-LM, demonstrating the cost-effectiveness of high-quality synthetic data generation.
Key Insights
- Quality over Quantity: Hypersim demonstrates that photorealism and asset quality are often more critical than dataset size. Despite being smaller than some competitors, it achieves superior performance in sim-to-real transfer.
- The Power of Factorization: The factorization of lighting into reflectance and illumination components makes Hypersim uniquely suited for inverse rendering tasks, where models must "un-light" a photo to understand the physical properties of a scene.
- Amodal Information: Because the dataset includes the full 3D mesh, it provides labels for parts of objects that are hidden behind other objects (occlusions), a capability real-world datasets cannot provide.
Experimental Results
The utility of Hypersim was validated through two primary tasks:
- Semantic Segmentation: Pre-training on Hypersim and fine-tuning on just 25% of the NYUv2 dataset outperformed training on the entire NYUv2 dataset from scratch.
- 3D Shape Prediction: Models pre-trained on Hypersim achieved state-of-the-art performance on the challenging Pix3D test set, specifically improving the
APmeshmetric, which measures the accuracy of predicted 3D geometry.
Conclusion
Hypersim represents a significant step forward in synthetic data for computer vision. By providing high-quality, photorealistic images with comprehensive geometric and lighting ground truth, it enables more robust sim-to-real transfer. Future work may involve moving beyond static scenes to interactive environments, further reducing the gap between synthetic training and real-world application.
Summary updated