ATISS: Autoregressive Transformers for Indoor Scene Synthesis

Despoina Paschalidou, Amlan Kar, Maria Shugrina, Karsten Kreis, Andreas Geiger, Sanja Fidler

Introduction

The creation of realistic 3D indoor environments is a fundamental task in computer graphics, with applications ranging from virtual reality and gaming to the generation of synthetic data for robotic training. Traditionally, this process has relied on manual labor by artists or procedural modeling based on rigid, hand-crafted rules. Recent advancements in deep learning have attempted to automate scene synthesis by treating it as a sequence generation problem, akin to natural language processing. However, these sequential models often struggle with the inherent lack of order in physical spaces.

The Limitations of Sequential Scene Synthesis

Existing autoregressive models typically generate scenes by placing objects in a fixed sequence—for instance, ordering items by size or frequency. While effective for language, where word order is paramount, this approach is fundamentally flawed for spatial layouts. A model trained to place a bed before a lamp may fail to suggest a bed if a lamp is already present in the scene. This brittleness limits the utility of such models for interactive applications like scene completion or editing, where the starting state is unpredictable.

ATISS: A Set-Based Approach

ATISS (Autoregressive Transformers for Indoor Scene Synthesis) addresses these limitations by treating a room as an unordered set of objects rather than a sequence. By formulating the problem through the lens of set generation, the model learns the underlying spatial relationships and functional constraints of a room without being tied to a specific construction order.

Architecture and Methodology

The ATISS architecture is built around a permutation-invariant Transformer. The process involves several key components:

  1. Input Encoding: The model receives a 2D floor plan, encoded via a ResNet-18, and a set of existing objects. Each object is represented by its category, 3D dimensions, location, and orientation.
  2. Permutation Invariance: Unlike standard Transformers used in NLP, ATISS omits positional encodings. This ensures that the model treats the input objects as a collection where the relative arrangement matters more than the input order.
  3. The Query Mechanism: A learnable "query token" is introduced to the set. The Transformer processes the floor plan and existing objects to produce a feature vector for this query, which effectively asks, "What object is missing from this context?"
  4. Attribute Prediction: A series of Multi-Layer Perceptrons (MLPs) then predict the attributes of the new object in a structured chain: Category → Location → Orientation → Size.
  5. Training Strategy: During training, the model is exposed to scenes in random permutations. This forces the network to learn how to complete a room regardless of which objects are already present, making it a robust "generalist" for scene synthesis.

Experimental Setup

The researchers evaluated ATISS using the 3D-FRONT dataset, which contains over 6,000 professionally designed houses, and the 3D-FUTURE dataset for high-quality furniture models. The study focused on four common room types: Bedrooms, Living Rooms, Dining Rooms, and Libraries.

Results and Performance

ATISS demonstrates significant improvements over prior state-of-the-art methods, such as SceneFormer, in terms of both quality and efficiency.

  • Inference Speed: ATISS is approximately 8x faster than SceneFormer. This speedup is crucial for interactive design tools where real-time feedback is required.
  • Visual Realism: In perceptual studies, human participants preferred ATISS-generated layouts over baseline models 73.1% of the time.
  • Distributional Accuracy: The model achieved substantially lower Frechet Inception Distance (FID) scores. In Living Room environments, ATISS recorded an FID of 33.14, a marked improvement over SceneFormer’s 69.54.
  • Discriminator Performance: A "real vs. synthetic" classifier struggled to distinguish ATISS scenes from human-designed ones, maintaining an accuracy of approximately 50%, indicating that the generated layouts are statistically similar to real-world data.

Applications and Future Work

Beyond fully automatic synthesis, the set-based nature of ATISS enables several interactive use cases:

  • Scene Completion: Users can place a few anchor objects, and the model can suggest the remaining furniture.
  • Room Cleaning: The model can identify and relocate objects that are statistically out of place.

Despite its success, the authors identify areas for further research. While the objects are treated as a set, the internal attributes of each object are still predicted sequentially. Additionally, the current model does not explicitly account for stylistic consistency (e.g., matching furniture styles), and the retrieval of specific 3D geometries remains a separate post-processing step. Future iterations may integrate style-aware retrieval or direct geometry generation to create even more cohesive environments.

Conclusion

ATISS represents a paradigm shift in indoor scene synthesis. By rethinking rooms as sets, the model achieves superior performance, faster inference, and greater flexibility than previous sequential approaches, paving the way for more intuitive and powerful AI-assisted design tools.

Summary updated

BibTeX
@inproceedings{paschalidou2021atiss,
  author = {Despoina Paschalidou and Amlan Kar and Maria Shugrina and Karsten Kreis and Andreas Geiger and Sanja Fidler},
  title = {{ATISS: Autoregressive Transformers for Indoor Scene Synthesis}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
}
Figure