PlaceNet: Neural Spatial Representation Learning with Multimodal Attention

Chung-Yeon Lee, Youngjae Yoo, Byoung-Tak ZhangIJCAI 2022 · DOI

Introduction

For mobile AI agents, such as autonomous vacuums or delivery robots, navigating a 3D environment requires more than just visual recognition; it necessitates a deep understanding of underlying spatial structures. Traditional approaches often rely on metric maps—grids of obstacles and landmarks—which can be brittle and difficult to scale. While recent neural rendering techniques like Generative Query Networks (GQN) or Neural Radiance Fields (NeRF) attempt to learn 3D representations by predicting novel viewpoints, they frequently struggle to generalize across diverse, large-scale environments or focus too narrowly on single-scene reconstruction. PlaceNet addresses these limitations by learning a rich, generalizable representation of indoor spaces through the integration of multimodal information.

Methods and Framework

PlaceNet is a generative model designed to learn spatial representations in a self-supervised manner. The core of the architecture is the Multimodal Triplet Attention mechanism, which enables the model to synthesize information from three distinct data streams:

  1. Visual: Standard RGB imagery.
  2. Topographic: Depth maps providing the structural "skeleton" of the scene.
  3. Semantic: Object-aware segmentation masks that identify specific entities (e.g., furniture, windows).
Overview of the PlaceNet framework, illustrating the integration of multimodal observations (RGB, Depth, Object masks) into attention contexts for scene generation.

The model utilizes a Query-Key-Value (QKV) attention structure. During the generation of a new viewpoint, the model's internal state acts as a query to retrieve relevant visual, structural, or semantic cues from the multimodal observations. A Convolutional LSTM (CLSTM) then progressively refines the predicted image, building the spatial representation layer by layer. To train this system, the authors utilized a massive dataset of approximately 120 million indoor scene images from the House3D simulator, categorized by complexity levels.

Experimental Results

PlaceNet demonstrates significant improvements over baseline models in both fidelity and generalization. Quantitative analysis shows that PlaceNet achieves a Structural Similarity (SSIM) score exceeding 0.6, notably higher than the GQN baseline of approximately 0.53.

Quantitative comparison of training metrics and SSIM between PlaceNet and GQN baselines.

Key findings include:

  • Multimodal Synergy: Integrating depth and semantics fundamentally improves internal knowledge organization, making the model robust to lighting and texture variations.
  • Generalization: The model performs nearly as well on entirely unseen house layouts as it does on training data, suggesting it has learned a general "grammar" of indoor architecture rather than merely memorizing specific scenes.
  • Sim-to-Real Transfer: PlaceNet successfully generated plausible views when tested on real-world data from a Toyota HSR mobile robot, despite being trained exclusively on simulated environments.

Conclusion and Future Work

PlaceNet establishes a powerful framework for neural spatial representation by leveraging multimodal attention. While the model currently depends on external semantic masks and operates at a 64x64 resolution, it provides a robust foundation for generalizable 3D understanding. Future research will focus on scaling to higher resolutions and integrating these learned representations into active tasks such as autonomous navigation and real-time path planning.

Summary updated

BibTeX
@inproceedings{lee2022placenet,
  author = {Chung-Yeon Lee and Youngjae Yoo and Byoung-Tak Zhang},
  title = {{PlaceNet: Neural Spatial Representation Learning with Multimodal Attention}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.24963/ijcai.2022/144},
}
Figure