LaLaLoc: Latent Layout Localisation in Dynamic, Unvisited Environments

Henry Howard-Jenkins, José-Raúl Ruiz-Sarmiento, Victor Adrian Prisacariu

Abstract

Camera relocalization in unknown environments remains a significant challenge, typically requiring prior mapping or static scenes. We introduce LaLaLoc, a novel framework designed for "zero-shot" localization that utilizes only a 2D floor plan. By establishing a shared latent space between cluttered RGB panoramas and clean geometric layouts, LaLaLoc achieves high-precision localization (sub-10cm) even in dynamic, fully furnished spaces. Our approach leverages a differentiable renderer for latent pose optimization, bypassing the need for explicit 3D reconstruction.

Introduction

Traditional camera relocalization systems often fail in real-world scenarios due to two primary constraints: the requirement for a prior mapping pass and sensitivity to scene dynamics. When furniture is moved or environments are encountered for the first time, feature-matching methods typically degrade. LaLaLoc addresses these challenges by focusing on the underlying structural geometry—the "geometric vibes"—of a space, which remain constant regardless of interior changes.

Methodology: Cross-Modal Latent Space

The core of LaLaLoc is a quasi-Siamese network architecture that learns a unified representation for disparate data modalities.

The Layout Branch ($\Phi_{layout}$)

Acting as the "teacher," this branch encodes structural geometry rendered from a 2D floor plan into a compact latent vector. To ensure that geometrically similar locations are mapped closely together, the model employs a log-ratio loss based on the Chamfer distance between point clouds. This metric proved superior to standard image-based losses in capturing structural similarity.

The Image Branch ($\Phi_{image}$)

The "student" branch learns to map cluttered RGB panoramas to the same latent coordinates as their corresponding geometric layouts. By training against the layout branch, the image branch learns to "see through" transient objects like furniture to identify the permanent structural skeleton of the room.

Localization Pipeline

Localization is performed in a two-stage process that balances efficiency and precision.

The LaLaLoc localization pipeline consists of coarse retrieval followed by fine-grained latent pose optimization.
  1. Coarse Retrieval: The system samples a grid of potential poses across the floor plan. It renders layouts for these poses and identifies the nearest neighbor in the latent space relative to the query image's embedding.
  2. Fine Refinement (Latent Pose Optimization): To achieve sub-grid accuracy, LaLaLoc utilizes Latent Pose Optimization (LPO). A differentiable renderer iteratively adjusts the camera pose, moving it in the latent space until the rendered layout's embedding matches the image's embedding. This gradient-based refinement avoids the computational overhead of explicit reconstruction.

Experimental Evaluation

The framework was evaluated using the Structured3D dataset, which includes 3,500 house designs with varying furniture levels (Empty, Simple, and Full).

Key Results

  • Precision: In fully furnished environments, LaLaLoc achieved a median localization error of 8.3 cm.
  • Robustness: While baseline methods like ICP saw errors escalate from 2.4 cm to 23.0 cm when moving from empty to full rooms, LaLaLoc remained stable (5.7 cm to 8.3 cm).
  • Efficiency: The complete pipeline averages 4.65 seconds, significantly faster than the 20+ seconds required by ICP-based alternatives.
Qualitative results showing localization probability heatmaps across different floor plan layouts compared to ground truth.

Discussion and Future Work

LaLaLoc demonstrates that anchoring latent representations to static floor plan geometry provides a powerful inductive bias for localization in dynamic scenes. By optimizing directly in the latent space, the system maintains high accuracy without the fragility of traditional feature matching.

Current limitations include potential ambiguity in symmetrical rooms and local minima caused by large furniture items that mimic wall structures. Future research will explore incorporating semantic labels to resolve layout ambiguities and extending the framework to multi-story environments.

Summary updated

Abstract

We present LaLaLoc to localise in environments without the need for prior visitation, and in a manner that is robust to large changes in scene appearance, such as a full rearrangement of furniture. Specifically, LaLaLoc performs localisation through latent representations of room layout. LaLaLoc learns a rich embedding space shared between RGB panoramas and layouts inferred from a known floor plan that encodes the structural similarity between locations. Further, LaLaLoc introduces direct, cross-modal pose optimisation in its latent space. Thus, LaLaLoc enables fine-grained pose estimation in a scene without the need for prior visitation, as well as being robust to dynamics, such as a change in furniture configuration. We show that in a domestic environment LaLaLoc is able to accurately localise a single RGB panorama image to within 8.3cm, given only a floor plan as a prior.
BibTeX
@misc{howardjenkins2021lalaloc,
  author = {Henry Howard-Jenkins and José-Raúl Ruiz-Sarmiento and Victor Adrian Prisacariu},
  title = {{LaLaLoc: Latent Layout Localisation in Dynamic, Unvisited Environments}},
  year = {2021},
  eprint = {2104.09169},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2104.09169},
}
Figure