Relaxing the Manhattan World Assumption

696 concepts457 papers13 proposals

When humans look at an indoor scene, we instantly grasp its geometry. We don't see a chaotic cloud of millions of independent 3D points; we see a floor, a ceiling, and a few flat walls meeting at right angles. For decades, computer vision and automated architectural design have relied on this exact intuition, formalizing it as the Manhattan World Assumption (MWA).

The MWA posits that all surfaces in an environment are aligned with three mutually perpendicular axes. It is the ultimate geometric cheat code. By collapsing the infinite degrees of freedom of 3D space into a rigid, predictable grid, it makes highly complex problems—like reconstructing a room from a single photograph or generating a floorplan from a graph—computationally tractable. For instance, in Extracting vanishing points across multiple views, Michael Hornacek demonstrated how enforcing these mutual orthogonality constraints early in a pipeline allows systems to stably calculate scene orientation across different camera viewpoints.

However, the real world is rarely a perfect box. Modern architecture thrives on L-shaped rooms, slanted roofs, alcoves, curved walls, and non-orthogonal boundaries. When an algorithm bound by the MWA encounters these features, the cheat code becomes a straightjacket. A growing body of research focuses on breaking free from this rectilinear constraint, developing flexible spatial priors that can reason about complex, non-rectilinear geometries without losing structural coherence.

The Straightjacket of the Rectilinear Prior

To understand why relaxing the MWA is difficult, we must look at how deeply embedded it is across different domains, and where it fails.

In 3D room layout estimation, early models frequently assumed that every room was a simple six-sided cuboid. As Jia Zheng et al. observe in Layout-Guided Novel View Synthesis From a Single Indoor Panorama, this boxy assumption completely breaks down when a system attempts to model irregular architectural features or transition to outdoor settings where clear wall-and-ceiling boundaries disappear. When a model forced to find a cuboid encounters an L-shaped room, it either hallucinates non-existent walls or miscalculates the depth of the space entirely.

In generative architectural design, the bias toward axis-aligned rectangular primitives creates a similar bottleneck. Traditional generative adversarial networks (GANs) and layout models optimize for bounding boxes aligned to a global grid. In Graph Transformer GANs for Graph-Constrained House Generation, Hao Tang et al. point out that this grid-bias prevents the synthesis of organic, real-world shapes. If an architect wants to generate a floorplan with a diagonal wall or a curved facade, standard models cannot represent it because their underlying mathematical primitives are fundamentally rectilinear.

The consequences extend beyond graphics to physical simulations. For example, Yupeng Wan et al. in Research on Prediction Method of Objective Assessment of Building Acoustics Based on Machine Learning found that machine learning models trained exclusively on simplified rectangular room geometries fail to generalize to real-world acoustic environments. Sound waves behave radically differently when encountering non-parallel surfaces or curved walls, meaning MWA-biased models provide highly inaccurate feedback during the early stages of architectural design.

Strict Manhattan World:
┌─────────────────────────┐
│         Room            │  --> Limited to 90-degree intersections
└─────────────────────────┘      and axis-aligned boxes.

Relaxed Non-Rectilinear World:
┌─────────────────┐
 \               /           --> Accommodates arbitrary angles,
  \    Room     /                curved boundaries, and
   └───────────┘                 complex polygonal layouts.

Mechanisms of Geometric Relaxation

How do we abandon the simplicity of the Manhattan grid without letting the geometry devolve into an unstructured "polygon soup"? Researchers have converged on three primary strategies: shifting from global grids to relational graphs, leveraging continuous vector spaces, and reasoning via plane intersections.

1. Shifting from Grids to Relational Graphs

Instead of forcing structural elements onto a global coordinate grid, modern frameworks represent scenes as graphs where nodes are geometric primitives (like walls, roofs, or rooms) and edges represent topological relationships (like connectivity, colinearity, or adjacency).

For outdoor architecture, Fuyang Zhang et al. introduced a convolutional message passing neural network in Conv-MPN: Convolutional Message Passing Neural Network for Structured Outdoor Architecture Reconstruction that reconstructs buildings as planar graphs from satellite imagery. By allowing the network to reason about the relationships between planes rather than forcing them into 90-degree buckets, it successfully reconstructs complex, angled roof structures that defy standard orthogonal patterns.

Similarly, in Roof-GAN: Learning To Generate Roof Geometry and Relations for Residential Houses, Yiming Qian et al. use a graph-based representation to ensure that generated roofs respect real-world architectural constraints like coplanarity without restricting the design to simple orthogonal layouts.

2. Continuous Vector Denoising

In generative tasks, moving away from discrete pixel grids or rigid bounding boxes toward continuous vector representations allows models to express arbitrary angles.

A prime example of this is HouseDiffusion: Vector Floorplan Generation via a Diffusion Model with Discrete and Continuous Denoising. Mohammad Amin Shabani et al. developed a framework that generates vector-graphics floorplans directly from room-connectivity graphs (bubble diagrams). By utilizing a Transformer-based diffusion model with a hybrid discrete-continuous denoising objective, the model can manipulate the vertices of polygons freely in continuous space. This allows it to generate geometrically precise layouts that naturally include non-rectilinear or diagonal structures, outperforming previous grid-bound generative models.

3. Geometry-Driven Plane Intersections

In layout estimation, rather than trying to fit a predefined 3D box template to an image, models can predict the local geometric properties of individual surfaces and infer the global structure through their mathematical intersections.

In GeoLayout: Geometry Driven Room Layout Estimation Based on Depth Maps of Planes, Weidong Zhang et al. shifted the task from 2D edge detection to 3D geometric reasoning. Their model predicts the depth maps and orientations of dominant planes (walls, floors, ceilings) independently. By calculating where these predicted planes intersect in 3D space, the layout emerges organically. Because the model does not enforce a global cuboid constraint, it seamlessly handles complex, non-cuboid room shapes, such as alcoves or multi-walled spaces, that completely stymie traditional MWA-reliant methods.

Why the Shift Works

The transition away from the Manhattan World Assumption works because it replaces hard geometric constraints with flexible structural priors.

Approach Geometric Prior Failure Mode Solution
MWA-Bound Global Orthogonal Grid Fails on L-shapes, curves, and slanted roofs Forces irregular geometry into artificial 90° angles
Relaxed/Graph-Based Local Relational Topology Maintains consistency without restricting global angles Allows vertices and planes to orient freely while preserving connectivity

When Ashutosh Saxena et al. pioneered early 3D reconstruction in Learning 3-D Scene Structure from a Single Still Image, they modeled the environment as a collection of thousands of small, oriented planar "superpixels" governed by a Markov Random Field. This early intuition proved that the world is better understood as a network of local geometric relationships rather than a single rigid box.

By grounding modern deep networks in relational graphs and continuous vector spaces, today's models retain the "common sense" of architecture—knowing that walls connect to floors, and roofs sit atop walls—without forcing those elements onto an artificial grid. The result is a new generation of spatial AI capable of understanding, navigating, and designing the full complexity of the real world.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 478 papers
  2. Version 1 · 2026-10-04 · 546 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_manhattan_world_assumption_4622dfec,
  title = {Manhattan world assumption},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/manhattan-world-assumption/rdr_4622dfec55fe457c8c1c60602cf0a62a},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_manhattan_world_assumption_4622dfec_v2,
  title = {Manhattan world assumption},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/manhattan-world-assumption/rdr_4622dfec55fe457c8c1c60602cf0a62a/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake