Any2Building: Unifying Point Clouds and Images for Building Reconstruction via Shared Spatial Queries

Hongye Hou, Zhuoxiao Li, Liu Xiangyu, Tengxi Wang, Jing Ou, Dongli Wu, Shuai Zhang, Tongyan Hua, Wufan Zhao

TL;DR

Any2Building is a unified framework that reconstructs structured 3D building models (LoD2) from a variety of inputs, including partial point clouds, overhead images, or a combination of both. By using a shared set of spatial queries on a 3D grid, the model aligns heterogeneous observations to a common geometric target, outperforming specialized state-of-the-art methods across multiple reconstruction tasks.

What Problem Does This Paper Address?

Creating "digital twins" of cities requires accurate 3D building models. Traditionally, researchers have treated building reconstruction as two separate problems:

  1. Point Cloud Completion: Filling in the gaps of sparse, noisy LiDAR data.
  2. Image-to-3D Generation: Inferring 3D structure from 2D satellite or aerial imagery.

The challenge is that these modalities are complementary but hard to fuse. Point clouds provide precise geometry but are often incomplete; images provide clear footprints and roof layouts but lack height information and hide vertical surfaces. Existing models are usually tailored to one or the other, making it difficult to leverage all available data. Furthermore, most generative 3D models produce "bloppy" meshes that aren't suitable for the clean, "low-poly" Level of Detail 2 (LoD2) standards required for urban modeling.

Methods

The authors introduce a unified pipeline that centers on Shared Spatial Queries.

The Any2Building pipeline uses shared spatial queries to bridge the gap between points, images, and geometric priors.
  • Multimodal Tokenization: The system first encodes inputs—whether they are points (via EdgeConv), images (via DINOv3), or rough geometric priors (like bounding boxes)—into a sequence of tokens.
  • Shared Spatial Query Conditioner: This is the "brain" of the model. It maintains a $16^3$ grid of learnable queries. These queries use cross-attention to "grab" relevant features from the input tokens. Crucially, these queries are supervised by an auxiliary loss to predict building occupancy, forcing them to learn a consistent geometric representation regardless of the input type.
  • Flow-Matching DiT: A Diffusion Transformer (DiT) takes these query features and generates a structure latent. The authors use a technique called "Spatial Prompt Injection" to ensure the generated features stay perfectly aligned with the 3D grid.
  • LoD2 Construction: The output is decoded into a $64^3$ voxel grid. A dedicated surface generator then refines this into a high-poly mesh, which is finally simplified into a clean, polygonal LoD2 model using a low-poly remeshing pipeline.

Datasets

To train such a versatile model, the authors introduced BuildingWorld-Image. This massive dataset contains approximately one million aligned triplets:

  • Partial point clouds.
  • Real overhead imagery.
  • Ground-truth LoD2 building models.

The data spans 18 cities across four continents, providing the architectural diversity needed for a robust foundation model. The authors also evaluated their method on existing benchmarks like Building-PCC and SatSkylines.

Key Insights and Ideas

  • The Geometric Bridge: By supervising the spatial queries with complete building occupancy, the model learns to "imagine" missing parts of a building (like the back of a house or the height of a roof) by relating 2D visual cues to 3D geometric patterns.
  • Positive Transfer: One of the most striking findings is that training on multiple tasks simultaneously (joint training) actually makes the model better at individual tasks. For example, the model's ability to complete point clouds improved simply because it also learned how to generate buildings from images.
  • Modality Robustness: The model is exceptionally resilient to data degradation. If the point cloud is extremely sparse (e.g., only 41 points), the model can lean on the overhead image to maintain high reconstruction accuracy.

Key Results

Any2Building sets new benchmarks across the board:

  • Point Cloud Completion: On the Building-PCC dataset, it achieved a 96.12% F-score, significantly higher than previous leaders like BuildAnyPoint (91.38%).
  • Image-Based Generation: It reduced the Chamfer Distance (a measure of geometric error) from 54.89 to 38.90 compared to the TRELLIS generative model, while nearly doubling the F-score.
  • Multimodal Fusion: When both points and images are available, the model achieves a low-poly F-score of 77.41%, outperforming multimodal baselines like Hunyuan3D-Omni (50.00%).
Qualitative examples showing Any2Building's performance across different observation settings.

Limitations and Future Work

While powerful, the authors note a few areas for improvement:

  • Localization Dependency: The current evaluation assumes buildings are already localized with a 2D bounding box. Integrating an end-to-end building detector would be a natural next step for city-scale automation.
  • Pipeline Complexity: The multi-stage process (voxels $\rightarrow$ high-poly mesh $\rightarrow$ low-poly mesh) is effective but computationally intensive. Future work could explore direct polygonal generation.
  • Texture Synthesis: While the model supports downstream texturing, the current approach relies on external tools like ControlNet. A more integrated end-to-end geometry and texture generator could further improve visual fidelity.

Summary updated

Abstract

Abstracting structured buildings from limited observations is essential for 3D urban modeling and digital twins. Partial point clouds and overhead images of the same building provide incomplete 3D surface measurements and visual cues to its footprint, roof geometry, and rooftop structures, respectively. This motivates a unified framework that aligns geometric and visual cues in 3D and reconstructs buildings from either or both modalities. Building on this connection, we present Any2Building, a unified model for Level of Detail 2 (LoD2) building reconstruction from partial point clouds, overhead images, or both, with optional coarse geometric priors. At its core is an occupancy-grounded shared spatial queries representation that organizes available observations within a common 3D grid. Supervised by complete building surface voxels, these queries connect observational evidence to target geometry, enabling a shared generator to learn structural priors across tasks and providing spatial guidance for recovering missing structure. A trained voxel conditional surface-generation and low-poly remeshing pipeline then converts the predicted structure into compact LoD2 building models. To support this framework, we introduce BuildingWorld-Image, comprising approximately one million paired samples of partial point clouds, overhead images, and LoD2 building models across 18 cities. Experiments on multiple benchmarks demonstrate that Any2Building outperforms existing methods in geometric reconstruction accuracy across unimodal and multimodal point-cloud completion, single-image building generation, and prior-assisted generation, while maintaining detailed and robust recovery when point cloud input modality is degraded.
BibTeX
@misc{hou2026any,
  author = {Hongye Hou and Zhuoxiao Li and Liu Xiangyu and Tengxi Wang and Jing Ou and Dongli Wu and Shuai Zhang and Tongyan Hua and Wufan Zhao},
  title = {{Any2Building: Unifying Point Clouds and Images for Building Reconstruction via Shared Spatial Queries}},
  year = {2026},
  eprint = {2611.23187},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23187},
}
Figure