TL;DR
Any2Building is a unified framework that reconstructs structured 3D building models (LoD2) from a variety of inputs, including partial point clouds, overhead images, or a combination of both. By using a shared set of spatial queries on a 3D grid, the model aligns heterogeneous observations to a common geometric target, outperforming specialized state-of-the-art methods across multiple reconstruction tasks.
What Problem Does This Paper Address?
Creating "digital twins" of cities requires accurate 3D building models. Traditionally, researchers have treated building reconstruction as two separate problems:
- Point Cloud Completion: Filling in the gaps of sparse, noisy LiDAR data.
- Image-to-3D Generation: Inferring 3D structure from 2D satellite or aerial imagery.
The challenge is that these modalities are complementary but hard to fuse. Point clouds provide precise geometry but are often incomplete; images provide clear footprints and roof layouts but lack height information and hide vertical surfaces. Existing models are usually tailored to one or the other, making it difficult to leverage all available data. Furthermore, most generative 3D models produce "bloppy" meshes that aren't suitable for the clean, "low-poly" Level of Detail 2 (LoD2) standards required for urban modeling.
Methods
The authors introduce a unified pipeline that centers on Shared Spatial Queries.
- Multimodal Tokenization: The system first encodes inputs—whether they are points (via EdgeConv), images (via DINOv3), or rough geometric priors (like bounding boxes)—into a sequence of tokens.
- Shared Spatial Query Conditioner: This is the "brain" of the model. It maintains a $16^3$ grid of learnable queries. These queries use cross-attention to "grab" relevant features from the input tokens. Crucially, these queries are supervised by an auxiliary loss to predict building occupancy, forcing them to learn a consistent geometric representation regardless of the input type.
- Flow-Matching DiT: A Diffusion Transformer (DiT) takes these query features and generates a structure latent. The authors use a technique called "Spatial Prompt Injection" to ensure the generated features stay perfectly aligned with the 3D grid.
- LoD2 Construction: The output is decoded into a $64^3$ voxel grid. A dedicated surface generator then refines this into a high-poly mesh, which is finally simplified into a clean, polygonal LoD2 model using a low-poly remeshing pipeline.
Datasets
To train such a versatile model, the authors introduced BuildingWorld-Image. This massive dataset contains approximately one million aligned triplets:
- Partial point clouds.
- Real overhead imagery.
- Ground-truth LoD2 building models.
The data spans 18 cities across four continents, providing the architectural diversity needed for a robust foundation model. The authors also evaluated their method on existing benchmarks like Building-PCC and SatSkylines.
Key Insights and Ideas
- The Geometric Bridge: By supervising the spatial queries with complete building occupancy, the model learns to "imagine" missing parts of a building (like the back of a house or the height of a roof) by relating 2D visual cues to 3D geometric patterns.
- Positive Transfer: One of the most striking findings is that training on multiple tasks simultaneously (joint training) actually makes the model better at individual tasks. For example, the model's ability to complete point clouds improved simply because it also learned how to generate buildings from images.
- Modality Robustness: The model is exceptionally resilient to data degradation. If the point cloud is extremely sparse (e.g., only 41 points), the model can lean on the overhead image to maintain high reconstruction accuracy.
Key Results
Any2Building sets new benchmarks across the board:
- Point Cloud Completion: On the Building-PCC dataset, it achieved a 96.12% F-score, significantly higher than previous leaders like BuildAnyPoint (91.38%).
- Image-Based Generation: It reduced the Chamfer Distance (a measure of geometric error) from 54.89 to 38.90 compared to the TRELLIS generative model, while nearly doubling the F-score.
- Multimodal Fusion: When both points and images are available, the model achieves a low-poly F-score of 77.41%, outperforming multimodal baselines like Hunyuan3D-Omni (50.00%).
Limitations and Future Work
While powerful, the authors note a few areas for improvement:
- Localization Dependency: The current evaluation assumes buildings are already localized with a 2D bounding box. Integrating an end-to-end building detector would be a natural next step for city-scale automation.
- Pipeline Complexity: The multi-stage process (voxels $\rightarrow$ high-poly mesh $\rightarrow$ low-poly mesh) is effective but computationally intensive. Future work could explore direct polygonal generation.
- Texture Synthesis: While the model supports downstream texturing, the current approach relies on external tools like ControlNet. A more integrated end-to-end geometry and texture generator could further improve visual fidelity.
Summary updated