Semantic Segmentation Method for Automated Indoor 3D Reconstruction Based on Architectural-Knowledge-Aware Features

Yahan Chen, Wenzheng Liu, Xiaowei LuoWACV 2025

TL;DR

This paper introduces Architectural-Knowledge-Aware Features (AKAFs) to improve the semantic segmentation of indoor 3D point clouds, specifically for structural components like columns and beams. By incorporating architectural regularities into a novel two-stage framework, the method significantly boosts segmentation accuracy for these challenging elements, leading to more detailed and useful 3D indoor reconstructions.

What Problem Does This Paper Address?

Accurate 3D indoor reconstruction relies heavily on semantic segmentation, which assigns a label (e.g., wall, door, column) to each point in a 3D point cloud. While deep learning models have made strides, they often struggle with segmenting structural components. This is due to several factors:

  • Data Imbalance: Structural components are often less prevalent than basic elements like walls and floors.
  • Feature Ambiguity: Standard point cloud features (coordinates, colors, local geometry) can be similar for structural and non-structural elements, leading to confusion.
  • Lack of Architectural Context: Existing methods often overlook the inherent spatial regularities and relationships between building components that architects and humans naturally use for understanding a space.

This lack of accuracy for structural components limits the granularity and practical value of reconstructed indoor models, hindering applications like detailed interior design, escape simulations, and structural analysis.

Methods

The core contribution is the introduction of Architectural-Knowledge-Aware Features (AKAFs), specifically F1 and F2, integrated into a two-stage training framework.

  • F1 (Explicit Feature): This feature formalizes the position distribution pattern of structural components within a room. It's derived by considering the room's center as an origin and scaling global coordinates to a normalized range [-1, 1]. This explicit feature helps the model learn the typical locations of structural elements.
  • F2 (Implicit Feature): This feature captures the complex regularity of relative positions between different building components. It's extracted using a novel Semantic and Relative Position Fusing Module (SRPFM), which employs an attention mechanism. SRPFM fuses preliminary semantic predictions (from the explicit stage) with relative positional information from neighboring points.
  • Two-Stage Framework:
    • Explicit Stage: Takes raw point cloud data and F1 as input, performing an initial semantic segmentation.
  • Implicit Stage: Uses the SRPFM to derive F2 based on the initial semantic predictions and F1, then feeds F2 into the same backbone for a refined semantic segmentation.

The framework is designed to be compatible with various point-based semantic segmentation models, acting as a plug-in enhancement.

Datasets

The experiments were conducted on the S3DIS (Stanford 3D Indoor Scene Dataset). The authors re-categorized furniture points into "clutter" to focus on the 8 core building component classes relevant to their study.

Key Insights and Ideas

  • Architectural Regularity is Key: The paper demonstrates that incorporating explicit architectural knowledge (like component placement rules) significantly improves segmentation, especially for challenging structural elements.
  • Two-Stage Learning for Implicit Features: The two-stage approach is crucial. The first stage provides a necessary "warm-up" for the model to generate preliminary semantic information, which is then leveraged by the SRPFM in the second stage to learn more complex, implicit relationships.
  • Attention for Relational Understanding: The SRPFM's use of attention mechanisms effectively captures how points relate to their neighbors, both semantically and positionally, which is vital for distinguishing complex structural arrangements.
  • Portability of AKAFs: The proposed features and framework are designed to be compatible with existing state-of-the-art models, suggesting they can enhance a wide range of segmentation approaches.

Key Results

  • Improved Structural Component Segmentation: AKAFs increased the segmentation accuracy for structural components (beams and columns) by over 5% when using Pointnet++ as a backbone.
  • Overall Accuracy Boost: The overall semantic segmentation performance (mIoU) improved by approximately 2% with the full AKAFs (F1 & F2) compared to the baseline Pointnet++.
  • Portability Demonstrated: Experiments with KPConv and the state-of-the-art Point Transformer v3 (PT-v3) showed that AKAFs universally improve segmentation results across different backbones, though the effectiveness of F2 on PT-v3 was limited by computational constraints (smaller batch size).
  • Qualitative Improvements: Visual comparisons showed a more accurate distinction between structural components and surrounding elements, leading to visually closer segmentations to the ground truth.

Limitations and Future Work

  • Computational Cost for SOTA Models: Applying the full AKAFs (especially F2) to very large and complex models like PT-v3 can significantly increase memory requirements, necessitating smaller batch sizes which may limit the full potential of F2. The paper validates F1's effectiveness on PT-v3 but not F2.
  • Generalization of F2: While F1 is a global feature, F2's effectiveness might be influenced by the local context captured by the attention mechanism. Further exploration into how F2 generalizes across vastly different room layouts or building types could be beneficial.
  • Future Work: The authors suggest further investigation into integrating architectural knowledge into DL-based methods, implying that AKAFs are a promising step in this direction. The portability and compatibility analysis also points towards applying AKAFs to even more advanced models as they emerge.

Summary updated

BibTeX
@inproceedings{chen2025semantic,
  author = {Yahan Chen and Wenzheng Liu and Xiaowei Luo},
  title = {{Semantic Segmentation Method for Automated Indoor 3D Reconstruction Based on Architectural-Knowledge-Aware Features}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}

Loading related content...

Figure