Rohbau3D-Semantics: Dense Semantic and Instance Labels for Point Clouds of Indoor Shell Construction Scenes

Lukas Rauch, Thomas BramlarXiv 2026 · arXiv

TL;DR

Rohbau3D-Semantics is a massive new dataset featuring 504 high-resolution terrestrial laser scans from 14 active shell construction sites, providing 12.5 billion points with dense semantic and instance-level labels. It offers a realistic, challenging benchmark for 3D scene understanding in unfinished, cluttered environments, moving beyond the "clean" furnished interiors typical of existing datasets.

What Problem Does This Paper Address?

While 3D scene understanding has flourished thanks to datasets like ScanNet and S3DIS, these benchmarks primarily focus on finished, furnished residential and office spaces. In contrast, shell construction sites—referred to by the German term Rohbau—present a much messier reality.

These environments are characterized by:

  • Dynamic conditions: Temporary structures, scaffolding, and piles of debris.
  • Geometric complexity: Unfinished building elements, partial occlusions, and protective coverings.
  • Sensor artifacts: Raw terrestrial laser scanner (TLS) data includes non-uniform sampling, reflections from shiny surfaces, and "ghost points" (measurement errors near sharp edges).

Existing construction-specific datasets are often too small or lack the dense, point-wise instance annotations required for advanced engineering tasks like automated progress monitoring or comparing "as-built" conditions against Building Information Models (BIM).

Methods

The researchers built upon the previously released Rohbau3D domain data, focusing on high-fidelity acquisition and rigorous annotation:

  • Acquisition: Data was captured using a tripod-mounted FARO Focus M70 terrestrial laser scanner (see Figure 2). To preserve the raw characteristics of physical sensing, the authors avoided aggressive filtering or mesh reconstruction, keeping the data in its original scanner-centered local coordinate system.
  • Annotation: A professional team spent over 3,800 hours manually labeling points. This process was overseen by civil engineering experts to ensure technical accuracy. They used a taxonomy of 17 foreground classes (like Beam, Column, TGA/MEP) and one background class (None).
  • 2D Projections: Recognizing the value of multimodal research, the team developed a pipeline to project the 3D point clouds into geometrically aligned 2D panoramas and cube-maps. These include color, depth, intensity, surface normals, and label maps (see Figure 1).
Figure 1. Rohbau3D-Semantics: Dataset features in both 3D and 2D-projected form. A) color, B) semantic classes, C) class instances, D) color, E) depth, F) reflection intensity, G) surface normals, H) semantic classes, I) class instances.

Datasets

The primary contribution is the Rohbau3D-Semantics dataset. As shown in the paper's comparison table (Figure 6), it stands out by providing the largest collection of annotated points (12.5 billion) and area (66,000 m²) specifically for the construction domain.

  • Scope: 504 standalone scans from 14 different sites (residential, office, schools, etc.).
  • Labels: 17 foreground classes + 1 background class.
  • Instances: 20,786 annotated object instances.
Figure 2. Illustration of the terrestrial laser scanner device that was used to record the data, and a representative construction site scene in the background. The laser sensor head rotates simultaneously around two axes (yaw and roll), thereby mapping the space in a spherical grid.

Key Insights and Ideas

  • Embracing the "Mess": Unlike many datasets that clean up sensor noise, Rohbau3D-Semantics explicitly labels artifacts like reflections and ghost points as the "None" class (see Figure 3). This forces models to learn to distinguish between real geometry and sensor-induced errors.
  • Instance-Level Engineering: By providing instance labels (e.g., distinguishing one specific wall segment from another), the dataset supports downstream engineering tasks like tracking the installation state of individual components.
  • Class Imbalance is a Feature: The dataset reflects the reality of buildings—floors, walls, and ceilings make up 80% of the points. However, the instance-level distribution is less skewed, showing that smaller, critical components like columns and beams appear frequently despite their lower point count (see Figure 4).
Figure 3. Illustration of reflection noise and ghost points from LiDAR sensing. Reflections typically exhibit nonsensical color textures in RGB space. Ghost points typically appear suspended in the air or near sharp edges. Black polygons highlight such regions that are assigned to the None class (gray).

Key Results

The authors established a baseline for 3D semantic segmentation using several state-of-the-art models, including PointNeXt, Swin3D, and Point Transformer V3 (PTv3).

  • Top Performer: PTv3 achieved the highest mean Intersection over Union (mIoU) of 0.72.
  • Performance Split: Models performed exceptionally well on large structural elements (Ceiling, Floor, and Wall consistently hit high IoU). However, semantically specific or smaller categories like Window Cutout, Elevator Doors, and Wall Cutout remain significant challenges for current AI architectures.
  • Qualitative Success: Figure 5 demonstrates that while models can generally grasp the scene layout, they often struggle with fine-grained boundaries in cluttered areas.

Limitations and Future Work

  • Baseline Scope: The current evaluation only covers semantic segmentation. The authors have released instance labels and 2D modalities, but benchmarking instance segmentation and multimodal fusion is left for future research.
  • Scanner Type: The data comes from stationary terrestrial scanners. While highly accurate, future work could explore how well models trained on this data generalize to lower-density mobile LiDAR used by robots.
  • Generalization: The authors suggest that site-level cross-validation would provide even stronger evidence of how well these models work on entirely new, unseen construction projects.

Rohbau3D-Semantics is now publicly available on GitHub, offering a new frontier for researchers looking to tackle the "wild" side of indoor 3D scene understanding.

Summary updated

Abstract

We present the Rohbau3D-Semantics dataset, which adds dense semantic and instance-level labels to laser scans of the previously released Rohbau3D domain data from shell construction sites. The dataset comprises 504 point cloud scenes from 14 survey sites, each captured with a terrestrial laser scanner and stored in a scanner-centered local coordinate system. Unlike point clouds sampled from reconstructed meshes or synthetic building models, the data preserve the characteristic properties of real laser scanning, including non-uniform sampling, measurement artifacts, reflections, and fine geometric detail. Combined with manual labels, expert oversight during pointwise annotation, and full-scene review, this yields a high-quality benchmark for semantic scene understanding in complex built environments. The dataset includes 17 class labels of building components and one background class, along with instance annotations for object-level analysis. In addition to the 3D point clouds, the release also provides scripts for deriving geometrically aligned panoramic and cube-map representations of color, LiDAR reflection intensity, depth, surface normals, semantic labels, and instance labels from the 3D arrays. We describe the acquisition pipeline, annotation protocol, quality-control process, and panorama-generation procedure, and provide benchmark splits and baseline experiments for semantic segmentation. By combining precise terrestrial laser scans, dense annotations, and a wide variety of shell-construction scenes, Rohbau3D-Semantics bridges the gap among synthetic indoor datasets, mesh-derived benchmarks, and smaller construction-specific resources. The dataset is intended to support research on 3D scene understanding, construction-site analytics, and multimodal scene analysis in the built environment. Access and download the full dataset via https://github.com/RauchLukas/rohbau3d
BibTeX
@misc{rauch2026rohbau,
  author = {Lukas Rauch and Thomas Braml},
  title = {{Rohbau3D-Semantics: Dense Semantic and Instance Labels for Point Clouds of Indoor Shell Construction Scenes}},
  year = {2026},
  eprint = {2611.06180},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.06180},
}

Loading related content...

Figure