Designing a Human-in-the-Loop System for Object Detection in Floor Plans

Johannes Jakubik, Patrick Hemmer, Michael Vössing, Benedikt Blumenstiel, Andrea Bartos, Kamilla MohrAAAI 2022 · DOI

Introduction

In the Architecture, Engineering, and Construction (AEC) industry, "quantity determination" is a critical but labor-intensive task. Engineers must manually count every door, window, and electrical outlet in a floor plan to create a Bill of Materials (BOM) for bidding and billing. This process is not only time-consuming but also prone to human error. While automated object detection seems like a natural solution, two major hurdles have historically prevented widespread adoption: a lack of symbol standardization across different firms and the high financial stakes of even minor counting errors.

To address these challenges, researchers have developed a human-in-the-loop (HITL) system. This approach combines the speed of deep learning with the reliability of human expertise, creating a workflow that adapts to new projects quickly and knows when to ask for help.

System Architecture

The proposed system is built upon three main pillars: synthetic data generation, an uncertainty-aware object detection model, and a specialized user interface for expert review.

The three-pillar workflow: Synthetic data generation, Object detection model training, and the HITL system review.

1. Synthetic Data Generation

To solve the "cold start" problem—where a model needs labeled data for a new project's specific symbols—the authors developed an automated pipeline. By extracting symbols directly from a floor plan's legend, the system "pastes" these icons onto empty background images. Using various augmentations such as rotation, blurring, and lighting changes, the pipeline generates 20,000 synthetic training images without requiring manual human labeling.

2. Uncertainty-Aware Object Detection

The core of the system is a Faster R-CNN model with a ResNet-50 backbone. To make the model "self-aware" of its own limitations, the researchers employed a Deep Ensemble approach, training five separate versions of the model. By measuring the variance between these five models, the system can calculate an uncertainty score. If the ensemble disagrees on a classification, the symbol is flagged for human intervention.

3. Human-in-the-Loop (HITL) Interface

Symbols flagged with high uncertainty are sent to a dedicated interface. Rather than scanning an entire blueprint, human experts use a "Gallery View" to quickly confirm or correct the AI's predictions. This targeted review process is significantly more efficient than traditional manual counting.

The HITL interface featuring the 'Gallery View' for rapid symbol verification and correction.

Performance and Key Insights

The system was evaluated using a real-world industry dataset consisting of 44 large-scale industrial floor plans containing nearly 6,000 annotated symbols across 39 classes.

  • Automated Baseline: The standalone deep learning model achieved a mean Average Precision (mAP) of 82.7%.
  • HITL Performance: By integrating the human review process for uncertain detections, the overall prediction accuracy rose to 92.1%, representing a 12.9% improvement over the fully automated method.
  • Efficiency Gains: On average, the model correctly predicted 86.9% of symbols autonomously. This allowed human experts to focus exclusively on the most difficult cases, drastically reducing the total manual effort required for a project.

The researchers found that using ensembles as uncertainty proxies was far more effective than relying on the confidence scores of a single model, which are often overconfident. Specifically, the "Variation Ratio" acquisition function proved highly effective at identifying false positives—instances where the AI mistook background noise for a construction symbol.

Limitations and Future Outlook

While the system significantly improves precision and reduces manual labor, a "Recall" limitation remains. The current HITL workflow primarily asks humans to verify regions where the AI detected something. If the model misses a symbol entirely (a false negative), it may not appear in the Gallery View. Consequently, a final "Plan View" check by a human is still necessary to ensure 100% accuracy.

Future work will focus on refining the interface to help humans catch these elusive missed symbols more efficiently and deploying the prototype in live business environments to quantify the exact time and cost savings in real-world construction bidding.

Summary updated

BibTeX
@inproceedings{jakubik2022designing,
  author = {Johannes Jakubik and Patrick Hemmer and Michael Vössing and Benedikt Blumenstiel and Andrea Bartos and Kamilla Mohr},
  title = {{Designing a Human-in-the-Loop System for Object Detection in Floor Plans}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i11.21522},
}
Figure