Human-in-the-Loop Local Corrections of 3D Scene Layouts via Infilling

Christopher Xie, Armen Avetisyan, Henry Howard-Jenkins, Yawar Siddiqui, Julian Straub, Richard A. Newcombe, Vasileios Balntas, Jakob Engel

Abstract: We present a novel human-in-the-loop approach to estimate 3D scene layout that uses human feedback from an egocentric standpoint. We study this approach through introduction of a novel local correction task, where users identify local errors and prompt a model to automatically correct them. Building on SceneScript, a state-of-the-art framework for 3D scene layout estimation that leverages structured language, we propose a solution that structures this problem as "infilling", a task studied in natural language processing. We train a multi-task version of SceneScript that maintains performance on global predictions while significantly improving its local correction ability. We integrate this into a human-in-the-loop system, enabling a user to iteratively refine scene layout estimates via a low-friction "one-click fix" workflow. Our system enables the final refined layout to diverge from the training distribution, allowing for more accurate modelling of complex layouts.

Introduction

Current 3D scene understanding models typically follow a "global prediction" paradigm, where a model attempts to generate an entire room layout from a single point cloud. While effective for standard environments, these models often struggle with "out-of-distribution" real-world settings that differ significantly from synthetic training data. When a model misplaces a wall or misses a door, users lack low-friction tools to correct these local errors, particularly from an egocentric perspective common in Augmented Reality (AR) headsets.

System overview showing the iterative refinement of an initial layout through local corrections.

To address this, we introduce a human-in-the-loop system that allows users to refine 3D scene layouts using a "one-click fix" workflow. By framing local geometric corrections as a text-based "infilling" task—analogous to how large language models predict missing words—the system enables iterative correction of walls, doors, and windows.

Methods

Our approach builds upon SceneScript, which represents 3D geometry as a structured language sequence (e.g., make_wall(x, y, ...)). We treat local correction as an infilling task, adapting the "Fill In the Middle" (FIM) strategy from Natural Language Processing.

Key technical components include:

  1. Multi-task Training: A Transformer model is trained simultaneously on Global Prediction (full scene generation) and Local Correction (filling in missing pieces).
  2. The "One-Click Fix": In a mixed-reality setting, a user identifies an error. The system masks the corresponding entity in the script and prompts the model to generate a replacement based on the surrounding context and local point cloud.
  3. Egocentric Anchoring: The coordinate system is centered on the user’s current pose, reducing the complexity of the model's output.
  4. Subsequence Positional Embeddings: Prefix, target, and suffix segments receive unique learnable embeddings to better leverage the segmented structure of the layout language.

Results and Insights

Jointly training on global and local tasks maintains global performance while significantly boosting local correction accuracy. On the Aria Synthetic Environments (ASE) dataset, local correction accuracy (AvgF1) increased from 92% to 98%.

The improvement is more pronounced on real-world data (Aria Everyday Objects dataset). For wall corrections, accuracy improved from 5.0% to 56.0% when the model was trained on the more complex iASE dataset. In a user study, participants preferred corrections from the multi-task model 57.2% of the time, compared to 18.9% for the baseline.

Crucially, iterative local corrections allow the final layout to diverge from the training distribution. This enables the human-AI team to model complex, unique rooms that a standalone AI could not represent accurately. Each local correction is highly efficient, taking less than one second.

Conclusion

While the system enables rapid refinement, limitations remain. The model occasionally struggles with highly cluttered entities (e.g., closely spaced windows) and does not always align with user heuristics for placement. Future work will focus on reducing the interaction count for complex scenes and extending the infilling approach to volumetric objects like furniture.

Summary updated

Abstract

We present a novel human-in-the-loop approach to estimate 3D scene layout that uses human feedback from an egocentric standpoint. We study this approach through introduction of a novel local correction task, where users identify local errors and prompt a model to automatically correct them. Building on SceneScript, a state-of-the-art framework for 3D scene layout estimation that leverages structured language, we propose a solution that structures this problem as "infilling", a task studied in natural language processing. We train a multi-task version of SceneScript that maintains performance on global predictions while significantly improving its local correction ability. We integrate this into a human-in-the-loop system, enabling a user to iteratively refine scene layout estimates via a low-friction "one-click fix'' workflow. Our system enables the final refined layout to diverge from the training distribution, allowing for more accurate modelling of complex layouts.
BibTeX
@inproceedings{xie2025human,
  author = {Christopher Xie and Armen Avetisyan and Henry Howard-Jenkins and Yawar Siddiqui and Julian Straub and Richard A. Newcombe and Vasileios Balntas and Jakob Engel},
  title = {{Human-in-the-Loop Local Corrections of 3D Scene Layouts via Infilling}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2025},
}

Loading related content...

Figure