User-in-the-Loop View Sampling with Error Peaking Visualization

Ayaka Yasunaga, Hideo Saito, Shohei Mori

Abstract

Capturing optimal image sets for high-quality 3D scene reconstruction is a mentally taxing task for users. We introduce "error peaking," an Augmented Reality (AR) visualization that highlights rendering artifacts in real-time. This system allows users to intuitively "paint away" errors by capturing new views precisely where they are needed, moving away from rigid, prescriptive sampling grids toward a more flexible, assessment-based workflow.

Introduction

To achieve high-quality novel view synthesis, traditional methods rely on Plenoptic Sampling Theory, which often dictates a dense, uniform grid of camera positions. Existing AR tools attempt to assist users by overlaying 3D boxes or axes to guide phone placement. However, these approaches are often mentally demanding, requiring tedious alignment tasks, and can be obstructive, hiding the subject of interest. Furthermore, they apply a "one-size-fits-all" density that ignores the varying geometric complexity of different scene regions.

Methodology

We propose a "user-in-the-loop" system that replaces prescriptive guides with a feedback loop based on Multi-Plane Images (MPI).

The system architecture facilitates a feedback loop between local capture and server-side MPI generation.

System Workflow

  1. Instant MPI Generation: Upon capturing a photo, the system utilizes a neural network (AdaMPI) to transform the 2D image into a stack of semi-transparent 3D layers. To overcome mobile hardware limitations, this processing is offloaded to a remote server.
  2. Metric Scale Alignment: The virtual depth of the generated MPI is aligned with the real-world metric depth provided by the device's SLAM sensors.
  3. Error Peaking Overlay: The system renders a "digital double" of the scene from the current MPI and compares it to the live camera feed. Pixels where the reconstruction significantly deviates from the live feed are highlighted in bright red.

Real-Time Feedback and Sampling

The core innovation is the "error peaking" visualization. Users see red "peaks" where the reconstruction fails—typically around complex edges or areas of disocclusion. By moving the camera toward these red regions and capturing additional frames, users effectively "diminish" the errors as the system updates the local light field.

Error peaking highlights reconstruction artifacts in red, guiding the user to capture missing data.

This shift from alignment to assessment allows users to explore 3D space freely, intervening only when the reconstruction "breaks." This targeted sampling achieves high-quality results with fewer images than uniform grids require.

Evaluation and Results

A user study with 20 participants compared the Error Peaking method against the industry-standard LLFF prescriptive grid approach.

  • User Experience: Participants reported significantly higher satisfaction (6.0/7 vs. 4.5/7) and felt better able to focus on the scene (6.5/7) rather than the interface.
  • Efficiency: The error peaking method required an average of 11.0 images, compared to 14.0 for the grid method.
  • Reconstruction Quality: Quantitative analysis using 3D Gaussian Splatting (3DGS) showed that our sampling strategy outperformed both random and uniform sampling in terms of PSNR and LPIPS metrics.

Conclusion and Future Work

While effective, the system currently struggles with non-Lambertian surfaces like metallic or reflective materials, which can trigger false error peaks. Future work will focus on reducing server latency (currently ~3.6s) to move processing entirely on-device and incorporating human factors, such as foveal vision prioritization, to further refine the error visualization.

Summary updated

Abstract

Augmented reality (AR) provides ways to visualize missing view samples for novel view synthesis. Existing approaches present 3D annotations for new view samples and task users with taking images by aligning the AR display. This data collection task is known to be mentally demanding and limits capture areas to pre-defined small areas due to the ideal but restrictive underlying sampling theory. To free users from 3D annotations and limited scene exploration, we propose using locally reconstructed light fields and visualizing errors to be removed by inserting new views. Our results show that the error-peaking visualization is less invasive, reduces disappointment in final results, and is satisfactory with fewer view samples in our mobile view synthesis system. We also show that our approach can contribute to recent radiance field reconstruction for larger scenes, such as 3D Gaussian splatting.
BibTeX
@misc{yasunaga2025user,
  author = {Ayaka Yasunaga and Hideo Saito and Shohei Mori},
  title = {{User-in-the-Loop View Sampling with Error Peaking Visualization}},
  year = {2025},
  eprint = {2506.21009},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2506.21009},
}
Figure