Abstract
Capturing optimal image sets for high-quality 3D scene reconstruction is a mentally taxing task for users. We introduce "error peaking," an Augmented Reality (AR) visualization that highlights rendering artifacts in real-time. This system allows users to intuitively "paint away" errors by capturing new views precisely where they are needed, moving away from rigid, prescriptive sampling grids toward a more flexible, assessment-based workflow.
Introduction
To achieve high-quality novel view synthesis, traditional methods rely on Plenoptic Sampling Theory, which often dictates a dense, uniform grid of camera positions. Existing AR tools attempt to assist users by overlaying 3D boxes or axes to guide phone placement. However, these approaches are often mentally demanding, requiring tedious alignment tasks, and can be obstructive, hiding the subject of interest. Furthermore, they apply a "one-size-fits-all" density that ignores the varying geometric complexity of different scene regions.
Methodology
We propose a "user-in-the-loop" system that replaces prescriptive guides with a feedback loop based on Multi-Plane Images (MPI).
System Workflow
- Instant MPI Generation: Upon capturing a photo, the system utilizes a neural network (AdaMPI) to transform the 2D image into a stack of semi-transparent 3D layers. To overcome mobile hardware limitations, this processing is offloaded to a remote server.
- Metric Scale Alignment: The virtual depth of the generated MPI is aligned with the real-world metric depth provided by the device's SLAM sensors.
- Error Peaking Overlay: The system renders a "digital double" of the scene from the current MPI and compares it to the live camera feed. Pixels where the reconstruction significantly deviates from the live feed are highlighted in bright red.
Real-Time Feedback and Sampling
The core innovation is the "error peaking" visualization. Users see red "peaks" where the reconstruction fails—typically around complex edges or areas of disocclusion. By moving the camera toward these red regions and capturing additional frames, users effectively "diminish" the errors as the system updates the local light field.
This shift from alignment to assessment allows users to explore 3D space freely, intervening only when the reconstruction "breaks." This targeted sampling achieves high-quality results with fewer images than uniform grids require.
Evaluation and Results
A user study with 20 participants compared the Error Peaking method against the industry-standard LLFF prescriptive grid approach.
- User Experience: Participants reported significantly higher satisfaction (6.0/7 vs. 4.5/7) and felt better able to focus on the scene (6.5/7) rather than the interface.
- Efficiency: The error peaking method required an average of 11.0 images, compared to 14.0 for the grid method.
- Reconstruction Quality: Quantitative analysis using 3D Gaussian Splatting (3DGS) showed that our sampling strategy outperformed both random and uniform sampling in terms of PSNR and LPIPS metrics.
Conclusion and Future Work
While effective, the system currently struggles with non-Lambertian surfaces like metallic or reflective materials, which can trigger false error peaks. Future work will focus on reducing server latency (currently ~3.6s) to move processing entirely on-device and incorporating human factors, such as foveal vision prioritization, to further refine the error visualization.
Summary updated