Automatic Adjustment of Stereoscopic Content for Long-Range Projections in Outdoor Areas

Behnam Maneshgar, Leila Sujir, Sudhir P. Mudur, Charalambos PoullisACM Multimedia 2017 · DOI

TL;DR

This paper presents a novel framework for automatically adjusting stereoscopic content (specifically anaglyph 3D) for projection onto large, complex outdoor surfaces. By accurately modeling the projection surface's geometry and light-reflecting properties, their system significantly improves the perceived depth and color quality for viewers, overcoming limitations of traditional projection mapping techniques.

What Problem Does This Paper Address?

Projecting stereoscopic content, especially anaglyph 3D, onto large outdoor surfaces like building facades presents significant challenges. Traditional anaglyph 3D relies on red and cyan color encoding, which can be severely distorted by the surface's color and reflectivity, leading to poor or non-existent 3D perception. Furthermore, accurately capturing the geometry and light behavior of these large, distant surfaces is difficult with standard calibration techniques that assume close-range, in-focus imaging. This paper tackles the problem of compensating for these adverse projection surface characteristics to deliver a high-quality stereoscopic experience in real-world outdoor settings.

Methods

The proposed framework comprises four key modules:

  • Long-Range System Calibration: Instead of relying on traditional calibration boards, this system uses Structure-from-Motion (SfM) on images captured by multiple cameras (two fixed, one roving) to estimate camera poses and intrinsic parameters. This overcomes the challenges of focusing on distant objects.
  • Dense 3D Reconstruction: Structured Light Scanning (SLS) is employed using the calibrated cameras and a projector to generate a dense 3D point cloud of the projection surface. This point cloud is then meshed to represent the surface's geometry.
  • Reflectance Function Estimation: The Bidirectional Reflectance Distribution Function (BRDF) of the projection surface is modeled using the LaFortune BRDF model. This is achieved by analyzing per-pixel samples captured by the roving camera and estimating diffuse and specular components, effectively modeling how light interacts with the surface.
  • Automatic Adjustment: The original stereoscopic content is iteratively rendered from the projector's viewpoint, taking into account the reconstructed geometry and estimated BRDF. An optimization process adjusts the content to minimize the difference between the rendered and original images, compensating for distortions caused by the surface.

Datasets

The paper does not explicitly mention the use of pre-existing datasets for training. Instead, it relies on data captured in real-world scenarios for its experiments. The "datasets" are effectively the captured images and videos of the projection surfaces and the projected content during user studies.

Key Insights and Ideas

  • SfM for Long-Range Calibration: The insight that SfM can robustly calibrate cameras for long-range projection mapping, bypassing the need for calibration boards and their associated focusing issues, is crucial.
  • Combining SfM and SLS: The synergistic use of SfM for camera pose estimation and SLS for dense surface reconstruction provides a powerful pipeline for accurately capturing complex outdoor environments.
  • Pixel-Level BRDF Modeling: Capturing and modeling the reflectance properties at a per-pixel level, especially using a roving camera for diverse viewpoints, is key to accurately compensating for surface color and light interaction.
  • Iterative Rendering for Adjustment: The iterative rendering and adjustment process, driven by minimizing perceptual differences, is an effective way to "undo" the distortions introduced by the projection surface.

Key Results

The framework was evaluated through two user experience studies involving projection mapping onto building facades.

  • Qualitative Improvements: Visual examples (Figure 5 and 8) demonstrate a significant improvement in the visibility and perceived quality of the projected stereoscopic content after adjustment, particularly on surfaces with challenging colors (e.g., red sheds).
  • Quantitative Improvements: Statistical analysis of participant responses showed a high degree of statistical significance in improved color and depth perception after the adjustment process. For instance, in Experiment #1, a substantial percentage of participants who rated their perception poorly before the adjustment rated it significantly higher afterward.
  • Robustness: The system demonstrated effectiveness in both short-range (3.3m) and long-range (over 20m) outdoor projection scenarios.

Limitations and Future Work

  • Ambient Lighting: The paper acknowledges that excessive ambient lighting in outdoor scenes can still pose challenges, particularly for capturing reflected light accurately.
  • General Light Scattering: The current BRDF model primarily captures reflected light. Extending the system to model more general light scattering behavior (e.g., through glass windows) is a future direction.
  • Computational Efficiency: While the iterative adjustment process is effective, it is computationally intensive (e.g., 4 minutes for a 1600x1200 image). Optimizing computations, potentially through clustering surface properties or leveraging multi-core/GPU processing, is suggested.
  • Material Property Coherence: Exploring the coherence of material properties within clustered regions of the projection surface could lead to more efficient and robust computations.

Summary updated

BibTeX
@inproceedings{maneshgar2017automatic,
  author = {Behnam Maneshgar and Leila Sujir and Sudhir P. Mudur and Charalambos Poullis},
  title = {{Automatic Adjustment of Stereoscopic Content for Long-Range Projections in Outdoor Areas}},
  booktitle = {ACM Multimedia},
  year = {2017},
  doi = {10.1145/3123266.3123269},
}
Figure