A Work Area Visualization by Multi-View Camera-Based Diminished Reality

Shohei Mori, Momoko Maezawa, Hideo SaitoMultimodal Technol. Interact 2017 · DOI

TL;DR

This paper introduces a novel Diminished Reality (DR) system that uses multiple cameras to visualize occluded work areas in real-time, specifically for hand-held tool operations. By redesigning an existing arbitrary viewpoint image generation method, their system effectively removes visual obstructions like hands and tools, offering a clearer view of the task at hand and outperforming conventional methods in image similarity.

What Problem Does This Paper Address?

Many manual tasks, from woodworking to surgery, rely on hand-held tools. However, these tools often obscure the very area the user needs to see, creating a significant visual occlusion problem. While augmented reality (AR) can overlay information, Diminished Reality (DR) aims to make these occluding objects "invisible" to reveal the hidden background. Existing DR methods often struggle with dynamic scenes, require cumbersome manual viewpoint conversions, or are computationally expensive, limiting their real-time applicability. This paper tackles the challenge of creating a real-time DR system that can effectively visualize dynamic work areas while preserving view-dependent properties, a feat not yet achieved by prior work.

Methods

The core of the proposed system is a redesigned arbitrary viewpoint image generation method, specifically an adaptation of unstructured lumigraph rendering (ULR) [8]. The pipeline involves several key steps:

  • Multi-View Camera Calibration: Calibrating RGB-D and conventional cameras to obtain intrinsic and extrinsic parameters, aligning them to a reference RGB-D camera.
  • Camera Blending Field (CBF) Calculation: This is a crucial step where blending weights are calculated for multiple cameras based on the worker's viewpoint, data camera positions, surface points, and "penalty points" representing the object to be diminished. This ensures that rays passing near the occluding object are de-emphasized.
  • Surface Extraction: Extracting foreground and background depth maps from depth frames. This involves back-projecting depth maps to 3D point clouds, transforming them to the worker's viewpoint, and using techniques like surface splatting [18] and past depth frames to fill in missing regions.
  • Region of Interest (ROI) Detection: Identifying the object to be removed (e.g., a drill) using ellipse detection [20] on the generated mask image. The center points of these ellipses are used as penalty points.
  • View Synthesis: Generating the final diminished reality frame. Images from the most weighted data cameras are projected onto triangle meshes (geometric proxy) representing the background. Alpha blending is used to smoothly integrate the rendered background, with special handling for regions near the ROI to create gradual transitions.

The system is designed to handle dynamic backgrounds and preserve view-dependent properties, which is a significant advancement over previous DR methods.

Datasets

The paper does not explicitly mention using pre-existing public datasets for training. Instead, they collected their own data for experiments. They captured multi-view images and depth data using a Microsoft Kinect sensor and three USB cameras in various non-planar work scenarios (e.g., "Box," "Laptop," "Keyboards"). They also created "ground truth" images by capturing scenes without the occluding object.

Key Insights and Ideas

  • Redefining ULR for DR: The paper demonstrates that an existing free-viewpoint image generation technique (ULR) can be effectively adapted for Diminished Reality by incorporating penalty points and a specific blending strategy. This is a key insight into repurposing established computer vision techniques for new applications.
  • Handling Dynamic Scenes: The method's ability to process dynamic backgrounds, where objects and the scene itself are in motion, is a significant advantage over many static-background DR approaches.
  • View-Dependent Properties: Preserving view-dependent properties means the synthesized background looks realistic from the user's perspective, adapting to changes in lighting and surface appearance as the viewpoint shifts. This is crucial for an immersive and intuitive DR experience.
  • Real-time Performance: Achieving around 40.1 fps is a critical insight, as it makes the system practical for real-time use during manual tasks, unlike many prior methods that required several seconds per frame.

Key Results

  • Qualitative Evaluation: Visual comparisons (Figure 4 and 6) show that the proposed method produces more natural and complete backgrounds compared to Photoshop Content-Aware Fill (PS) and surface splatting (SS). PS often resulted in unnatural patchwork, while SS suffered from black regions due to unobservable pixels. The proposed method successfully filled occluded regions and maintained fine textures.
  • Quantitative Evaluation: Similarity measures (NCC, PSNR, SSIM) were used to compare the proposed method against PS and SS.
  • NCC and SSIM: The proposed method significantly outperformed both PS and SS in Normalized Cross-Correlation (NCC) and Structural Similarity (SSIM) measures, with statistical analysis confirming these differences. The hypotheses H1 (our methods exceed others) and H2 (PS < SS < Ours) were supported for NCC and SSIM.
  • PSNR: While PSNR showed less significant differences, the mean values still indicated an improvement, though the authors note PSNR's sensitivity to small misalignments.
  • Real-time Performance: The system achieved a frame rate of approximately 40.1 fps, confirming its real-time capability.

Limitations and Future Work

  • Camera Movement: The current system assumes the user camera and data cameras are static. Allowing these cameras to move would significantly expand the system's applicability. Integrating multi-camera collaborative methods is suggested for this.
  • DR Representation: A key limitation is the potential for user confusion when the occluding object is completely removed, making the user's hand and tool invisible. The paper discusses semi-transparent representations (like AR X-ray) but argues they don't fully solve the problem, especially for critical information like the drill tip's position. Developing better DR representations remains an open area for future work.
  • Geometric Issues: The quality of the DR image relies heavily on the accuracy of surface estimation. While multiple RGB-D cameras could improve this, interference issues are noted. Exploring partial depth-searching for limited regions is suggested to reduce computational costs.

Summary updated

BibTeX
@article{mori2017work,
  author = {Shohei Mori and Momoko Maezawa and Hideo Saito},
  title = {{A Work Area Visualization by Multi-View Camera-Based Diminished Reality}},
  journal = {Multimodal Technol. Interact.},
  year = {2017},
  doi = {10.3390/mti1030018},
}
Figure