Poster: Real-Time Object Substitution for Mobile Diminished Reality with Edge Computing

Hongyu Ke, Haoxin Wang

Abstract

Diminished Reality (DR) is the conceptual counterpart to Augmented Reality (AR), allowing for the removal of physical content from the real world rather than just adding virtual overlays. When combined with object replacement, DR offers a powerful tool for scene construction in the metaverse. However, achieving high-quality, real-time object substitution on mobile devices remains a challenge due to high computational demands. This paper proposes an end-to-end architecture that leverages edge computing and intelligent optimization modules to facilitate immersive and real-time scene reconstruction for mobile users.

Introduction

While Augmented Reality (AR) has become a staple of mobile technology, its primary function is additive—placing digital objects into a physical environment. Diminished Reality (DR) addresses the subtractive side of this equation. High-quality object substitution requires a seamless transition where a physical object is first "diminished" (removed and the background inpainted) before a virtual replacement is added. Without effective DR, a virtual replacement that is smaller than the original physical object would result in visual artifacts, where parts of the original object remain visible.

The Challenge of Mobile DR

The authors identify two primary hurdles for mobile DR: computational cost and latency. Calculating 3D poses for moving objects and performing real-time 2D inpainting to fill the "holes" left by removed objects are tasks that typically exceed the processing power of standard mobile hardware. To maintain an immersive experience, the system must aim for 30 frames per second (fps), a target that is difficult to hit without significant lag or battery drain.

Proposed Architecture

The proposed system splits the workload between the mobile device and an Edge Server, utilizing two parallel pipelines to handle the "Eraser" and "Replacer" functions:

  1. The 2D Pipeline (The Eraser): This pipeline handles the DR aspect. It uses instance segmentation to identify the target object and 2D inpainting to fill the resulting void with a plausible background. This ensures the scene remains natural even if the replacement object does not perfectly cover the original footprint.
  2. The 3D Pipeline (The Replacer): This pipeline performs 3D pose estimation on the physical object. By mapping these coordinates to the virtual replacement, the digital entity "inherits" the movement and orientation of its real-world counterpart, a concept the authors call semantic inheritance.
The object substitution pipeline: original frame, segmentation mask, and virtual object replacement.

Optimization Strategies

To bridge the gap to real-time performance, the architecture introduces two key optimization modules designed to exploit temporal redundancy in video streams:

  • Frame Passer: This module analyzes consecutive frames for similarity. If the background remains static, the system bypasses the 2D segmentation module and uses cached data, significantly reducing processing time.
  • Early Stop: This module monitors 3D pose estimation. If movement between frames is negligible, the system stops the calculation early and applies the previous frame's pose to the current one.

Conclusion and Future Work

This research provides a qualitative framework for achieving seamless, low-latency object substitution. By offloading heavy tasks to the edge and implementing smart bypass modules, the system moves closer to the 30fps threshold required for a "senseless" latency experience.

Future work will focus on the full implementation and quantitative benchmarking of the architecture. Key challenges remain, including establishing industry-standard metrics for DR quality and expanding the system to support collaborative, multi-user environments where synchronized views of a substituted reality are required.

Summary updated

Abstract

Diminished Reality (DR) is considered as the conceptual counterpart to Augmented Reality (AR), and has recently gained increasing attention from both industry and academia. Unlike AR which adds virtual objects to the real world, DR allows users to remove physical content from the real world. When combined with object replacement technology, it presents an further exciting avenue for exploration within the metaverse. Although a few researches have been conducted on the intersection of object substitution and DR, there is no real-time object substitution for mobile diminished reality architecture with high quality. In this paper, we propose an end-to-end architecture to facilitate immersive and real-time scene construction for mobile devices with edge computing.
BibTeX
@inproceedings{ke2023poster,
  author = {Hongyu Ke and Haoxin Wang},
  title = {{Poster: Real-Time Object Substitution for Mobile Diminished Reality with Edge Computing}},
  booktitle = {Sec},
  year = {2023},
  doi = {10.1145/3583740.3628422},
}
Figure