Poster: Real-Time Object Substitution for Mobile Diminished Reality with Edge Computing

Haoxin Wang, Hongyu Ke, Haoxin Wang 0003

TL;DR

This paper proposes an end-to-end architecture for "Diminished Reality" (DR) on mobile devices, allowing users to seamlessly remove real-world objects and replace them with virtual ones in real-time. By offloading heavy computation to edge servers and implementing smart frame-skipping modules, the system aims to bridge the gap between static AR overlays and truly interactive, science-fiction-style scene reconstruction.

What Problem Does This Paper Address?

While Augmented Reality (AR) has mastered the art of overlaying digital content onto the physical world, it struggles with the opposite: removing or replacing what is already there. This is known as Diminished Reality (DR).

The authors identify a significant hurdle in current mobile AR: if you want to replace a real-world pedestrian with a cyberpunk robot, simply "pasting" the robot over the person often fails. If the robot is smaller than the person, the original human still peeks out from behind. Furthermore, calculating the 3D pose of a moving object and "inpainting" the background where they used to be is computationally expensive. Doing this on a mobile device in real-time (aiming for 30fps) without massive lag or battery drain is currently an unsolved challenge.

Methods

The authors propose a system that splits the workload between a mobile device and an Edge Server using two parallel pipelines:

  1. The 2D Pipeline (The "Eraser"): This handles the Diminished Reality aspect. It uses Instance Segmentation to identify the object to be removed and 2D Inpainting to fill the resulting "hole" with a plausible background. This ensures that even if the replacement object is smaller than the original, the scene looks natural.
  2. The 3D Pipeline (The "Replacer"): This performs 3D Pose Estimation on the physical object and maps those coordinates to the virtual replacement. This allows the virtual object to "inherit" the movement and orientation of the real-world counterpart.

To make this run in real-time, the authors introduce two optimization modules:

  • Frame Passer: This module checks if consecutive frames are similar. If the background hasn't changed, it uses cached data to bypass the 2D segmentation module, saving precious processing time.
  • Early Stop: This monitors the 3D pose estimation. If the movement between frames is negligible, it stops the calculation early and assigns the previous frame's pose to the current one.

Datasets

Not discussed in the paper. As this is a poster focused on system architecture and initial design, specific training datasets or benchmark suites were not detailed.

Key Insights and Ideas

  • DR as a Prerequisite for Substitution: The authors argue that high-quality object substitution is impossible without DR. You cannot simply "add" a virtual object; you must first "subtract" the physical one to avoid occlusion mismatches and visual artifacts.
  • Semantic Inheritance: By linking the 3D pose of the virtual object to the physical one, the virtual entity inherits the "semantics" of the real world. A virtual avatar doesn't just stand there; it moves, walks, or plays basketball exactly as the real person does.
  • Exploiting Temporal Redundancy: The "Frame Passer" and "Early Stop" modules recognize that video is often redundant. By identifying when not to compute, the system can significantly reduce the latency introduced by offloading tasks to the edge.

Key Results

As this is a proposal for an initial design, the paper does not provide quantitative performance metrics (like exact milliseconds of latency or accuracy percentages). Instead, it provides a qualitative framework for achieving:

  • Seamless Integration: Combining inpainting with pose-mapped rendering to ensure virtual objects look grounded.
  • Latency Reduction: Using edge computing and bypass modules to move toward a "senseless" latency level (30fps).
  • Environmental Flexibility: The inpainting approach allows the system to work without needing a pre-captured map of the environment.

Limitations and Future Work

The authors are transparent about the hurdles ahead:

  • The 30fps Barrier: Even with edge offloading, maintaining a consistent 30fps for complex segmentation and inpainting tasks remains a significant engineering challenge.
  • Lack of Metrics: There is currently no industry-standard way to quantitatively measure the "quality" of a Diminished Reality scene.
  • Single-User Focus: The current design is for one user. Expanding this to Collaborative AR/DR, where multiple users see the same substituted reality from different angles, introduces massive synchronization and data-sharing complexities.
  • Implementation: The next step for the researchers is the full implementation of the edge-server architecture and the "Frame Passer/Early Stop" modules to test their theoretical efficiency gains.

Summary updated

Abstract

Diminished Reality (DR) is considered as the conceptual counterpart to Augmented Reality (AR), and has recently gained increasing attention from both industry and academia. Unlike AR which adds virtual objects to the real world, DR allows users to remove physical content from the real world. When combined with object replacement technology, it presents an further exciting avenue for exploration within the metaverse. Although a few researches have been conducted on the intersection of object substitution and DR, there is no real-time object substitution for mobile diminished reality architecture with high quality. In this paper, we propose an end-to-end architecture to facilitate immersive and real-time scene construction for mobile devices with edge computing.
BibTeX
@inproceedings{wang2023poster,
  author = {Haoxin Wang and Hongyu Ke and Haoxin Wang},
  title = {{Poster: Real-Time Object Substitution for Mobile Diminished Reality with Edge Computing}},
  booktitle = {Sec},
  year = {2023},
  doi = {10.1145/3583740.3628422},
}

Loading related content...

Figure