Semantic Augmented Reality Environment with Material-Aware Physical Interactions

Long Chen, Karl Francis, Wen Tang

Introduction

Current Augmented Reality (AR) systems excel at geometric understanding—knowing where surfaces are—but often lack semantic awareness—knowing what those surfaces are made of. This limitation results in virtual objects interacting with the physical world in a generic manner; for instance, a virtual ball might bounce off a marble floor and a plush rug with identical physics. To achieve true immersion, virtual entities must recognize real-world materials to exhibit realistic physical behaviors, such as material-specific sounds and visual effects. This paper proposes a framework to bridge the gap between raw 3D geometry and high-level material semantics in real-time.

Methodology

The proposed pipeline integrates computer vision, deep learning, and 3D mapping to create a material-aware environment.

  1. 3D Reconstruction: The system utilizes KinectFusion to generate a dense 3D geometric model of the environment using depth sensor data.
  2. Material Recognition: A VGG-16 network, converted into a Fully Convolutional Network (FCN), performs pixel-wise segmentation on 2D camera frames. This allows the system to label surfaces as specific materials like wood, glass, or fabric.
  3. Semantic Fusion: To ensure temporal and spatial consistency, Conditional Random Fields (CRF) are used to map 2D material labels onto the 3D geometric model.
  4. Physics Integration: The resulting semantic 3D map is imported into Unity. An oct-tree data structure enables efficient ray-casting, allowing the game engine to query surface materials instantly during physical interactions.
The pipeline for fusing 2D material segmentation with 3D KinectFusion models to create a semantic environment.

Key Insights

  • Material-Centric Semantics: Unlike traditional object detection (e.g., identifying a "table"), this approach prioritizes material identification (e.g., "wood"). For physics engines, material properties like friction and restitution are more critical than object labels.
  • Dual-Layer World Representation: The system maintains a visible RGB feed for the user while simultaneously processing a hidden, material-coded 3D model that governs physical logic.
  • Real-Time Efficiency: By utilizing oct-trees for the semantic model, the system achieves the low latency required for interactive applications, such as 60fps gaming.

Experimental Results

The framework was validated through a prototype AR shooting game. The system demonstrated dynamic, material-specific responses: shooting a wooden cabinet produced wood chips and a "thud," while hitting a window resulted in glass shards and a "clink." The semantic fusion remained stable even during camera movement, providing a persistent interactive environment.

Material-aware interactions in a prototype game: (a) wood impacts, (b) glass impacts, (c) fabric impacts, and (d) the semantic segmentation map.

Limitations and Future Work

While the system successfully demonstrates semantic-aware AR, it faces challenges regarding computational load when running deep FCNs alongside dense SLAM. Additionally, real-world environments contain complex composite materials that may require more sophisticated classification models. Future research could explore advanced physical simulations, such as virtual liquids interacting with porous real-world surfaces or thermal propagation across different materials.

Summary updated

Abstract

In Augmented Reality (AR) environment, realistic interactions between the virtual and real objects play a crucial role in user experience. Much of recent advances in AR has been largely focused on developing geometry-aware environment, but little has been done in dealing with interactions at the semantic level. High-level scene understanding and semantic descriptions in AR would allow effective design of complex applications and enhanced user experience. In this paper, we present a novel approach and a prototype system that enables the deeper understanding of semantic properties of the real world environment, so that realistic physical interactions between the real and the virtual objects can be generated. A material-aware AR environment has been created based on the deep material learning using a fully convolutional network (FCN). The state-of-the-art dense Simultaneous Localisation and Mapping (SLAM) has been used for the semantic mapping. Together with efficient accelerated 3D ray casting, natural and realistic physical interactions are generated for interactive AR games. Our approach has significant impact on the future development of advanced AR systems and applications.
BibTeX
@inproceedings{chen2017semantic,
  author = {Long Chen and Karl Francis and Wen Tang},
  title = {{Semantic Augmented Reality Environment with Material-Aware Physical Interactions}},
  booktitle = {ISMAR Adjunct},
  year = {2017},
  url = {https://openreview.net/forum?id=mOECjY2n3_},
}
Figure