Millimeter Wave MIMO-Based Depth Maps for Wireless Virtual and Augmented Reality

Abdelrahman Taha, Qi Qu, Sam Alex, Ping Wang, William L. Abbott, Ahmed Alkhateeb

TL;DR

This paper proposes a novel method for creating accurate depth maps for AR/VR systems by leveraging existing millimeter-wave (mmWave) MIMO communication transceivers. This approach overcomes limitations of current RGB-D cameras, especially in challenging scenarios like shiny or dark surfaces, by using mmWave signals for sensing.

What Problem Does This Paper Address?

Immersive AR/VR experiences rely heavily on accurate depth perception of the surrounding environment. Current AR/VR devices typically use RGB-D cameras for this, but these cameras struggle with certain surfaces (shiny, dark, transparent) and abrupt lighting changes. The paper aims to address these limitations by proposing a new depth sensing method that utilizes mmWave MIMO communication systems, which are already being integrated into future AR/VR devices for high data rates.

Methods

The core of the proposed solution is a framework that leverages mmWave MIMO transceivers for depth map construction. Key components include:

  • mmWave MIMO Depth Map Construction Framework: A general framework designed to build depth maps while accounting for the constraints of mmWave communication hardware and frame structures.
  • Depth-Map Suitable Sensing Beamforming Codebook: A novel codebook design tailored for efficient depth map construction, considering the characteristics of mmWave sensing.
  • High-Resolution Depth Map Construction Approach: A signal processing method that jointly processes signals received by sensing beams to generate high-resolution depth maps. This involves:
  • Oversampled/Overlapped Beams: Using oversampled sensing codebooks to scan the environment and increase potential depth map resolution.
  • Successive Interference Cancellation (SIC): An algorithm to manage inter-target and inter-path interference by detecting dominant channel paths.
  • Joint Processing (JP): A solution that combines signals from overlapped beams and uses surrounding grid point information to refine depth estimates and detect object/surface transitions.
  • Sidelobe Reduction (SLR) Approach: An efficient technique to reduce sidelobe levels in beamforming patterns, mitigating inter-path interference.

Datasets

The paper uses simulated data generated using:

  • Remcom Wireless InSite: A 3D ray-tracing simulator for generating realistic channel models.
  • Blender: A game engine used to create detailed 3D floor plans and generate ground truth depth maps.
  • DeepMIMO dataset generation code: Used for generating channel paths.

Key Insights and Ideas

  • Leveraging Existing Infrastructure: The paper demonstrates that mmWave communication systems, already slated for AR/VR, can be repurposed for depth sensing without additional hardware costs, making it a cost-effective solution.
  • Overcoming RGB-D Limitations: mmWave signals are less affected by lighting conditions and can better penetrate certain materials, offering a significant advantage over RGB-D cameras for challenging surfaces.
  • Codebook Design for Depth: A custom beamforming codebook is crucial for matching the desired rectangular grid of depth sensors, avoiding distortions caused by traditional parabolic grids.
  • Joint Processing for Accuracy: Combining signals from multiple beams and using information from neighboring grid points (JP) significantly improves depth estimation accuracy and the detection of scene transitions.
  • Sidelobe Reduction for Interference: Effectively managing interference from undesired channel paths is critical, and the proposed SLR approach helps in this regard.

Key Results

  • The proposed mmWave MIMO based depth estimation approach can construct relatively high-resolution depth maps with less than 10cm error in practical scenarios.
  • In a conference room scenario, the mmWave MIMO estimator outperforms RGB-based estimators in recognizing transparent and dark objects. For instance, a glass wall was not detected by the RGB estimator but was partially recovered by the mmWave system.
  • The approach shows robustness in low illumination scenarios and for surfaces with low scattering ratios.
  • Increasing the number of antennas and/or codebook oversampling factors generally improves depth estimation accuracy, though at the cost of increased sensing time and computational complexity.
  • The system demonstrates good performance even at relatively low Signal-to-Noise Ratios (SNRs). For example, a transmit power of 5dBm (SNR of -10dB) was sufficient for around 10cm error in depth estimation.

Limitations and Future Work

  • Computational Complexity and Sensing Time: Increasing the number of antennas or codebook oversampling factors improves accuracy but also increases sensing time and computational complexity.
  • Phase Quantization: While the impact is small, RF phase shifter quantization can affect the beam pattern shape and, consequently, depth estimation accuracy.
  • Future Work: The authors mention that accounting for self-interference in mmWave transceivers is an important direction for future extensions. Further research could also explore optimizing the trade-off between accuracy, sensing time, and computational resources. The paper also suggests that the proposed solution can be integrated with RGB-D depth cameras for enhanced performance.

Summary updated

Abstract

Augmented and virtual reality systems (AR/VR) are rapidly becoming key components of the wireless landscape. For immersive AR/VR experience, these devices should be able to construct accurate depth perception of the surrounding environment. Current AR/VR devices rely heavily on using RGB-D depth cameras to achieve this goal. The performance of these depth cameras, however, has clear limitations in several scenarios, such as the cases with shiny objects, dark surfaces, and abrupt color transition among other limitations. In this paper, we propose a novel solution for AR/VR depth map construction using mmWave MIMO communication transceivers. This is motivated by the deployment of advanced mmWave communication systems in future AR/VR devices for meeting the high data rate demands and by the interesting propagation characteristics of mmWave signals. Accounting for the constraints on these systems, we develop a comprehensive framework for constructing accurate and high-resolution depth maps using mmWave systems. In this framework, we developed new sensing beamforming codebook approaches that are specific for the depth map construction objective. Using these codebooks, and leveraging tools from successive interference cancellation, we develop a joint beam processing approach that can construct high-resolution depth maps using practical mmWave antenna arrays. Extensive simulation results highlight the potential of the proposed solution in building accurate depth maps. Further, these simulations show the promising gains of mmWave based depth perception compared to RGB-based approaches in several important use cases.
BibTeX
@inproceedings{taha2021millimeter,
  author = {Abdelrahman Taha and Qi Qu and Sam Alex and Ping Wang and William L. Abbott and Ahmed Alkhateeb},
  title = {{Millimeter Wave MIMO-Based Depth Maps for Wireless Virtual and Augmented Reality}},
  booktitle = {IEEE Access},
  year = {2021},
  doi = {10.1109/access.2021.3067839},
}
Figure