Towards Holographic Video Communications: A Promising AI-driven Solution

Yakun Huang, Yuanwei Zhu, Xiuquan Qiao, Xiang Su, Schahram Dustdar, Ping Zhang

Abstract

Holographic Point Cloud Video (PCV) represents the next frontier in immersive media, offering six degrees of freedom (6-DoF) for a truly interactive experience. However, the massive data rates—often exceeding 2 Gbps—and high computational costs of traditional codecs like MPEG's V-PCC hinder real-time delivery on mobile devices. This paper presents a novel AI-driven end-to-end architecture that replaces traditional compression with a lightweight neural network. By extracting and reconstructing key semantic features, the proposed solution achieves a 30x compression ratio and significantly reduced latency, enabling real-time holographic communication on resource-constrained hardware.

Introduction: The Challenge of Real-Time Holography

Holographic video is often considered the "holy grail" of the metaverse, allowing users to walk around and view digital objects from any perspective. Unlike 2D or even 360-degree video, Point Cloud Video (PCV) consists of millions of 3D points, each with spatial coordinates and color information. This leads to staggering bandwidth requirements; a single second of raw PCV can saturate even the most advanced 5G networks.

Current industry standards, such as MPEG’s V-PCC, are primarily designed for offline storage rather than real-time streaming. Encoding a single second of video can take nearly an hour on a standard PC, and decoding is equally taxing for mobile GPUs. This creates a significant gap between the immersive vision of the future and the practical limitations of current hardware and network infrastructure.

Proposed AI-Native Architecture

To bridge this gap, the researchers propose an AI-native transmission pipeline that prioritizes semantic features over raw geometric data. Instead of a traditional "compress-then-transmit" approach, the system uses a deep learning framework to identify and transmit only the most essential characteristics of the 3D shape.

Overview of the proposed AI-driven end-to-end architecture, including offline training and online DRL-based streaming control.

The architecture is divided into three core components:

  1. Feature Extraction (Encoder): Utilizing a hierarchical structure based on PointNet++, the sender extracts a compact "feature vector." This vector represents the core geometry of the point cloud, drastically reducing the amount of data sent over the network.
  2. Lightweight Reconstruction (Decoder): On the receiver side, a lightweight Generative Adversarial Network (GAN) "hallucinates" the full point cloud from the received features. To ensure compatibility with mobile CPUs, the authors employed weight pruning and 8/16-bit quantization, removing the need for high-end desktop GPUs.
  3. DRL-based Streaming Controller: To handle fluctuating network conditions (from 3G to 5G) and varying device capabilities, a Deep Reinforcement Learning (DRL) controller was implemented. Using the A3C algorithm, it dynamically selects the optimal encoder-decoder model pair to maximize the user's Quality of Experience (QoE), balancing visual fidelity against latency.

Evaluation and Key Results

The system was trained on a dataset of 147 3D point cloud objects and evaluated using real-world human PCV sequences like "longdress" and "redandblack."

Visual comparison of point cloud reconstruction between the Draco compressor and the proposed AI-driven method.

The experimental results highlight several breakthroughs:

  • Compression and Fidelity: The AI-driven method achieved a maximum compression ratio of 30x. As shown in the visual comparison, while traditional methods like Draco produce "blocky" or non-uniform artifacts at high compression levels, the AI-driven approach maintains smooth, recognizable human forms by reconstructing points onto the target surface.
  • Latency Reduction: In constrained network environments (3G/4G), the proposed method reduced transmission latency by over 50% compared to Draco.
  • Mobile Viability: The optimized decoder allowed a mobile device with only two CPU cores to maintain a usable frame rate, a feat previously impossible with heavy traditional codecs.

Discussion and Future Directions

While this AI-driven approach marks a significant step forward, several challenges remain. Current capture systems are still limited to 10–15 FPS, falling short of the 30 FPS standard for fluid motion. Furthermore, the current model processes frames independently; future work will focus on incorporating temporal redundancy (inter-frame compression) to further lower data rates.

Other promising avenues include "interest-aware" capturing, where only the parts of the point cloud within the user's immediate field of view are transmitted, and optimizing the storage overhead required for caching multiple AI models on mobile devices. Despite these hurdles, this research provides a viable blueprint for bringing high-quality holographic communication to the palm of our hands.

Summary updated

Abstract

Real-time holographic video communications enable immersive experiences for next-generation video services in the future metaverse era. However, high-fidelity holographic videos require high bandwidth and significant computation resources, which exceed the transferring and computing capacity of 5G networks. This article reviews state-of-the-art holographic point cloud video (PCV) transmission techniques and highlights the critical challenges of delivering such immersive services. We further implement a preliminary prototype of an AI-driven holographic video communication system and present critical experimental results to evaluate its performance. Finally, we identify future research directions and discuss potential solutions for providing real-time and high-quality holographic experiences.
BibTeX
@article{huang2022towards,
  author = {Yakun Huang and Yuanwei Zhu and Xiuquan Qiao and Xiang Su and Schahram Dustdar and Ping Zhang},
  title = {{Towards Holographic Video Communications: A Promising AI-driven Solution}},
  journal = {IEEE Communications Magazine},
  year = {2022},
  doi = {10.1109/mcom.001.220021},
}
Figure