ISCom: Interest-aware Semantic Communication Scheme for Point Cloud Video Streaming

Yakun Huang, Boyuan Bai, Yuanwei Zhu, Xiuquan Qiao, Xiang Su, Ping Zhang

Abstract

The provisioning of immersive point cloud video (PCV) streaming on pervasive mobile devices is a cornerstone for enabling immersive communication and interactions in the future 6G metaverse era. However, most streaming techniques are dedicated to efficient PCV compression and codec extending from traditional 3-DoF video services. Some emerging AI-enabled approaches are still in their infancy phase and are constrained by intensive computational and adaptive flow techniques. In this paper, we present ISCom, an Interest-aware Semantic Communication Scheme for PCV, consisting of a region-of-interest (ROI) selection module, a lightweight PCV streaming module, and an intelligent scheduler. First, we propose a two-stage efficient ROI selection method for providing interest-aware PCV streaming, which significantly reduces the data volume. Second, we design a lightweight PCV encoder-decoder network for resource-constrained devices, adapting to the heterogeneous computing capabilities of terminals. Third, we train a deep reinforcement learning (DRL)-based scheduler to adapt an optimal encoder-decoder network for various devices, considering the dynamic network environments and computing capabilities of different devices. Extensive experiments show that ISCom outperforms baselines on mobile devices at least 10 FPS and up to 22 FPS.

Introduction

Point Cloud Video (PCV) is essential for immersive "6-DoF" (6 Degrees of Freedom) experiences in the Metaverse, allowing users to move around and interact with 3D objects. However, PCV is a data-intensive medium, requiring bandwidth ranging from 1 Gbps to 1 Tbps and demanding ultra-low latency (<5ms). Standard compression methods, such as MPEG’s V-PCC, are often too computationally heavy for smartphones, frequently taking minutes to decode a single second of video. While newer AI-driven "semantic communication" methods show promise by transmitting key features instead of raw points, they are typically too resource-heavy for mobile hardware, leading to low frame rates and high battery drain. ISCom aims to bridge this gap by making semantic communication efficient enough for real-time mobile use.

System Architecture

ISCom addresses the mobile decoding bottleneck through a three-pronged architecture designed for efficiency and adaptability.

The ISCom framework architecture, illustrating the server-side ROI selection and the device-side lightweight decoding process managed by an intelligent DRL scheduler.

1. Two-Stage ROI Selection

Instead of streaming the entire 3D scene, ISCom identifies the Region of Interest (ROI) to reduce data volume:

  • Coarse-grained: Uses the device's IMU sensors to predict user head movement and crops the "view frustum." It also utilizes FlowNet3D to identify "dynamic saliency," prioritizing moving objects.
  • Fine-grained: Analyzes "static saliency" based on texture and geometric complexity to further downsample less critical background areas.

2. Lightweight Semantic Codec

The system employs a GAN-based encoder-decoder optimized through joint pruning and quantization. By stripping unnecessary neural network weights and converting 32-bit floating-point numbers into 8-bit or 16-bit integers, the model size is drastically reduced, enabling faster inference on mobile CPUs.

3. Intelligent DRL Scheduler

To handle varying network conditions and device capabilities, ISCom uses a Deep Reinforcement Learning (DRL) agent (specifically the A3C algorithm). This scheduler senses the environment and automatically selects the optimal encoder-decoder model version to balance video quality against frame rate.

Key Results

Experimental evaluations using the S3DIS and 8iVFB datasets demonstrate that ISCom significantly outperforms traditional and AI-based baselines.

Performance comparison of ISCom against baselines (Octree, Draco, G-PCC, AITransfer) across encoding/decoding times, transmission latency, and frame rate.
  • Real-Time Performance: ISCom achieved between 10 and 22 FPS on mobile devices, whereas traditional AI baselines like AITransfer struggled at roughly 1–2 FPS.
  • Resource Efficiency: Compared to other AI-driven methods, ISCom reduced CPU usage by 47.5% and memory consumption by 41.7%. The lightweight models also kept device temperatures lower (approx. 42°C) compared to heavier AI models (47°C+).
  • Bandwidth Reduction: The ROI selection module alone reduced input data size by approximately 30x (from 34.4 MB to 1.1 MB per segment) without sacrificing the quality of the user's focal area.

Conclusion and Future Work

ISCom successfully shifts the focus from raw compression to efficient decoding and interest-aware transmission. While the current framework requires caching multiple pre-trained models on-device, future work will explore incorporating temporal features (motion vectors) to further enhance compression ratios and investigate scalability for multi-user metaverse environments.

Summary updated

Abstract

The provisioning of immersive point cloud video (PCV) streaming on pervasive mobile devices is a cornerstone for enabling immersive communication and interactions in the future 6G metaverse era. However, most streaming techniques are dedicated to efficient PCV compression and codec extending from traditional 3-DoF video services. Some emerging AI-enabled approaches are still in their infancy phase and are constrained by intensive computational and adaptive flow techniques. In this paper, we present ISCom, an Interest-aware Semantic Communication Scheme for PCV, consisting of a region-of-interest (ROI) selection module, a lightweight PCV streaming module, and an intelligent scheduler. First, we propose a two-stage efficient ROI selection method for providing interest-aware PCV streaming, which significantly reduces the data volume. Second, we design a lightweight PCV encoder-decoder network for resource-constrained devices, adapting to the heterogeneous computing capabilities of terminals. Third, we train a deep reinforcement learning (DRL)-based scheduler to adapt an optimal encoder-decoder network for various devices, considering the dynamic network environments and computing capabilities of different devices. Extensive experiments show that ISCom outperforms baselines on mobile devices at least 10 FPS and up to 22 FPS.
BibTeX
@misc{huang2022iscom,
  author = {Yakun Huang and Boyuan Bai and Yuanwei Zhu and Xiuquan Qiao and Xiang Su and Ping Zhang},
  title = {{ISCom: Interest-aware Semantic Communication Scheme for Point Cloud Video Streaming}},
  year = {2022},
  eprint = {2210.06808},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2210.06808},
}
Figure