Abstract
The provisioning of immersive point cloud video (PCV) streaming on pervasive mobile devices is a cornerstone for enabling immersive communication and interactions in the future 6G metaverse era. However, most streaming techniques are dedicated to efficient PCV compression and codec extending from traditional 3-DoF video services. Some emerging AI-enabled approaches are still in their infancy phase and are constrained by intensive computational and adaptive flow techniques. In this paper, we present ISCom, an Interest-aware Semantic Communication Scheme for PCV, consisting of a region-of-interest (ROI) selection module, a lightweight PCV streaming module, and an intelligent scheduler. First, we propose a two-stage efficient ROI selection method for providing interest-aware PCV streaming, which significantly reduces the data volume. Second, we design a lightweight PCV encoder-decoder network for resource-constrained devices, adapting to the heterogeneous computing capabilities of terminals. Third, we train a deep reinforcement learning (DRL)-based scheduler to adapt an optimal encoder-decoder network for various devices, considering the dynamic network environments and computing capabilities of different devices. Extensive experiments show that ISCom outperforms baselines on mobile devices at least 10 FPS and up to 22 FPS.
Introduction
Point Cloud Video (PCV) is essential for immersive "6-DoF" (6 Degrees of Freedom) experiences in the Metaverse, allowing users to move around and interact with 3D objects. However, PCV is a data-intensive medium, requiring bandwidth ranging from 1 Gbps to 1 Tbps and demanding ultra-low latency (<5ms). Standard compression methods, such as MPEG’s V-PCC, are often too computationally heavy for smartphones, frequently taking minutes to decode a single second of video. While newer AI-driven "semantic communication" methods show promise by transmitting key features instead of raw points, they are typically too resource-heavy for mobile hardware, leading to low frame rates and high battery drain. ISCom aims to bridge this gap by making semantic communication efficient enough for real-time mobile use.
System Architecture
ISCom addresses the mobile decoding bottleneck through a three-pronged architecture designed for efficiency and adaptability.
1. Two-Stage ROI Selection
Instead of streaming the entire 3D scene, ISCom identifies the Region of Interest (ROI) to reduce data volume:
- Coarse-grained: Uses the device's IMU sensors to predict user head movement and crops the "view frustum." It also utilizes FlowNet3D to identify "dynamic saliency," prioritizing moving objects.
- Fine-grained: Analyzes "static saliency" based on texture and geometric complexity to further downsample less critical background areas.
2. Lightweight Semantic Codec
The system employs a GAN-based encoder-decoder optimized through joint pruning and quantization. By stripping unnecessary neural network weights and converting 32-bit floating-point numbers into 8-bit or 16-bit integers, the model size is drastically reduced, enabling faster inference on mobile CPUs.
3. Intelligent DRL Scheduler
To handle varying network conditions and device capabilities, ISCom uses a Deep Reinforcement Learning (DRL) agent (specifically the A3C algorithm). This scheduler senses the environment and automatically selects the optimal encoder-decoder model version to balance video quality against frame rate.
Key Results
Experimental evaluations using the S3DIS and 8iVFB datasets demonstrate that ISCom significantly outperforms traditional and AI-based baselines.
- Real-Time Performance: ISCom achieved between 10 and 22 FPS on mobile devices, whereas traditional AI baselines like AITransfer struggled at roughly 1–2 FPS.
- Resource Efficiency: Compared to other AI-driven methods, ISCom reduced CPU usage by 47.5% and memory consumption by 41.7%. The lightweight models also kept device temperatures lower (approx. 42°C) compared to heavier AI models (47°C+).
- Bandwidth Reduction: The ROI selection module alone reduced input data size by approximately 30x (from 34.4 MB to 1.1 MB per segment) without sacrificing the quality of the user's focal area.
Conclusion and Future Work
ISCom successfully shifts the focus from raw compression to efficient decoding and interest-aware transmission. While the current framework requires caching multiple pre-trained models on-device, future work will explore incorporating temporal features (motion vectors) to further enhance compression ratios and investigate scalability for multi-user metaverse environments.
Summary updated