Progressive Frame Patching for FoV-based Point Cloud Video Streaming

Tongyu Zong, Yixiang Mao, Chen Li, Yong Liu, Yao Wang

TL;DR

Streaming high-fidelity 3D point cloud video (PCV) requires massive bandwidth and precise prediction of where a user is looking. This paper introduces a progressive "patching" framework that downloads multiple frames simultaneously in a sliding window, refining their quality as the playback time approaches and Field-of-View (FoV) prediction becomes more accurate.

What Problem Does This Paper Address?

Volumetric video—think holograms or immersive XR—is the next frontier of telepresence. Unlike 2D video, Point Cloud Video (PCV) allows for "Six Degrees of Freedom" (6-DoF), meaning users can move through and look around a 3D scene. However, a single high-quality PCV object can require a staggering 3.6 Gbps of raw data.

To make this streamable, researchers use FoV-adaptive streaming: only the parts of the 3D scene the user is currently looking at are downloaded. This creates a "bandwidth-latency paradox." To handle unstable internet connections, you need a long buffer (downloading frames far in advance). But to accurately predict what a user will look at, you need a short buffer, because human movement is unpredictable over long intervals. If the prediction is wrong, the user sees a "black screen" or low-quality artifacts where the data wasn't downloaded.

Methods

The authors propose a sliding-window based progressive streaming framework that leverages the inherent scalability of octree-based point cloud coding (like the MPEG G-PCC standard).

  1. Multi-Round Patching: Instead of downloading one frame at a time, the system looks at a window of future frames (e.g., 20 seconds). It downloads a low-resolution "base layer" for frames far in the future. As those frames get closer to the playback time, the system "patches" them with higher-resolution enhancement layers based on updated, more accurate FoV predictions.
  2. Distance-Aware Utility Model: Traditional video models don't account for 3D depth. The authors developed a new utility function, $Q(r, d)$, that calculates quality based on both the data rate ($r$) and the distance ($d$) between the viewer and the object. This reflects human visual acuity—you don't need high resolution for an object that is far away.
  3. KKT-based Rate Allocation: Deciding how much bandwidth to give to each "patch" in the window is a complex optimization problem. The authors formulated this as a utility maximization problem and solved it analytically using Karush–Kuhn–Tucker (KKT) conditions. Their specific implementation, KKT-exp, uses exponentially decreasing weights for future frames to prioritize data for frames that are about to be played.

Datasets

The researchers evaluated their system using a combination of real-world data:

  • Point Cloud Videos: The "8i voxelized full bodies" dataset, featuring four high-quality 3D captures (Soldier, Loot, Red and Black, and Long Dress).
  • User Traces: 6-DoF FoV traces from 26 real users watching these videos in a Unity game engine.
  • Network Traces: The NYU Metropolitan Mobile Bandwidth Trace, which contains real-world 4G bandwidth data collected from NYC buses, subways, and ferries.

Key Insights and Ideas

  • Hedged Bets: The "aha moment" is that you don't have to commit to a quality level 20 seconds in advance. By treating frames as "patchable" entities, the system can "hedge its bets"—sending just enough data to ensure something is visible if the prediction is rough, and refining it as certainty increases.
  • The Accuracy-Buffer Tradeoff: The paper quantifies how FoV prediction accuracy decays over time. By using a sliding window, they successfully balance the need for a long buffer (to absorb network jitter) with the need for short-term accuracy.
  • Analytical vs. Heuristic: Most existing solutions use "greedy" heuristics (taking the best immediate option). By using an analytical KKT approach, the authors ensure a more globally optimal distribution of bits across the entire buffer window.

Key Results

  • Quality Boost: The proposed KKT-exp algorithm outperformed the state-of-the-art "RUMA" algorithm by 22% in mean per-degree quality.
  • Bandwidth Efficiency: The system significantly reduced "wasted" bandwidth (data downloaded for areas the user never actually looked at) compared to non-progressive baselines.
  • Visual Fidelity: Qualitative results showed far fewer artifacts (like missing limbs or "weird" facial textures) during rapid user movements or network fluctuations.
  • Performance: The algorithm is efficient enough to run at 30 frames per second on a standard server CPU, making it viable for real-world deployment.

Limitations and Future Work

  • Temporal Flickering: The current model optimizes quality frame-by-frame. The authors note that future work should include "temporal smoothing" to prevent distracting quality jumps between consecutive frames.
  • Interactive Latency: While great for Video-on-Demand (VoD), the ultimate challenge is two-way interactive PCV (like a 3D video call), which requires even tighter latency constraints than the 20-second buffer used here.
  • Non-Concave Objectives: The authors plan to explore more complex, non-concave objective functions using techniques like iterative Linear Quadratic Regulators (iLQR) to handle the coupling of rate allocations between adjacent frames.

Summary updated

Abstract

Many XR applications require the delivery of volumetric video to users with six degrees of freedom (6-DoF) movements. Point Cloud has become a popular volumetric video format. A dense point cloud consumes much higher bandwidth than a 2D/360 degree video frame. User Field of View (FoV) is more dynamic with 6-DoF movement than 3-DoF movement. To save bandwidth, FoV-adaptive streaming predicts a user's FoV and only downloads point cloud data falling in the predicted FoV. However, it is vulnerable to FoV prediction errors, which can be significant when a long buffer is utilized for smoothed streaming. In this work, we propose a multi-round progressive refinement framework for point cloud video streaming. Instead of sequentially downloading point cloud frames, our solution simultaneously downloads/patches multiple frames falling into a sliding time-window, leveraging the inherent scalability of octree-based point-cloud coding. The optimal rate allocation among all tiles of active frames are solved analytically using the heterogeneous tile rate-quality functions calibrated by the predicted user FoV. Multi-frame downloading/patching simultaneously takes advantage of the streaming smoothness resulting from long buffer and the FoV prediction accuracy at short buffer length. We evaluate our streaming solution using simulations driven by real point cloud videos, real bandwidth traces, and 6-DoF FoV traces of real users. Our solution is robust against the bandwidth/FoV prediction errors, and can deliver high and smooth view quality in the face of bandwidth variations and dynamic user and point cloud movements.
BibTeX
@article{zong2023progressive,
  author = {Tongyu Zong and Yixiang Mao and Chen Li and Yong Liu and Yao Wang},
  title = {{Progressive Frame Patching for FoV-based Point Cloud Video Streaming}},
  journal = {IEEE Transactions on Multimedia},
  year = {2023},
}
Figure