View-dependent real-time 3d video compression for mobile devices

Shu Shi, Klara Nahrstedt, Roy H. CampbellACM Multimedia 2008 · DOI

TL;DR

This paper introduces a novel view-dependent compression technique for real-time 3D video streams, specifically designed to overcome the bandwidth and computational limitations of mobile devices. By intelligently transmitting only the visible parts of a 3D scene from the user's perspective, they achieve significant compression ratios, enabling 3D video playback on devices like the Nokia N800 PDA.

What Problem Does This Paper Address?

The promise of immersive 3D video experiences is currently hampered by the immense bandwidth and computational power required by existing systems. These systems often rely on high-speed networks and powerful workstations, making them inaccessible for mobile devices. The core challenge is to adapt 3D video technology to the constraints of mobile platforms, which have limited wireless bandwidth (Mbps vs. Gbps) and less processing power. This paper aims to bridge this gap by developing a compression methodology that prioritizes the user's viewpoint, transmitting only the necessary data for a seamless experience on a mobile device.

Methods

The proposed approach is a view-dependent compression scheme operating within a closed-loop feedback system between a server and a mobile client.

  • Data Model: The system uses depth images as its primary data model. A depth image consists of an RGB color image map and a corresponding depth map, where each pixel has a depth value.
  • Server-Side Compression:
    • 3D Point Reconstruction: Multiple synchronized and calibrated depth image streams are used to reconstruct 3D points.
  • Viewpoint Projection: Based on the client's current viewpoint (received from the client), a projection plane is generated. All 3D points are projected onto this plane.
  • Visibility Filtering: Points invisible to the client's viewpoint are filtered out. Visible points form the basic layer of the compressed stream, containing a pre-rendered image map and depth information.
  • Downsampling (Enhanced Layer): Invisible points are grouped into spatial cubes and downsampled (e.g., by averaging coordinates and colors). This downsampled data forms the enhanced layer, which can be used to approximate the scene if the viewpoint changes rapidly.
  • Viewpoint Layer: Information about the client's viewpoint is also included.
  • Client-Side Decompression:
    • Parsing: The compressed stream is parsed into viewpoint, basic, and enhanced layers.
  • Rendering: If the viewpoint hasn't changed, only the basic layer's image map is displayed. If the viewpoint has changed, both basic and enhanced layers are used to reconstruct and render the 3D points from the new perspective.
  • Feedback: The client continuously sends its updated viewpoint information back to the server.

The key novelty lies in the view-dependent filtering and downsampling of invisible points, significantly reducing the amount of data that needs to be transmitted.

Datasets

The experiments were conducted using 3D video data previously recorded for the TEEVE project, consisting of 12 depth image streams. No new datasets were introduced.

Key Insights and Ideas

  • Viewpoint is King: The core insight is that for a mobile user, only the visible portion of the 3D scene matters. Transmitting data for occluded or unseen objects is a waste of precious bandwidth.
  • Server-Side Intelligence: Offloading the heavy lifting of 3D reconstruction and rendering to a powerful server is crucial for mobile clients. The server intelligently processes the data based on the client's dynamic viewpoint.
  • Layered Compression for Adaptability: The layered structure (viewpoint, basic, enhanced) allows for graceful degradation and adaptation. The basic layer provides immediate display, while the enhanced layer offers a fallback for viewpoint changes.
  • Trade-off Between Compression and Quality: The size of the spatial cubes used for downsampling directly impacts the compression ratio and the quality of the scene when the viewpoint changes. Larger cubes lead to higher compression but potentially more visual artifacts.

Key Results

  • Significant Bandwidth Reduction: The proposed scheme achieved a dramatic reduction in bandwidth. For a 320x240 display window at 10 fps, the bit rate dropped from 56.6 Mbps (original compressed depth streams) to just 480 Kbps.
  • Compression Ratios:
    • Basic Layer: Compression ratios of up to 25.3:1 were achieved for the basic layer (visible points) when reducing the display window size.
  • Enhanced Layer: Compression ratios of up to 82.9:1 were achieved for the enhanced layer (invisible points) by using larger downsampling cubes (64x64x64).
  • Real-time Performance: The system demonstrated real-time compression on the server (6.8 fps) and real-time display on the Nokia N800 PDA (14.5 fps when the viewpoint remained unchanged).
  • Viewpoint Change Impact: When the viewpoint changed, the frame rate on the PDA dropped to less than 4 fps, highlighting the computational cost of reconstructing and rendering from new perspectives. The authors note that it takes approximately 120 ms for the server to update with new viewpoint information.

Limitations and Future Work

  • Viewpoint Change Latency: The significant drop in frame rate when the viewpoint changes is a major limitation. This is attributed to the need for 3D point reconstruction and rendering.
  • Quality Degradation with Viewpoint Change: The quality of the rendered scene can degrade when the viewpoint changes, especially with aggressive downsampling in the enhanced layer.
  • Hardware Acceleration: The authors suggest that hardware acceleration for graphics on mobile devices could significantly improve performance during viewpoint changes.
  • Adaptive Frame Dropping: Implementing adaptive frame dropping is proposed as a method to maintain Quality of Service (QoS) when decoding rates fluctuate.
  • Further Improvements:
    • Optimizing the viewpoint layer transmission (only sending it when the viewpoint changes).
  • Exploring better downsampling methods for the enhanced layer.
  • Leveraging graphics card pipelines (e.g., OpenGL) on the server for faster projection computations.

Summary updated

BibTeX
@inproceedings{shi2008view,
  author = {Shu Shi and Klara Nahrstedt and Roy H. Campbell},
  title = {{View-dependent real-time 3d video compression for mobile devices}},
  booktitle = {ACM Multimedia},
  year = {2008},
  doi = {10.1145/1459359.1459485},
}
Figure