Efficient Content Adaptive Plenoptic Video Coding

Wang Tu, Xin Jin, Lingjun Li, Chenggang Yan, Yaoqi Sun, Mang Xiao, Weidong Han, Jiyong Zhang

Introduction: The 4D Data Explosion

Traditional photography captures a 2D projection of a scene, losing the directional information of light rays. Plenoptic (light field) cameras, such as the Lytro Illum, capture the full 4D light field, enabling post-capture capabilities like refocusing and perspective shifting. However, this richness comes at a massive cost: a single plenoptic video contains hundreds of subaperture images (viewpoints) for every frame, creating a data volume that overwhelms standard storage and transmission systems.

The Core Challenge: Redundancy in Light Fields

Standard video compression standards like H.264 or HEVC are optimized for 2D temporal redundancy. They struggle with the complex, multi-dimensional redundancies inherent in plenoptic video, where information is shared both across different viewpoints (spatial) and across time (temporal). Existing methods often apply a one-size-fits-all approach, failing to account for the specific motion characteristics of different scenes.

The Proposed Content-Adaptive Framework

This research introduces a "divide and conquer" approach that intelligently adapts its compression strategy based on the video content. The system first splits the video into Central Views (high-quality reference viewpoints) and Adjacent Views. Instead of encoding adjacent views directly, the system calculates the "residual"—the difference between the adjacent views and the reconstructed central views—which significantly reduces the amount of data to be processed.

The core innovation lies in the adaptive selection between two primary coding strategies:

  1. Spatial Multiview Coding (S-MVC): This method organizes views into streams based on their spatial arrangement within a single time step. It is most effective for scenes with fast or complex motion where temporal correlation is low.
Spatial Multiview Coding (S-MVC) organizes views into streams at a single time step (T0) to exploit spatial correlation.
  1. Temporal Multiview Coding (T-MVC): This method organizes each individual viewpoint into its own temporal stream. It is ideal for scenes with slow, predictable motion where a specific viewpoint remains highly correlated over time.
Temporal Multiview Coding (T-MVC) tracks a single viewpoint (e.g., #1) across multiple time steps (T0 to Ti) to exploit temporal correlation.

Decision Logic and Implementation

To choose the optimal strategy, the encoder performs a correlation analysis. If the results are ambiguous, it executes a "residual energy" analysis to determine which method leaves the least data to compress. To maintain compatibility with existing standards, the authors cleverly repurposed a reserved bit (Type 13) in the NAL header of the H.265/HEVC standard to signal the chosen method to the decoder, ensuring zero-overhead signaling.

Experimental Results

The framework was tested on six diverse video sequences (e.g., Auto Race, Cube, Flowers) captured with a Lytro Illum camera, representing various motion types like rotation, diagonal movement, and complex vertical/horizontal shifts.

  • Bitrate Savings: The adaptive method achieved an average 25.06% bitrate saving compared to standard anchor methods.
  • Consistency: Unlike simpler fixed methods that can occasionally increase bitrate by up to 80% in complex scenes, the adaptive approach consistently identified the most efficient path for every motion type tested.
  • Efficiency: The decision-making logic is computationally lightweight, allowing for implementation without significant processing overhead.

Conclusion

By introducing content-adaptive intelligence into the plenoptic video pipeline, this research demonstrates that significant compression gains are possible without sacrificing compatibility. Future work will explore dynamic optimization of the number of central views and scaling the logic to handle longer cinematic sequences with frequent scene cuts.

Summary updated

Abstract

In this paper, a content adaptive coding method for plenoptic video is proposed to reduce both of the spatial and temporal redundancy. Based on the spatial correlations among subapertures, the plenoptic video is divided into two subaperture groups: the central view videos and the residual videos. Coding methods, multiview coding method based on spatial correlation(S-MVC) and multiview coding method based on temporal correlation(T-MVC), are adopted for residual videos according to correlation and residual energy analysis, while the central view videos are always encoded by S-MVC. Correlation analysis is performed first, and residual energy analysis is further performed if necessary. Coding the central view videos and the residual videos separately reduces bitstream, choosing different encoding methods for residual videos makes full use of the characteristics of different video content. Both the central view videos and the residual videos are compressed by MV-HEVC. The experimental results demonstrate that the proposed method outperforms a pseudo-sequence-based MVC method for plenoptic video by an average of 25.06% bitrate saving.
BibTeX
@article{tu2020efficient,
  author = {Wang Tu and Xin Jin and Lingjun Li and Chenggang Yan and Yaoqi Sun and Mang Xiao and Weidong Han and Jiyong Zhang},
  title = {{Efficient Content Adaptive Plenoptic Video Coding}},
  journal = {IEEE Access},
  year = {2020},
  doi = {10.1109/access.2020.2964056},
}
Figure