Learned Focused Plenoptic Image Compression with Microimage Preprocessing and Global Attention

Kedeng Tong, Xin Jin, Yuqing Yang, Chen Wang, Jinshi Kang, Fan Jiang

Abstract

Focused plenoptic cameras (Plenoptic 2.0) capture high-resolution 4D light fields by recording both spatial and angular information. However, the resulting images contain complex micro-textures and long-distance correlations that challenge standard compression codecs. This paper proposes an end-to-end learning architecture specifically designed for focused plenoptic images. The framework incorporates a microimage preprocessing scheme to align data and remove ineffective pixels, alongside a Global Attention Module (GAM) to capture wide-range spatial redundancies. Evaluated on a new 1,910-image dataset (FPI2k), the method achieves over 50% bitrate reduction compared to VVC intra-coding and significantly outperforms state-of-the-art learned compression models.

The Challenge of Focused Plenoptic Data

Plenoptic cameras use a microlens array (MLA) to capture the light field, enabling applications like post-capture refocusing and depth estimation. Unlike traditional plenoptic cameras (Plenoptic 1.0), focused plenoptic cameras (Plenoptic 2.0) use the MLA as a relay system, positioning the sensor away from the microlens focal plane.

Comparison of (a) traditional plenoptic camera and (b) focused plenoptic camera imaging principles.

This configuration yields higher spatial resolution but introduces a complex hexagonal grid of "microimages." Each microimage captures a small portion of the scene from a specific viewpoint, leading to significant redundancy across distant parts of the sensor. Standard codecs like HEVC or VVC struggle with these non-local correlations and the high-frequency textures of the microimage boundaries, where nearly 50% of the raw pixels are often "ineffective" (dark gaps or vignetted edges).

Visual representation of raw focused plenoptic images and their complex microimage structures.

Proposed Methodology

The proposed compression framework addresses the unique geometry of focused plenoptic data through two primary innovations:

1. Microimage Preprocessing

Raw plenoptic data is inherently inefficient for standard neural networks. The authors developed a preprocessing pipeline that identifies the center of each microlens, extracts the effective square area, and aligns these patches into a standard rectangular grid. This step losslessly discards irrelevant data and makes the input compatible with standard deep learning architectures.

2. Global Attention Module (GAM)

Because the same object point is often projected into multiple microimages across the sensor, the network requires a massive receptive field to exploit these redundancies. The Global Attention Module is integrated into a hierarchical hyperprior-context network. It utilizes pixel-wise vector attention to compute correlations across the entire feature map during resampling, allowing the model to effectively "bridge" the distance between related microimages and capture global dependencies.

Experimental Results

The researchers introduced FPI2k, a large-scale dataset of 1,910 focused plenoptic images captured with a custom-built camera. Benchmarking against traditional standards and state-of-the-art (SOTA) learned models revealed significant gains:

  • Bitrate Efficiency: The model achieved an average 62.57% bitrate reduction over HEVC intra-coding and 51.67% over VVC intra-coding. It also outperformed SOTA learned models by 18.73%.
  • Perceptual Quality: Qualitative analysis showed that the proposed method preserves sharp textures and fine details (e.g., text on signs or tire treads) far better than standard codecs, which often introduce blurring or blocking artifacts in microimage regions.

Conclusion

By combining optical-aware preprocessing with a global attention mechanism, this research provides a highly efficient solution for the storage and transmission of high-resolution light field data. While decoding complexity remains a challenge for real-time use, the significant gains in compression efficiency and perceptual quality mark a major step forward for focused plenoptic imaging systems. The dataset and code are publicly available at https://github.com/VincentChandelier/GACN.

Summary updated

Abstract

Focused plenoptic cameras can record spatial and angular information of the light field (LF) simultaneously with higher spatial resolution relative to traditional plenoptic cameras, which facilitate various applications in computer vision. However, the existing plenoptic image compression methods present ineffectiveness to the captured images due to the complex micro-textures generated by the microlens relay imaging and long-distance correlations among the microimages. In this paper, a lossy end-to-end learning architecture is proposed to compress the focused plenoptic images efficiently. First, a data preprocessing scheme is designed according to the imaging principle to remove the sub-aperture image ineffective pixels in the recorded light field and align the microimages to the rectangular grid. Then, the global attention module with large receptive field is proposed to capture the global correlation among the feature maps using pixel-wise vector attention computed in the resampling process. Also, a new image dataset consisting of 1910 focused plenoptic images with content and depth diversity is built to benefit training and testing. Extensive experimental evaluations demonstrate the effectiveness of the proposed approach. It outperforms intra coding of HEVC and VVC by an average of 62.57% and 51.67% bitrate reduction on the 20 preprocessed focused plenoptic images, respectively. Also, it achieves 18.73% bitrate saving and generates perceptually pleasant reconstructions compared to the state-of-the-art end-to-end image compression methods, which benefits the applications of focused plenoptic cameras greatly. The dataset and code are publicly available at https://github.com/VincentChandelier/GACN.
BibTeX
@article{tong2023learned,
  author = {Kedeng Tong and Xin Jin and Yuqing Yang and Chen Wang and Jinshi Kang and Fan Jiang},
  title = {{Learned Focused Plenoptic Image Compression with Microimage Preprocessing and Global Attention}},
  journal = {IEEE Transactions on Multimedia},
  year = {2023},
  doi = {10.1109/tmm.2023.3272747},
}
Figure