Learning a Compact Spatial-Angular Representation for Light Field

Yangfan Sun, Li Li 0040, Zhu Li 0001, Shizheng Wang, Shan Liu, Ge Li

Abstract

The recent emergence of light field technology has led to new opportunities for immersive visual communication that has a need for high spatial and angular resolution, both of which contribute to a large image storage footprint and high-latency transmission. Task-driven downsampling methods have been proposed as a solution, and have shown improvements in single-image restoration. However, they are inevitable to disregard light field's intrinsic properties in the corresponding tasks. In this paper, we propose a light-field-specific task-driven downsampling framework, called the LFCrNet. The LFCrNet operates on a learning-based decreasing and increasing resolution in an end-to-end manner in order to utilize a cross-view asymmetric sampling technique. In detail, it separates raw data into disparity and non-disparity patterns by measuring pixel-wise residuals between the sub-aperture central view and auxiliary views. Then, a chain of 3-D deformable residual blocks (DRBs) is used to extract disparity features and manage these features regard of their intrinsic property individually. Afterwards, they are compacted into spatio-angular domains through a 3-D deformable downsampler (3-DDS). The non-disparity information is integrated into a separate pipeline that leverages spatial similarity across multiple light field views. This technique is capable of preserving specific occlusion components, and subsequently, restoring them using a learning-based upscaling method to generate a high-quality reconstruction. In general, our method has shown superior performance on multiple open-source datasets by a significant margin.
BibTeX
@article{sun2023learning,
  author = {Yangfan Sun and Li Li and Zhu Li and Shizheng Wang and Shan Liu and Ge Li},
  title = {{Learning a Compact Spatial-Angular Representation for Light Field}},
  journal = {IEEE Trans. Multimedia},
  year = {2023},
  doi = {10.1109/tmm.2022.3219671},
}

Loading related content...

Figure