Style Transfer for Light Field Photography

David Hart, Bryan Morse, Jessica GreenlandWACV 2020

TL;DR

This paper introduces a novel method for applying neural style transfer to light field images, preserving angular consistency across multiple views. By adapting existing feed-forward style transfer networks and incorporating a novel disparity loss, they achieve stylized light fields that maintain their inherent geometric properties and refocussing capabilities, overcoming limitations of naive per-view stylization.

What Problem Does This Paper Address?

Light field photography captures a scene with multiple viewpoints, enabling unique capabilities like novel view synthesis and focal stack generation. However, editing these multi-view images, particularly with advanced techniques like neural style transfer, presents a significant challenge. Existing neural style transfer methods, while effective for single images, fail to maintain consistency across the many views of a light field. Naively applying style transfer to each view independently results in visual inconsistencies that break the geometric coherence of the light field, rendering its special capabilities useless in the stylized output. Furthermore, training new networks from scratch for light fields is hindered by the scarcity of large light field datasets.

Methods

The core of this paper's contribution lies in adapting existing feed-forward neural style transfer networks to handle the multi-view nature of light fields while enforcing angular consistency.

Here's a breakdown of their approach:

  • Disparity Map Generation: They first generate disparity maps for each view of the light field. Crucially, they achieve this by using a single central-view depth map and a novel method to invert and calibrate it, producing disparity maps for all other views. This is a key step as it leverages existing depth information without requiring per-view depth maps.
  • Feature Warping and Fusion: For each non-central view, they encode its features and then warp the fixed features of the central stylized view using the generated disparity map. This warped central feature map is then fused with the current view's features using a confidence map derived from the disparity estimation. This fusion process is similar to prior work on stereo and video stylization but adapted for the larger number of views in a light field.
  • Disparity Loss for Consistency: A novel disparity loss is introduced, which penalizes inconsistencies between the stylized current view and a warped version of the stylized central view. This loss is modulated by the confidence map to avoid penalizing areas with uncertain correspondences.
  • Iterative Optimization: Instead of training a new network from scratch, they leverage pre-trained feed-forward style transfer networks. The disparity loss is then backpropagated through the network to iteratively optimize the stylization for each view, ensuring consistency across the entire light field. This approach avoids the need for large light field training datasets.

Datasets

The paper utilizes existing light field datasets for evaluation. Specific datasets mentioned include:

  • Stanford Lytro light field archive ([35])
  • EPFL light field image dataset ([36])

The authors note that these datasets are "small and are insufficient for training purely feed-forward style-transfer networks from scratch," which motivates their approach of adapting existing networks.

Key Insights and Ideas

  • Angular Consistency is Paramount: The paper strongly emphasizes that for light field editing to be meaningful, angular consistency across views is not just desirable but essential. Inconsistencies break the fundamental properties of light fields.
  • Leveraging Central View as Anchor: Using the central view's stylized features as a fixed reference point and warping them to other views is a critical architectural choice. This prevents the central view from being overly influenced by other views, which could lead to blurry results.
  • Disparity Maps as Geometric Priors: The method effectively uses disparity maps not just for warping but also to derive confidence maps, which are crucial for robustly enforcing consistency, especially in areas with uncertain correspondences or occlusions.
  • Adaptation Over Retraining: The success of their method hinges on adapting existing, powerful feed-forward style transfer networks rather than attempting to train new ones from scratch, directly addressing the dataset scarcity problem.

Key Results

  • Qualitative: The method produces visually consistent stylized light fields. Figures demonstrate that depth maps reconstructed from their stylized light fields closely resemble those from the original unstylized fields, unlike naive stylization which introduces significant errors. The ability to refocus the stylized light field is also demonstrated.
  • Quantitative:
    • Perceptual Loss: The proposed method maintains a perceptual loss very similar to naive per-view stylization, indicating that the style is well-preserved.
  • Disparity Loss: The disparity loss (a measure of angular inconsistency) is dramatically reduced, dropping by an order of magnitude or more compared to the naive method. This quantitatively validates the improved angular consistency.
  • Execution Time: While not the primary focus, the paper notes that their iterative optimization approach is significantly faster than pure Gatys-style optimization from scratch.

Limitations and Future Work

  • Reliance on Central Depth Map: The method's ability to generate disparity maps for all views relies on having a pre-computed depth map for the central view. While this is often available in light field datasets, it's a dependency.
  • Artifacts from Image-Space Fusion: The paper notes that image-space fusion (as opposed to feature-space fusion) is highly dependent on accurate disparity maps and can lead to visual artifacts when disparities are imperfect, especially in occluded regions. Their feature-space fusion approach mitigates this.
  • Future Work: The authors suggest that their work lays a foundation for more generalizable light-field consistency optimizations. They also explore variations of their method, including trade-offs between speed, consistency, and sensitivity to depth map errors.

Summary updated

BibTeX
@inproceedings{hart2020style,
  author = {David Hart and Bryan Morse and Jessica Greenland},
  title = {{Style Transfer for Light Field Photography}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2020},
}
Figure