LF Tracy: A Unified Single-Pipeline Approach for Salient Object Detection in Light Field Cameras

Fei Teng, Jiaming Zhang, Jiawei Liu, Kunyu Peng, Xina Cheng, Zhiyong Li, Kailun YangarXiv 2024 · DOI · arXiv

Abstract

Salient Object Detection (SOD) for light field (LF) cameras has traditionally relied on complex two-stream architectures that process All-Focused (AF) images and Focal Stacks (FS) separately. This paper introduces LF Tracy, a streamlined single-pipeline framework that addresses the limitations of intra- and inter-network connectivity in existing models. By employing a lightweight Information Aggregation (IA) module and a novel MixLD data augmentation strategy, LF Tracy achieves a significant performance improvement, including a 23% reduction in Mean Absolute Error (MAE) on the large-scale PKU-LF benchmark.

Introduction

Salient Object Detection (SOD) aims to identify the most visually prominent objects in a scene, mimicking human visual prioritization. Light Field (LF) cameras provide a rich data source for this task, capturing spatial, depth, and angular information through multiple perspectives. However, current LF SOD methods predominantly utilize "two-stream" networks, which process the All-Focused (AF) image and the Focal Stack (FS) in isolated paths. This paradigm suffers from two primary drawbacks: a lack of intra-network connectivity, preventing the learning of angular relationships between viewpoints, and a lack of inter-network connectivity, where data augmentation fails to bridge the gap between texture and depth information.

Methods

LF Tracy proposes a paradigm shift from dual-stream complexity to a unified single-pipeline architecture.

Single-Pipeline Encoder and IA Module

Instead of separate backbones, LF Tracy utilizes a single encoder to process both AF and FS features. To integrate these diverse data sources, we designed a lightweight Information Aggregation (IA) Module (approximately 8M parameters). This module employs a cross-attention mechanism where the Focal Stack guides "Query" and "Key" generation, while the All-Focused image guides the "Value." This process realigns mismatched features and fuses them into a comprehensive representation capturing spatial, depth, and angular cues.

The proposed single-pipeline paradigm (b) compared to the conventional two-stream approach (a).
Architecture of the Information Aggregation (IA) Module, showing the fusion of AF and FS features across multiple stages.

MixLD Data Augmentation

To enhance inter-network connectivity, we introduced MixLD, a two-step blending strategy. First, FS2AF randomly blends a slice of the Focal Stack into the AF image. Second, AF2FS blends the augmented AF image back into all slices of the Focal Stack. This "pre-mixing" forces the network to learn implicit relationships between texture and depth from the outset, breaking the silos of conventional data representations.

Results and Discussion

LF Tracy was evaluated on standard benchmarks (LFSD, DUT-LF, HFUT) and the large-scale PKU-LF dataset. On the PKU-LF benchmark, which contains over 10,000 challenging images from terrestrial and aquatic environments, LF Tracy achieved a 23% improvement in MAE over previous state-of-the-art methods.

Key insights from our evaluation include:

  • Expanded Search Space: TSNE visualizations indicate that the single-pipeline approach explores a significantly larger and more diverse feature search space than dual-stream counterparts.
  • Robustness: The model excels in identifying objects with complex boundaries, thin structures (e.g., animal necks), and transparent surfaces (e.g., glass backboards) that typically confound standard SOD models.
  • Efficiency: The IA module is highly efficient, adding only 6M-8M parameters to the backbone while significantly boosting performance.

Conclusion

LF Tracy demonstrates that a unified single-pipeline approach is superior for LF SOD. By bridging the gap between different light field data representations through the IA module and MixLD augmentation, we achieve state-of-the-art performance with high efficiency. Future work will explore extending this unified framework to other dense prediction tasks.

Summary updated

Abstract

Leveraging rich information is crucial for dense prediction tasks. Light field (LF) cameras are instrumental in this regard, as they allow data to be sampled from various perspectives. This capability provides valuable spatial, depth, and angular information, enhancing scene-parsing tasks. However, we have identified two overlooked issues for the LF salient object detection (SOD) task. (1): Previous approaches predominantly employ a customized two-stream design to discover the spatial and depth features within light field images. The network struggles to learn the implicit angular information between different images due to a lack of intra-network data connectivity. (2): Little research has been directed towards the data augmentation strategy for LF SOD. Research on inter-network data connectivity is scant. In this study, we propose an efficient paradigm (LF Tracy) to address those issues. This comprises a single-pipeline encoder paired with a highly efficient information aggregation (IA) module (around 8M parameters) to establish an intra-network connection. Then, a simple yet effective data augmentation strategy called MixLD is designed to bridge the inter-network connections. Owing to this innovative paradigm, our model surpasses the existing state-of-the-art method through extensive experiments. Especially, LF Tracy demonstrates a 23% improvement over previous results on the latest large-scale PKU dataset. The source code is publicly available at: https://github.com/FeiBryantkit/LF-Tracy.
BibTeX
@misc{teng2024lf,
  author = {Fei Teng and Jiaming Zhang and Jiawei Liu and Kunyu Peng and Xina Cheng and Zhiyong Li and Kailun Yang},
  title = {{LF Tracy: A Unified Single-Pipeline Approach for Salient Object Detection in Light Field Cameras}},
  year = {2024},
  eprint = {2401.16712},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2401.16712},
}
Figure