Abstract
Salient Object Detection (SOD) for light field (LF) cameras has traditionally relied on complex two-stream architectures that process All-Focused (AF) images and Focal Stacks (FS) separately. This paper introduces LF Tracy, a streamlined single-pipeline framework that addresses the limitations of intra- and inter-network connectivity in existing models. By employing a lightweight Information Aggregation (IA) module and a novel MixLD data augmentation strategy, LF Tracy achieves a significant performance improvement, including a 23% reduction in Mean Absolute Error (MAE) on the large-scale PKU-LF benchmark.
Introduction
Salient Object Detection (SOD) aims to identify the most visually prominent objects in a scene, mimicking human visual prioritization. Light Field (LF) cameras provide a rich data source for this task, capturing spatial, depth, and angular information through multiple perspectives. However, current LF SOD methods predominantly utilize "two-stream" networks, which process the All-Focused (AF) image and the Focal Stack (FS) in isolated paths. This paradigm suffers from two primary drawbacks: a lack of intra-network connectivity, preventing the learning of angular relationships between viewpoints, and a lack of inter-network connectivity, where data augmentation fails to bridge the gap between texture and depth information.
Methods
LF Tracy proposes a paradigm shift from dual-stream complexity to a unified single-pipeline architecture.
Single-Pipeline Encoder and IA Module
Instead of separate backbones, LF Tracy utilizes a single encoder to process both AF and FS features. To integrate these diverse data sources, we designed a lightweight Information Aggregation (IA) Module (approximately 8M parameters). This module employs a cross-attention mechanism where the Focal Stack guides "Query" and "Key" generation, while the All-Focused image guides the "Value." This process realigns mismatched features and fuses them into a comprehensive representation capturing spatial, depth, and angular cues.
MixLD Data Augmentation
To enhance inter-network connectivity, we introduced MixLD, a two-step blending strategy. First, FS2AF randomly blends a slice of the Focal Stack into the AF image. Second, AF2FS blends the augmented AF image back into all slices of the Focal Stack. This "pre-mixing" forces the network to learn implicit relationships between texture and depth from the outset, breaking the silos of conventional data representations.
Results and Discussion
LF Tracy was evaluated on standard benchmarks (LFSD, DUT-LF, HFUT) and the large-scale PKU-LF dataset. On the PKU-LF benchmark, which contains over 10,000 challenging images from terrestrial and aquatic environments, LF Tracy achieved a 23% improvement in MAE over previous state-of-the-art methods.
Key insights from our evaluation include:
- Expanded Search Space: TSNE visualizations indicate that the single-pipeline approach explores a significantly larger and more diverse feature search space than dual-stream counterparts.
- Robustness: The model excels in identifying objects with complex boundaries, thin structures (e.g., animal necks), and transparent surfaces (e.g., glass backboards) that typically confound standard SOD models.
- Efficiency: The IA module is highly efficient, adding only 6M-8M parameters to the backbone while significantly boosting performance.
Conclusion
LF Tracy demonstrates that a unified single-pipeline approach is superior for LF SOD. By bridging the gap between different light field data representations through the IA module and MixLD augmentation, we achieve state-of-the-art performance with high efficiency. Future work will explore extending this unified framework to other dense prediction tasks.
Summary updated