Learning Adaptive Fusion Bank for Multi-modal Salient Object Detection

Kunpeng Wang, Zhengzheng Tu, Chenglong Li, Cheng Zhang, Bin Luo

Introduction

Salient Object Detection (SOD) is a fundamental computer vision task aimed at identifying and segmenting the most visually prominent objects in a scene. While standard RGB-based methods have achieved significant success in controlled environments, they often falter in real-world scenarios characterized by low illumination, camouflaged backgrounds, or complex image clutter. To address these limitations, Multi-modal Salient Object Detection (MSOD) incorporates auxiliary data—such as depth maps (RGB-D) or thermal infrared images (RGB-T)—to provide additional geometric or thermal cues.

However, traditional MSOD models typically rely on a single, fixed fusion strategy to combine these modalities. Such "one-size-fits-all" approaches often fail to generalize across diverse challenges; a strategy optimized for low-light conditions may prove ineffective when dealing with scale variations or noisy depth data. Furthermore, many existing models struggle with "hollow" objects (e.g., rings or frames), frequently misidentifying the empty center as part of the salient object. This paper introduces the Learning Adaptive Fusion Bank (LAFB), a novel framework designed to dynamically adapt its fusion strategy to the specific challenges of each input frame.

Methodology

The LAFB framework is built upon three core components: the Adaptive Fusion Bank, the Adaptive Ensemble Module, and the Indirect Interactive Guidance Module.

1. Adaptive Fusion Bank (AFB)

Rather than employing a single complex fusion module, the researchers developed a bank of five specialized, lightweight fusion experts. Each expert is tailored to address a specific environmental or structural challenge:

  • Center Bias: Focuses on locating objects positioned near the center of the frame.
  • Scale Variation: Utilizes dilated convolutions to accurately capture objects of varying sizes, from very small to very large.
  • Image Clutter: Employs residual connections to filter out noisy or distracting background elements.
  • Low Illumination: Uses depth or thermal cues to guide the RGB branch when visual light is insufficient.
  • Thermal Crossover/Depth Ambiguity: Uses RGB features to "clean up" and refine unreliable depth or thermal data.

2. Adaptive Ensemble Module (AEM)

The AEM serves as the system's decision-making core. It analyzes the input features and dynamically assigns weights to the five experts in the bank. By creating a custom blend of these specialized modules, the AEM allows the network to handle hybrid challenges—such as a scene that is simultaneously dark and cluttered—by leaning on multiple experts at once.

3. Indirect Interactive Guidance Module (IIGM)

To solve the persistent problem of "hollow" objects, the authors introduced the IIGM. Traditional models often lose fine spatial details during aggressive feature merging, leading to the accidental filling of holes. The IIGM avoids direct merging, instead using middle-level features as a "springboard." This allows high-level semantic information (identifying what the object is) and low-level spatial details (identifying where the edges are) to interact smoothly, preserving the spatial integrity of empty centers while maintaining a robust object identity.

Experimental Evaluation

The LAFB model was rigorously evaluated across several benchmark datasets for both RGB-T and RGB-D modalities:

  • RGB-T Datasets: VT821, VT1000, and VT5000.
  • RGB-D Datasets: DUT-RGBD, NJUD, NLPR, SIP, SSD, STERE, and ReDWeb-S.

The VT5000 dataset, which includes specific annotations for various environmental challenges, was instrumental in demonstrating the effectiveness of the adaptive bank in selecting the appropriate experts for specific scene types.

Results and Discussion

State-of-the-Art Performance

LAFB consistently outperformed existing state-of-the-art (SOTA) methods across multiple evaluation metrics, including E-measure, F-measure, and Mean Absolute Error (MAE). On the challenging VT5000 RGB-T dataset, the method achieved a significant 2.5% average improvement over previous leading models.

Efficiency and Real-time Processing

Despite the inclusion of multiple fusion modules, the individual experts are designed for simplicity and efficiency. The model maintains a high inference speed of 45-50 FPS on standard GPU hardware, making it well-suited for real-time applications.

Qualitative Advantages

Visual analysis confirms that LAFB is uniquely capable of ignoring "thermal crossovers" (where background heat signatures mimic human shapes) and accurately segmenting the boundaries of hollow objects. By decoupling the fusion process into specialized sub-tasks, the network achieves a level of robustness that monolithic fusion strategies cannot match.

Conclusion and Future Work

The Learning Adaptive Fusion Bank represents a significant step forward in multi-modal perception. By replacing rigid fusion architectures with a dynamic, ensemble-based approach, the model effectively navigates the complexities of real-world environments.

While highly robust, the authors identified remaining challenges, such as the detection of transparent objects (e.g., glass) that lack distinct thermal or depth signatures, and extreme overexposure that washes out both modalities. Future research will focus on expanding the fusion bank with experts dedicated to these edge cases and exploring more generalized fusion schemes to further enhance the model's versatility.

Summary updated

BibTeX
@article{wang2024learning,
  author = {Kunpeng Wang and Zhengzheng Tu and Chenglong Li and Cheng Zhang and Bin Luo},
  title = {{Learning Adaptive Fusion Bank for Multi-modal Salient Object Detection}},
  journal = {IEEE Trans. Circuits Syst. Video Technol.},
  year = {2024},
  doi = {10.1109/tcsvt.2024.3375505},
}
Figure