RXFOOD: Plug-in RGB-X Fusion for Object of Interest Detection

Jin Ma, Jinlong Li, Qing Guo, Tianyun Zhang, Yuewei Lin, Hongkai YuarXiv 2023 · DOI · arXiv

Introduction

Standard RGB imaging often fails in challenging environments such as low-light conditions, complex depth scenarios, or when faced with sophisticated image manipulations. To address these limitations, RGB-X fusion tasks incorporate secondary modalities (X), such as Near-Infrared (NIR), Depth maps, or Frequency information, to provide complementary data. However, effectively fusing these disparate data streams remains a significant challenge. Existing methods often rely on simple averaging or are limited to single-scale fusion, failing to capture the complex inter-relationships between high-level semantics and low-level structural details across different modalities and scales.

Methodology: The RXFOOD Framework

The authors propose RXFOOD (Plug-in RGB-X Fusion for Object of Interest Detection), a flexible module designed to be integrated into existing dual-branch encoder-decoder architectures. The core of RXFOOD is the Energy Exchange Module (EEM), which shifts the focus from raw feature values to the "energy" of feature relationships.

RGB-X applications including Near-infrared, Depth, and Frequency modalities.

Energy Exchange Module (EEM)

The EEM operates by calculating energy matrices that represent spatial and channel-wise relationships within feature maps.

  1. Spatial and Channel Energy: These matrices capture how different positions and channels relate to one another, providing a more robust representation of scene structure than raw pixels.
  2. Cross-Modality and Cross-Scale Exchange: Unlike traditional methods that fuse features at the same resolution, RXFOOD enables simultaneous interaction across different modalities and scales. Energy matrices from various network layers are rescaled to a common size, refined through 1x1 convolutions to exchange "knowledge," and then projected back to their original scales.

Experimental Results

RXFOOD was evaluated across three primary benchmarks: RGB-NIR salient object detection (RGBN-SOD), RGB-D salient object detection (ReDWeb-S), and image manipulation detection (CASIA).

Qualitative comparison of RXFOOD against Average Fusion and Mutual Attention methods.

Performance Gains

  • RGB-NIR: Integration with the TBNet backbone resulted in a 5.9% improvement in F-measure and a 5% increase in maxF score, demonstrating high robustness in dark environments.
  • RGB-D: On an FPN backbone, RXFOOD improved the E-measure by 3.99% and the F-measure by 4.84%.
  • Image Manipulation: The module achieved state-of-the-art results in detecting tampered regions, particularly in challenging "copy-move" forgeries where forged areas are visually consistent with the background.

Efficiency

Despite the sophistication of the energy exchange mechanism, RXFOOD is highly efficient. On an NVIDIA 3090 GPU, the module adds only 2 to 4 milliseconds of inference time per image, making it suitable for real-time deployment.

Conclusion

RXFOOD provides a lightweight, "plug-and-play" solution for multi-modal object detection. By prioritizing the exchange of relational "energy" over raw data and enabling cross-scale interactions, it significantly enhances the performance of standard backbones with minimal computational cost. Future work may explore extending this energy-based attention to video-based detection and triple-branch fusion systems.

Summary updated

Abstract

The emergence of different sensors (Near-Infrared, Depth, etc.) is a remedy for the limited application scenarios of traditional RGB camera. The RGB-X tasks, which rely on RGB input and another type of data input to resolve specific problems, have become a popular research topic in multimedia. A crucial part in two-branch RGB-X deep neural networks is how to fuse information across modalities. Given the tremendous information inside RGB-X networks, previous works typically apply naive fusion (e.g., average or max fusion) or only focus on the feature fusion at the same scale(s). While in this paper, we propose a novel method called RXFOOD for the fusion of features across different scales within the same modality branch and from different modality branches simultaneously in a unified attention mechanism. An Energy Exchange Module is designed for the interaction of each feature map's energy matrix, who reflects the inter-relationship of different positions and different channels inside a feature map. The RXFOOD method can be easily incorporated to any dual-branch encoder-decoder network as a plug-in module, and help the original backbone network better focus on important positions and channels for object of interest detection. Experimental results on RGB-NIR salient object detection, RGB-D salient object detection, and RGBFrequency image manipulation detection demonstrate the clear effectiveness of the proposed RXFOOD.
BibTeX
@misc{ma2023rxfood,
  author = {Jin Ma and Jinlong Li and Qing Guo and Tianyun Zhang and Yuewei Lin and Hongkai Yu},
  title = {{RXFOOD: Plug-in RGB-X Fusion for Object of Interest Detection}},
  year = {2023},
  eprint = {2306.12621},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2306.12621},
}
Figure