Introduction
Standard RGB imaging often fails in challenging environments such as low-light conditions, complex depth scenarios, or when faced with sophisticated image manipulations. To address these limitations, RGB-X fusion tasks incorporate secondary modalities (X), such as Near-Infrared (NIR), Depth maps, or Frequency information, to provide complementary data. However, effectively fusing these disparate data streams remains a significant challenge. Existing methods often rely on simple averaging or are limited to single-scale fusion, failing to capture the complex inter-relationships between high-level semantics and low-level structural details across different modalities and scales.
Methodology: The RXFOOD Framework
The authors propose RXFOOD (Plug-in RGB-X Fusion for Object of Interest Detection), a flexible module designed to be integrated into existing dual-branch encoder-decoder architectures. The core of RXFOOD is the Energy Exchange Module (EEM), which shifts the focus from raw feature values to the "energy" of feature relationships.
Energy Exchange Module (EEM)
The EEM operates by calculating energy matrices that represent spatial and channel-wise relationships within feature maps.
- Spatial and Channel Energy: These matrices capture how different positions and channels relate to one another, providing a more robust representation of scene structure than raw pixels.
- Cross-Modality and Cross-Scale Exchange: Unlike traditional methods that fuse features at the same resolution, RXFOOD enables simultaneous interaction across different modalities and scales. Energy matrices from various network layers are rescaled to a common size, refined through 1x1 convolutions to exchange "knowledge," and then projected back to their original scales.
Experimental Results
RXFOOD was evaluated across three primary benchmarks: RGB-NIR salient object detection (RGBN-SOD), RGB-D salient object detection (ReDWeb-S), and image manipulation detection (CASIA).
Performance Gains
- RGB-NIR: Integration with the TBNet backbone resulted in a 5.9% improvement in F-measure and a 5% increase in maxF score, demonstrating high robustness in dark environments.
- RGB-D: On an FPN backbone, RXFOOD improved the E-measure by 3.99% and the F-measure by 4.84%.
- Image Manipulation: The module achieved state-of-the-art results in detecting tampered regions, particularly in challenging "copy-move" forgeries where forged areas are visually consistent with the background.
Efficiency
Despite the sophistication of the energy exchange mechanism, RXFOOD is highly efficient. On an NVIDIA 3090 GPU, the module adds only 2 to 4 milliseconds of inference time per image, making it suitable for real-time deployment.
Conclusion
RXFOOD provides a lightweight, "plug-and-play" solution for multi-modal object detection. By prioritizing the exchange of relational "energy" over raw data and enabling cross-scale interactions, it significantly enhances the performance of standard backbones with minimal computational cost. Future work may explore extending this energy-based attention to video-based detection and triple-branch fusion systems.
Summary updated