Abstract
Multi-modal object detection, combining RGB and Infrared (IR) data, is essential for robust perception in autonomous systems, particularly in low-light environments. However, existing fusion operators often face a trade-off between computational efficiency and the ability to capture global context. We introduce PoolFuser, a lightweight fusion operator that replaces computationally expensive self-attention with a simple pooling mechanism within a MetaFormer-style architecture. Evaluated on the LLVIP dataset, PoolFuser achieves a state-of-the-art mAP of 0.384, outperforming Transformer-based models while reducing inference latency by approximately 31% and memory consumption by 12% compared to standard Transformer-based architectures.
Introduction
In autonomous driving and surveillance, relying on a single sensor modality is often insufficient. RGB cameras provide rich texture and color information but are highly susceptible to poor lighting conditions. Conversely, Infrared (IR) sensors excel in total darkness by detecting heat signatures but lack fine-grained semantic detail. Integrating these modalities through multi-modal fusion is the standard approach for ensuring system robustness.
The challenge lies in the design of the "fusion operator"—the mechanism that combines data streams. Simple operators like element-wise summation are efficient but fail to capture complex inter-modal relationships. Advanced attention modules, such as CBAM, improve feature selection but often focus on local context. While Transformers (e.g., TransFuser) effectively capture global context via multi-head self-attention, their quadratic complexity leads to high latency and memory usage, making them unsuitable for real-time deployment on edge devices. PoolFuser aims to bridge this gap by providing global reach with the efficiency of a simpler model.
Methodology
PoolFuser is inspired by the MetaFormer (or PoolFormer) architecture, which posits that the overall structure of a Transformer is more critical to its success than the specific attention mechanism used.
PoolFuser Architecture
Instead of utilizing mathematically intensive self-attention, PoolFuser employs a parameter-free pooling operator as a spatial mixer. This allows the model to aggregate information across the entire spatial domain without the quadratic cost associated with calculating pixel-to-pixel relationships. The operator retains the standard Transformer layout, including normalization layers and feed-forward networks, but swaps the attention component for pooling.
As illustrated in Figure 1, the system employs two ResNet-34 backbones. Feature maps from both modalities are concatenated and processed through four PoolFuser blocks. These blocks compute attention weights to generate a final fused feature map, which is then passed to a Faster R-CNN detection head for final object localization and classification.
Experimental Setup
The model was rigorously evaluated using the LLVIP (Visible-infrared Paired Dataset for Low-light Vision). This dataset contains 15,488 strictly aligned RGB and IR image pairs focused on pedestrian detection in dimly lit urban environments. The data was split into an 8:2 ratio for training and testing, with performance measured using standard COCO metrics, including mean Average Precision (mAP), AP50, and AP75.
Results and Discussion
The performance of PoolFuser was compared against several baselines, including single-modality models and various fusion operators.
Accuracy and Efficiency
As shown in Figure 2, PoolFuser achieved the highest accuracy among all tested operators, with an mAP of 0.384 and an AP50 of 0.803. Notably, it outperformed the Transformer-based TransFuser (0.369 mAP) while being significantly more efficient. PoolFuser reduced inference latency by approximately 31% (from 0.0712s to 0.0490s) and saved 241 MB of GPU memory compared to TransFuser.
Key Insights
- Structural Importance: The results reinforce the MetaFormer hypothesis: the macro-architecture of the Transformer is a powerful feature extractor even when the complex self-attention mechanism is replaced by simpler pooling.
- Modality Contribution: Baseline tests showed that "Mono IR" (0.327 mAP) significantly outperformed "Mono RGB" (0.195 mAP), confirming that IR is the primary signal carrier in low-light pedestrian detection.
- Optimal Trade-off: PoolFuser occupies the "Goldilocks" zone, providing the global context necessary for high accuracy without the prohibitive computational costs of standard Transformers.
Conclusion and Future Work
PoolFuser represents a significant step toward deploying robust, multi-modal object detection on edge hardware. By leveraging a lightweight pooling-based spatial mixer, it achieves state-of-the-art performance on low-light pedestrian detection with reduced latency and memory overhead.
Future research will focus on extending PoolFuser to other detection frameworks, such as YOLO or SSD, and evaluating its robustness across a wider range of environmental conditions, including fog, rain, and snow. Additionally, exploring other lightweight spatial mixers, such as depth-wise convolutions, may further optimize the accuracy-to-latency ratio.
Summary updated