DEYOLO: Dual-Feature-Enhancement YOLO for Cross-Modality Object Detection

Yishuo Chen, Boran Wang, Xinyu Guo, Wenbin Zhu, J. He, Xiaobin Liu, Jing Yuan, WenbinZhuLecture Notes in Computer Science 2024 · DOI · arXiv · OpenAlexGitHub resource

Abstract

Object detection in uncontrolled environments often faces significant challenges due to poor illumination, such as heavy smoke, nighttime conditions, or intense glare. While infrared (IR) sensors provide critical structural information and heat signatures in these scenarios, effectively fusing them with RGB data remains a complex task. This paper presents DEYOLO, a novel cross-modality object detection framework based on the YOLOv8 architecture. DEYOLO introduces dual-enhancement modules—DECA for semantic features and DEPA for spatial features—to extract complementary information while suppressing mutual interference between modalities. Experimental results on the M³FD, LLVIP, and KAIST datasets demonstrate that DEYOLO significantly outperforms current state-of-the-art fusion-and-detection methods in both accuracy and robustness.

Introduction

Standard RGB cameras are highly sensitive to lighting conditions, often losing visibility in low-light or obscured environments. Infrared sensors can compensate for these deficiencies, yet combining the two modalities is not straightforward. Existing research identifies two primary gaps: first, many methods prioritize "image fusion" (creating visually pleasing images) over "object detection" (optimizing for bounding box accuracy); second, simple concatenation of RGB and IR channels often introduces noise, where a lack of detail in one modality degrades high-quality features in the other.

DEYOLO addresses these issues by performing fusion in the feature space rather than at the input level, allowing the model to maintain modality-specific features longer and use them to guide the detection process more effectively.

Methodology

DEYOLO is built upon the YOLOv8 architecture, enhanced by three primary innovations designed to handle the nuances of cross-modality data.

Dual-Enhancement Modules

The core of the framework lies in two specialized modules that decouple the enhancement of "what" an object is from "where" it is located:

  • DECA (Dual Semantic Enhancing Channel Weight Assignment): This module operates in the channel dimension. It utilizes a dual-enhancement mechanism that first uses single-modality features to refine a mixed feature map, then redistributes weights back to the original modalities. This ensures the network prioritizes the most discriminative semantic channels for classification.
  • DEPA (Dual Spatial Enhancing Pixel Weight Assignment): DEPA focuses on the spatial dimension. By using varying convolutional kernel sizes, it extracts pixel-level weights to emphasize structural details, such as edges, which may be present in one modality but obscured in the other.
The overall architecture of DEYOLO, featuring the dual-enhancement modules (DECA and DEPA) and the Bi-Directional Decoupled Focus.

Bi-directional Decoupled Focus

To improve the receptive field in early layers, the authors introduced a Bi-directional Decoupled Focus module. Unlike standard downsampling, this module slices feature maps horizontally and vertically to retain surrounding pixel information, making the backbone more adaptive to the subsequent dual-enhancement process.

Experimental Results

The authors evaluated DEYOLO on three major pixel-aligned RGB-IR datasets: M³FD (4,200 pairs), LLVIP (16,836 pairs for pedestrian detection), and KAIST (for generalization testing).

Quantitative Performance

DEYOLO demonstrated a clear performance lead over both single-modality detectors and complex fusion-and-detection pipelines:

  • M³FD Dataset: DEYOLO-l achieved an mAP₅₀ of 91.2%, a significant improvement over other state-of-the-art (SOTA) methods like SeAFusion (80.7%) and DetFusion (80.6%).
  • LLVIP Dataset: DEYOLO-n reached 96.8% mAP₅₀, outperforming established models like PIAFusion (96.1%) and U2Fusion (92.2%).

Qualitative Analysis

Visual comparisons highlight DEYOLO's ability to maintain clear object boundaries and detect small targets in high-noise environments where other fusion methods produce blurry or incorrect results.

Qualitative comparison of DEYOLO against other state-of-the-art fusion-and-detection methods across various challenging scenarios.

Conclusion

DEYOLO represents a significant step forward in cross-modality object detection by treating semantic and spatial enhancement as separate, mutually reinforcing processes. By fusing information in the feature space and employing dual-enhancement strategies, the model effectively suppresses inter-modality noise. While the model showed slightly lower performance on the KAIST dataset due to lower thermal sensor quality, its modular components (DECA and DEPA) offer high "plug-and-play" potential for other detection architectures. Future work will focus on extending these modules to more complex multi-sensor environments.

Summary updated

BibTeX
@inproceedings{chen2024deyolo,
  author = {Yishuo Chen and Boran Wang and Xinyu Guo and Wenbin Zhu and J. He and Xiaobin Liu and Jing Yuan and WenbinZhu},
  title = {{DEYOLO: Dual-Feature-Enhancement YOLO for Cross-Modality Object Detection}},
  booktitle = {Lecture Notes in Computer Science},
  year = {2024},
  doi = {10.1007/978-3-031-78447-7\_16},
}

Loading related content...

Figure