E2E-MFD: Towards End-to-End Synchronous Multimodal Fusion Detection

Jiaqing Zhang, Mingxiang Cao, Weiying Xie, Jie Lei, Daixun Li, Wenbo Huang, Yunsong Li, Xue Yang

In the world of autonomous driving and remote sensing, "seeing" isn't always easy. Visible light cameras struggle in fog, rain, or darkness, while infrared sensors provide heat signatures but lack fine texture. The solution is multimodal image fusion (MF)—combining both worlds into one clear picture—and then performing object detection (OD).

Traditionally, this has been a clunky, multi-stage process. A new paper from researchers at Xidian University and their collaborators, titled "E2E-MFD: Towards End-to-End Synchronous Multimodal Fusion Detection," proposes a streamlined, single-stage alternative that is both faster and more accurate.

TL;DR

E2E-MFD is a pioneering end-to-end framework that simultaneously optimizes image fusion and object detection in a single training phase. By aligning task gradients and using a multi-granularity feature tree, it achieves state-of-the-art detection performance (up to a 3.9% mAP increase) while significantly reducing training time.

What Problem Does This Paper Address?

Most current Multimodal Fusion Detection (MFD) systems are "cascaded." They first train a network to fuse images and then, in a separate step, train a detector on those fused images. This approach has two major flaws:

  1. Inefficiency: The multi-step training process is tedious and time-consuming.
  2. Suboptimal Results: The fusion network often optimizes for "visually appealing" images (good for humans) rather than "detection-friendly" images (good for algorithms). Because the tasks are optimized separately, they can reach local optima that don't benefit the final goal of accurate detection.
Figure 1: Comparison of (d) E2E-MFD with existing MF-OD task paradigms (a) Two-Stage (Separate Cascaded), (b) Two-stage (Joint Cascaded) and (c) Multi-stage (Joint Cascaded).

Methods

The authors introduce E2E-MFD, which integrates fusion and detection into a unified architecture optimized synchronously. Key innovations include:

  • Object-Region-Pixel Phylogenetic Tree (ORPPT): Inspired by human visual perception, this fusion network mines features across multiple scales—from fine-grained pixels to broader regions and finally to distinct objects. This ensures the fused image retains both sharp textures and high-level semantic information.
  • Coarse-to-Fine Diffusion Process (CFDP): Instead of a standard detection head, E2E-MFD uses a diffusion-based approach (similar to DiffusionDet). This helps the network focus more effectively on object areas during the fusion process.
  • Gradient Matrix Task-Alignment (GMTA): This is the "secret sauce" for training stability. In multi-task learning, one task's gradients often dominate or conflict with another's. GMTA uses Singular Value Decomposition (SVD) to orthogonalize and balance the gradients of shared parameters, ensuring the model converges to a configuration that benefits both fusion and detection.
Figure 2: An overview of the proposed E2E-MFD framework, which consists of a backbone, nodes, and branches. The backbone is utilized to extract multimodal image features. A fine-grained fusion network (ORPPT) and diffusion-based object detection network (CFDP) are optimized by synchronous joint optimization (GMTA) in an end-to-end manner.

Datasets

The model was evaluated on four widely-used visible-infrared datasets:

  • M3FD: Used for both fusion and horizontal object detection.
  • DroneVehicle: A large-scale dataset for oriented object detection in aerial imagery.
  • TNO & RoadScene: Used specifically to evaluate the quality of the image fusion results.

Key Insights and Ideas

  • Synchronous is Superior: By training fusion and detection together, the detection task provides "semantic feedback" that guides the fusion network to highlight the most important parts of a scene.
  • The Optimization Barrier: The authors identified that the primary hurdle in end-to-end MFD isn't the architecture, but the gradient conflict between low-level (pixel) and high-level (semantic) tasks.
  • Efficiency Gains: Moving to a one-stage process isn't just about accuracy; it drastically simplifies the deployment pipeline for real-world applications like self-driving cars.

Key Results

E2E-MFD outperformed several state-of-the-art (SOTA) methods across the board:

  • Detection Accuracy: On the M3FD dataset, it achieved a 3.9% increase in mAP50 compared to previous SOTA methods. On the DroneVehicle dataset, it saw a 2.0% increase.
  • Training Speed: E2E-MFD is significantly faster to train. It completed joint learning in roughly 2 hours and 50 minutes, whereas competing methods like MetaFusion took nearly 7 hours.
  • Visual Quality: Qualitatively, the fused images show higher contrast and clearer object boundaries, especially in challenging scenarios like occluded pedestrians or overlapping vehicles at night.

Limitations and Future Work

While highly effective, the authors note a few areas for growth:

  • Modality Diversity: The current validation is focused on visible and infrared pairs. The authors plan to explore other modalities (like LiDAR or SAR) in the future.
  • Data Regularization: As with many deep learning models, there is a risk of the model learning harmful content if the training data is biased. Future work will focus on better data regularization to ensure robustness across diverse real-world domains.

For those interested in the implementation, the code is available on GitHub.

Summary updated

Abstract

Multimodal image fusion and object detection are crucial for autonomous driving. While current methods have advanced the fusion of texture details and semantic information, their complex training processes hinder broader applications. Addressing this challenge, we introduce E2E-MFD, a novel end-to-end algorithm for multimodal fusion detection. E2E-MFD streamlines the process, achieving high performance with a single training phase. It employs synchronous joint optimization across components to avoid suboptimal solutions tied to individual tasks. Furthermore, it implements a comprehensive optimization strategy in the gradient matrix for shared parameters, ensuring convergence to an optimal fusion detection configuration. Our extensive testing on multiple public datasets reveals E2E-MFD's superior capabilities, showcasing not only visually appealing image fusion but also impressive detection outcomes, such as a 3.9% and 2.0% mAP50 increase on horizontal object detection dataset M3FD and oriented object detection dataset DroneVehicle, respectively, compared to state-of-the-art approaches. The code is released at https://github.com/icey-zhang/E2E-MFD.
BibTeX
@inproceedings{zhang2024e,
  author = {Jiaqing Zhang and Mingxiang Cao and Weiying Xie and Jie Lei and Daixun Li and Wenbo Huang and Yunsong Li and Xue Yang},
  title = {{E2E-MFD: Towards End-to-End Synchronous Multimodal Fusion Detection}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=47loYmzxep},
}
Figure