AutoFuse: Automatic Fusion Networks for Deformable Medical Image Registration

Mingyuan Meng, Michael Fulham, Dagan Feng, Lei Bi, Jinman KimarXiv 2023 · DOI · arXiv

TL;DR

AutoFuse introduces a data-driven fusion strategy for deformable medical image registration, moving away from traditional, hand-crafted empirical designs. By utilizing a novel Fusion Gate module within a multi-branch network, it automatically learns the optimal locations and weights for combining image features during training. This approach achieves state-of-the-art performance across both unsupervised and semi-supervised registration tasks.

What Problem Does This Paper Address?

Deformable image registration is a foundational task in medical image analysis, essential for tracking tumor growth, monitoring cardiac motion, and aligning anatomical structures across different patients. Unlike natural image alignment (like stitching a panorama), medical images exhibit complex, non-linear local deformations that require highly dense spatial mapping.

While Deep Neural Networks (DNNs) have made end-to-end registration incredibly fast, they face a fundamental architectural bottleneck: how and when should the network fuse information from the fixed and moving images?

Historically, researchers have relied on hand-crafted, empirical fusion strategies. These include "early fusion" (concatenating images at the input layer), "middle fusion" (fusing intermediate features), or "late fusion" (combining deep latent representations).

Fig. 1. Illustration of existing empirically-designed fusion strategies (a-f) and our data-driven fusion strategy (g) in unsupervised and semi-

The problem is that the optimal fusion strategy varies heavily depending on the dataset, the organ being imaged, and the severity of the deformations. Manually searching this massive design space is impractical, meaning empirical designs inherently cap a network's potential performance.

Methods

The authors propose AutoFuse (Automatic Fusion Network), an architecture that shifts the responsibility of fusion design from the human researcher to the data itself.

The Three-Branch Architecture

Instead of a standard single-stream encoder-decoder, AutoFuse uses a three-branch U-Net-style structure:

  1. Branch $B_f$ & Branch $B_m$: Two separate streams that independently extract features from the fixed ($I_f$) and moving ($I_m$) images.
  2. Branch $B_{fuse}$: A central stream that takes the concatenated input images and refines them.

Fusion Gate (FG) Module

At multiple scales throughout the network, AutoFuse embeds a novel Fusion Gate (FG) module. The FG module acts as a programmable switch. It uses an attention mechanism to dynamically compute adaptive weight maps ($w_{mf}$ and $w_{fuse}$). These maps determine exactly how much information to pull from the independent image branches versus the joint fusion branch.

To refine these fused features without blowing up the memory budget, the authors introduce an Efficient Large Kernel (ELK) block, which breaks down a massive 3D convolution into parallel directional kernels.

Fig. 2. Overview of the proposed Automatic Fusion network (AutoFuse), including the architecture of (a) AutoFuse, (b) Fusion Gate (FG) modules,

Architectural Flexibility

AutoFuse is framework-agnostic. The authors successfully implemented it using standard Convolutional Neural Networks (CNNs) as well as a Transformer-based variant (AutoFuse-Trans) utilizing Swin Transformer blocks to capture long-range spatial dependencies.

Datasets

The model was rigorously evaluated on two distinct clinical registration challenges across eight public datasets:

  • 3D Inter-Patient Brain MRI Registration: Trained on a massive semi-supervised dataset combining 314 labeled images from OASIS and 2,656 unlabeled images from ADNI, ABIDE, ADHD, and IXI. Independent testing was conducted on the Mindboggle and Buckner datasets.
  • 4D Intra-Patient Cardiac Cine-MRI Registration: Evaluated using the ACDC dataset, which tracks cardiac cycles from End-Diastole (ED) to End-Systole (ES).

Key Insights and Ideas

  • Dynamic Adaptation: Rather than settling on a single compromise strategy, AutoFuse alters its fusion behavior based on feature resolution. Model interpretation revealed that at lower resolutions (deeper in the network), the gate relies more heavily on the independent single-image branches ($B_f, B_m$). This suggests that downsampling operations degrade joint spatial correspondence, requiring the network to look back at the raw, un-fused feature representations to rebuild alignment.
  • Algorithmic Synergy: In semi-supervised settings, AutoFuse performs joint registration and segmentation. The learned segmentation masks are fed back into the input layer of the fusion branch, allowing the network to automatically discover the optimal synergy between understanding what an organ is (segmentation) and where it is moving (registration).

Key Results

AutoFuse consistently outperformed both traditional registration frameworks (like SyN and NiftyReg) and modern deep-learning baselines (including VoxelMorph and TransMorph) in both accuracy (Dice Similarity Coefficient, DSC) and transformation invertibility (Negative Jacobian Determinants, NJD).

  • Unsupervised Brain Registration (OASIS): AutoFuse achieved a DSC of 80.8% (and 81.3% for AutoFuse-Trans), significantly outperforming the baseline VoxelMorph (77.3%) and the sophisticated TransMatch (79.8%).
  • Semi-Supervised Cardiac Registration (ACDC): AutoFuse-Trans achieved a top-tier DSC of 86.2%, outperforming the state-of-the-art semi-supervised GL-Net (83.2%).
  • Smoothness and Feasibility: Crucially, AutoFuse achieved these accuracy gains while maintaining highly smooth, physically plausible deformation fields, keeping the percentage of negative Jacobian determinants exceptionally low (~0.02% to 0.06%).

Limitations and Future Work

While AutoFuse successfully optimizes where to fuse information within the network topology, the how is still relatively simple (relying primarily on standard convolutions and element-wise additions within the Fusion Gate). The authors note that future work will explore expanding the gate's search space to include more complex fusion operations, such as cross-attention transformers directly inside the gate modules. Additionally, the framework has yet to be validated on other highly deformable structures, such as lung or thoracic imaging.

Summary updated

Abstract

Deformable image registration aims to find a dense non-linear spatial correspondence between a pair of images, which is a crucial step for many medical tasks such as tumor growth monitoring and population analysis. Recently, Deep Neural Networks (DNNs) have been widely recognized for their ability to perform fast end-to-end registration. However, DNN-based registration needs to explore the spatial information of each image and fuse this information to characterize spatial correspondence. This raises an essential question: what is the optimal fusion strategy to characterize spatial correspondence? Existing fusion strategies (e.g., early fusion, late fusion) were empirically designed to fuse information by manually defined prior knowledge, which inevitably constrains the registration performance within the limits of empirical designs. In this study, we depart from existing empirically-designed fusion strategies and develop a data-driven fusion strategy for deformable image registration. To achieve this, we propose an Automatic Fusion network (AutoFuse) that provides flexibility to fuse information at many potential locations within the network. A Fusion Gate (FG) module is also proposed to control how to fuse information at each potential network location based on training data. Our AutoFuse can automatically optimize its fusion strategy during training and can be generalizable to both unsupervised registration (without any labels) and semi-supervised registration (with weak labels provided for partial training data). Extensive experiments on two well-benchmarked medical registration tasks (inter- and intra-patient registration) with eight public datasets show that our AutoFuse outperforms state-of-the-art unsupervised and semi-supervised registration methods.
BibTeX
@misc{meng2023autofuse,
  author = {Mingyuan Meng and Michael Fulham and Dagan Feng and Lei Bi and Jinman Kim},
  title = {{AutoFuse: Automatic Fusion Networks for Deformable Medical Image Registration}},
  year = {2023},
  eprint = {2309.05271},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2309.05271},
}
Figure