SynStitch: a Self-Supervised Learning Network for Ultrasound Image Stitching Using Synthetic Training Pairs and Indirect Supervision

Xing Yao, Runxuan Yu, Dewei Hu, Hao Yang, Ange Lou, Jiacheng Wang, Daiwei Lu, Gabriel Arenas, Baris U. Oguz, Alison M. Pouch, Nadav Schwartz, Brett C. Byram, Ipek Oguz

TL;DR

This paper introduces SynStitch, a novel self-supervised learning framework for stitching 2D ultrasound (US) images. It tackles the challenge of limited field-of-view in US imaging by generating realistic synthetic stitching pairs using a ControlNet-based module and then training an image stitching module on this synthetic data, achieving superior results compared to existing methods.

What Problem Does This Paper Address?

Ultrasound (US) imaging is a crucial diagnostic tool, but its limited field-of-view (FOV) often hinders comprehensive anatomical visualization and analysis. Image stitching aims to overcome this by combining multiple US images to create a wider FOV. However, accurately registering US images is difficult due to partially overlapping anatomical content, uncontrolled motion, and artifacts like noise and shadows. Existing stitching methods, whether feature-based, intensity-based, or tracker-based, struggle with the unique sector-shaped FOV of US probes, often leading to misalignment of anatomical structures while focusing on aligning the FOV boundaries. Furthermore, many advanced methods require extensive manual annotations, limiting their generalizability. This paper addresses the need for a robust, self-supervised deep learning approach to US image stitching that can learn effectively without manual annotations and overcome the challenges posed by the US FOV.

Methods

SynStitch is a two-module framework:

  1. Synthetic Stitching Pair Generation Module (SSPGM): This module is built upon the ControlNet architecture, a powerful tool for conditional image generation. SSPGM takes a single input US image and, using a patch-conditioned approach, generates a realistic synthetic image that forms a stitching pair with the original. The key innovation here is the use of ControlNet to synthesize an image that not only resembles the input in anatomical content but also incorporates a known affine transformation (translation, rotation, scaling). This synthetic pair, consisting of the original image and its affinely transformed counterpart, serves as ground truth for training the stitching module.
  2. Image Stitching Module (ISM): This module is designed as a plug-and-play component. It's trained in a supervised manner using the synthetic pairs generated by SSPGM. The ISM learns to align the "moving" image to the "fixed" image by minimizing the difference between the aligned image and the ground truth, effectively learning to correct for the affine transformation. This approach emphasizes aligning the shared anatomical content, mitigating the issues caused by the FOV shape.

Datasets

  • SSPGM Training Dataset: Extracted from 3D US kidney volumes, this dataset consists of central 2D slices from coronal and axial planes, resulting in 7478 training and 2494 validation images.
  • RealCurated Dataset: A subset of the SSPGM training data, curated to ensure noticeable probe motion within images by selecting either coronal or axial slices from the same 3D volume. Used for generating synthetic pairs.
  • SynStitch Dataset: Generated using the RealCurated Dataset and the pre-trained SSPGM. This dataset comprises synthetic stitching pairs and their corresponding ground truth affine matrices, used for training and validating the ISM.
  • RealTrain Dataset: Dynamically extracted real stitching pairs from the RealCurated Dataset for training baseline affine registration methods.
  • RealStitch Dataset: A benchmark dataset of 60 real 2DUS stitching pairs from 5 subjects, featuring manually annotated keypoints and manual alignments, used for evaluating the proposed method and baselines.

Key Insights and Ideas

  • Synthetic Data as a Powerful Supervision Signal: The core idea is that generating realistic synthetic stitching pairs with known ground truth transformations can provide sufficient supervision for training a robust image stitching network, bypassing the need for manual annotations on real data.
  • ControlNet for Anatomically Consistent Synthesis: Leveraging ControlNet's conditional generation capabilities allows for the creation of synthetic images that not only mimic the affine transformation but also maintain the anatomical integrity of the original image, which is crucial for accurate stitching.
  • Focusing on Anatomy over FOV: By training on synthetic pairs where anatomical content is aligned according to the ground truth affine matrix, the ISM learns to prioritize the alignment of structures rather than just the boundaries of the US probe's FOV, a common pitfall in existing methods.
  • Self-Supervised Learning for Generalizability: The self-supervised nature of SynStitch, relying on synthetically generated data, makes it more adaptable to different US datasets and potentially different anatomical regions without requiring extensive manual labeling for each new scenario.

Key Results

  • Qualitative Improvement: Figure 3 visually demonstrates that SynStitch accurately aligns anatomical structures, even with significant motion and challenging FOV shapes, outperforming conventional and other deep learning-based methods that tend to align the FOV but miss anatomical details.
  • Quantitative Superiority: Table 1 shows that SynStitch (both SynStitch-VXM and SynStitch-GLN variants) significantly outperforms baseline methods across key metrics like Mean Squared Error (MSE), Structural Similarity Index Measure (SSIM), Normalized Cross-Correlation (NCC), and Root Mean Square Error (RMSE) of keypoints on the RealStitch Dataset. For instance, SynStitch-GLN achieved an RMSE of 3.36 ± 1.94*, a significant improvement over the best baseline RMSE of 5.01 ± 2.62.
  • Effective Integration: The paper shows that SynStitch can effectively integrate with existing self-supervised learning frameworks like VoxelMorph and GlobalNet, enhancing their performance for 2DUS stitching.

Limitations and Future Work

  • Shearing in SynStitch-VXM: The paper notes that the SynStitch-VXM variant exhibits slight shearing relative to the ground truth, suggesting potential areas for refinement.
  • Future Work: The authors plan to validate SynStitch on additional models and datasets, and extend its application to other medical imaging domains such as fetoscopy and retinal imaging. They also aim to explore its potential in 3D US stitching.

Summary updated

Abstract

Ultrasound (US) image stitching can expand the field-of-view (FOV) by combining multiple US images from varied probe positions. However, registering US images with only partially overlapping anatomical contents is a challenging task. In this work, we introduce SynStitch, a self-supervised framework designed for 2DUS stitching. SynStitch consists of a synthetic stitching pair generation module (SSPGM) and an image stitching module (ISM). SSPGM utilizes a patch-conditioned ControlNet to generate realistic 2DUS stitching pairs with known affine matrix from a single input image. ISM then utilizes this synthetic paired data to learn 2DUS stitching in a supervised manner. Our framework was evaluated against multiple leading methods on a kidney ultrasound dataset, demonstrating superior 2DUS stitching performance through both qualitative and quantitative analyses. The code will be made public upon acceptance of the paper.
BibTeX
@inproceedings{yao2025synstitch,
  author = {Xing Yao and Runxuan Yu and Dewei Hu and Hao Yang and Ange Lou and Jiacheng Wang and Daiwei Lu and Gabriel Arenas and Baris U. Oguz and Alison M. Pouch and Nadav Schwartz and Brett C. Byram and Ipek Oguz},
  title = {{SynStitch: a Self-Supervised Learning Network for Ultrasound Image Stitching Using Synthetic Training Pairs and Indirect Supervision}},
  booktitle = {ISBI},
  year = {2025},
  url = {https://openreview.net/forum?id=tMp6XV045Q},
}
Figure