Flemme: A Flexible and Modular Learning Platform for Medical Images

Guoqing Zhang, Jingyun Yang, Yang LiBIBM 2024 · DOI

TL;DR

This paper introduces Flemme, a flexible and modular learning platform designed to streamline the development and evaluation of deep learning models for medical imaging. By decoupling encoders from model architectures, Flemme allows researchers to rapidly construct and test diverse models for segmentation, reconstruction, and generation tasks, addressing the current laborious manual process.

What Problem Does This Paper Address?

Medical imaging presents unique challenges for deep learning: a scarcity of large, annotated datasets and a variety of imaging modalities. This makes it difficult to train a single, general-purpose model that performs well across different datasets and tasks. Currently, researchers often manually combine various network backbones (like CNNs, Transformers, and State-Space Models) with different architectures (e.g., U-Net, Autoencoders, Diffusion Models) for specific tasks. This process is time-consuming, labor-intensive, and makes it hard to systematically analyze the effectiveness of different components. Flemme aims to solve this by providing a unified, modular platform that simplifies model construction and facilitates fair comparisons.

Methods

Flemme's core innovation lies in its modular design, separating encoders from model architectures.

  • Encoders: These are built using flexible "building blocks" based on three main paradigms:
  • Convolutional Blocks (ConvBlock, ResConvBlock): Standard CNN components for local feature extraction.
  • Transformer Blocks (ViTBlock, SwinBlock): Leveraging Vision Transformers, including Swin Transformer's efficient shifted-window attention for reduced computational complexity.
  • State-Space Model Blocks (MambaBlock, ResMambaBlock): Utilizing Mamba, a recent SSM that offers linear time complexity for modeling long-range dependencies.
  • Architectures: Flemme supports several standard architectures, including:
  • Encoder-Decoder (Base Architecture): A foundational structure for various tasks.
  • Segmentation Model (SeM): An encoder-decoder with segmentation loss.
  • Auto-Encoder (AE): For reconstruction tasks.
  • Denoising Diffusion Probabilistic Model (DDPM): For image generation.
  • Hierarchical Architecture (H-SeM, H-AE): A novel contribution is a hierarchical framework that fuses features vertically across different scales using a "pyramid loss." This aims to better capture both local details and global structures.
  • Context Embedding: The platform supports optional context embedding (e.g., class labels, prompt images, time steps) to condition the models.

Datasets

The paper evaluates Flemme on a variety of medical imaging datasets:

  • Segmentation: CVC-ClinicDB, Echonet, ISIC, TN3K, BraTS21, and ImageCAS.
  • Reconstruction: FastMRI (specifically for knee MRI).
  • Generation: The FastMRI dataset is also used for evaluating generation models.

Key Insights and Ideas

  • Modularity is Key: Decoupling encoders and architectures significantly accelerates experimentation and allows for systematic comparison of different components.
  • Hierarchical Fusion Matters: The proposed hierarchical architecture (H-SeM, H-AE) consistently improves performance across various U-shaped networks, demonstrating the benefit of vertical feature fusion.
  • SSMs Show Promise: Mamba-based encoders (Mamba-U) frequently achieve top performance in segmentation and reconstruction, suggesting their strong potential for medical imaging, especially for high-frequency component prediction.
  • CNNs Remain Relevant: Despite the rise of Transformers and SSMs, convolutional networks (ConvBlock) still offer superior time and memory efficiency, particularly for 2D images, and remain a strong contender for segmentation and reconstruction.
  • U-Net's Robustness: U-Net architectures, when combined with various encoders, show robust performance in segmentation, often outperforming simpler encoder designs.

Key Results

  • Segmentation: Flemme achieved an average improvement of 5.60% in Dice score and 7.81% in mIoU for segmentation models using the hierarchical architecture. Mamba-U consistently ranked among the top performers across multiple datasets.
  • Reconstruction: The hierarchical AE models showed an enhancement of 5.57% in PSNR and 8.22% in SSIM. Mamba-U also achieved the highest reconstruction accuracy.
  • Efficiency: While ConvBlocks are the most efficient in terms of training time and memory, SwinBlocks and MambaBlocks offer competitive performance, especially for 3D tasks, with manageable increases in computational cost.

Limitations and Future Work

  • Computational Cost of 3D: For 3D images, sequence modeling-based encoders (SwinBlock, MambaBlock) experience a significant leap in runtime and memory consumption due to the cubic increase in sequence length.
  • CNNs for Generation: Convolutional encoders perform poorly in generation tasks compared to sequence-modeling backbones.
  • Future Work: The authors plan to expand Flemme to handle diverse data types, including modeling point clouds and graphs, and to develop more efficient algorithms for structured state-space duality. They also aim to improve high-quality medical image generation and automated quality assessment.

Summary updated

BibTeX
@inproceedings{zhang2024flemme,
  author = {Guoqing Zhang and Jingyun Yang and Yang Li},
  title = {{Flemme: A Flexible and Modular Learning Platform for Medical Images}},
  booktitle = {BIBM},
  year = {2024},
  doi = {10.1109/bibm62325.2024.10821981},
}
Figure