LeFusion: Controllable Pathology Synthesis via Lesion-Focused Diffusion Models

Hantao Zhang, Yuhe Liu, Jiancheng Yang, Shouhong Wan, Xinyuan Wang, Wei Peng, Pascal Fua

TL;DR

LeFusion is a novel diffusion-based framework that synthesizes realistic, controllable medical lesions within existing healthy scans. By focusing the model's capacity solely on the pathology and introducing fine-grained texture and mask controls, it generates high-quality synthetic data that significantly boosts the performance of medical image segmentation models like nnU-Net.

What Problem Does This Paper Address?

Training robust AI for healthcare is notoriously difficult due to data scarcity and long-tail imbalances. In the real world, healthy scans are abundant, but pathological cases—especially rare or complex ones—are few and far between. This leads to biased models that perform poorly on the very cases they are meant to detect.

While generative models (like GANs or standard Diffusion Models) can create synthetic data, they face two major hurdles in medical imaging:

  1. Background Integrity: Medical backgrounds (organs, vessels, anatomy) are incredibly complex. Models often waste capacity trying to reconstruct the background, leading to anatomical "hallucinations" or blurry results.
  2. Lack of Control: Researchers need to control the type, size, and location of a lesion to create a diverse and useful training set. Standard models often struggle with "multi-peak" distributions (where one type of lesion, like a lung nodule, can have vastly different textures) and "multi-class" scenarios (where different types of pathology coexist).

Methods

The authors propose LeFusion, a lesion-focused diffusion model that decouples lesion generation from background preservation.

  • Lesion-Focused Diffusion: Instead of generating an entire image, LeFusion uses a redesigned objective function that calculates loss only within the lesion area. During inference, it blends forward-diffused real background context with reverse-diffused synthetic foregrounds. This ensures the background remains 100% anatomically authentic while the model focuses its entire "brainpower" on the lesion.
Figure 2: LeFusion: Lesion-Focused Diffusion Model. The top illustrates the training process of LeFusion, while the bottom shows the inference. During training, LeFusion avoids learning unnec- essary background generation using a lesion-focused loss. In inference, by combining forward- diffused real backgrounds with reverse-diffused generated foregrounds, LeFusion ensures high- quality background generation. Additionally, we introduce histogram-based texture control to han- dle multi-peak lesions and multi-channel decomposition for multi-class lesions.
  • Histogram-Based Texture Control: To handle diverse textures (e.g., solid vs. ground-glass lung nodules), the authors use the grayscale histogram of the lesion as a condition via cross-attention. This prevents the model from "averaging out" different lesion types and allows users to specify the desired texture during generation.
  • Multi-Channel Decomposition: For complex cases like cardiac MRI, where multiple pathologies (like Myocardial Infarction and Microvascular Obstruction) might appear together, LeFusion generates each class in a separate channel before combining them.
  • DiffMask: The authors also introduced a generative model for the lesion masks themselves. By using a "control sphere" (a bounding ball), they can precisely dictate the size and location of a synthetic lesion while letting the model generate a realistic, irregular boundary.
Figure 1: Standard Conditional Diffusion vs. Lesion-Focused Diffusion (LeFusion). (a) Stan- dard Conditional Diffusion concatenates background, lesion mask, and noise. The model gener- ates both lesion and background, risking background integrity and wasting capacity on difficult but unnecessary background generation, especially in data-limited settings. (b) Lesion-Focused Diffu- sion (LeFusion) uses forward-diffused backgrounds and reverse-diffused foregrounds as input. The model reconstructs only the lesion, ensuring realistic background preservation and simplifying the task. (c) LeFusion with Fine Control of Lesion Textures and Masks introduces histogram-based tex- ture control for multi-peak lesions, multi-channel decomposition for multi-class lesions, and lesion mask diffusion for control over size, location and boundary, enhancing lesion quality and diversity.

Datasets

The model was validated on two challenging 3D medical imaging tasks:

  1. LIDC-IDRI: A large dataset of 3D lung nodule CT scans.
  2. Emidec: A multi-class cardiac lesion MRI dataset featuring Myocardial Infarction (MI) and Persistent Microvascular Obstruction (PMO).

Key Insights and Ideas

  • Capacity Allocation: The "aha moment" here is that generating a human body is harder than generating a tumor. By masking the loss function, the authors stop the model from wasting parameters on the "easy" parts (the healthy background) and force it to master the "hard" parts (the pathology).
  • Histograms as a Proxy for Type: Grayscale histograms are a surprisingly effective way to capture lesion "sub-types" without needing explicit manual labels for every variation. As shown in Figure 3, these histograms naturally cluster into clinically meaningful groups like ground-glass or solid nodules.
Figure 3: Illustration of Lung Nodule Texture Histogram Distribution. Samples are clustered into three groups based on the grayscale image histogram of lesions. The visualized differences between groups are significant, indicating a typical multi-peak distribution. These clusters roughly correspond to ground-glass, part-solid, and solid nodules.
  • Controllable Diversity: By combining DiffMask (for shape/location) and Histogram Control (for texture), researchers can "program" a dataset to fill specific gaps in their training data.

Key Results

LeFusion-generated data provided a significant boost to state-of-the-art segmentation models:

  • Lung Nodule Segmentation: Adding LeFusion data improved the Dice score of nnU-Net by 5.18% and SwinUNETR by 4.75% compared to training on real data alone.
  • Cardiac Lesion Segmentation: The multi-channel approach (LeFusion-J) significantly outperformed standard diffusion and hand-crafted methods, particularly in capturing the correlation between different lesion types.
  • Qualitative Superiority: Visual assessments showed that LeFusion avoided the background disruption common in other conditional diffusion models and produced much more realistic textures than traditional "copy-paste" or hand-crafted heuristic methods.
Visualizing the effectiveness of histogram control on lesion appearance.

Limitations and Future Work

While impressive, the authors note a few areas for improvement:

  • Inference Speed: Like most diffusion models, LeFusion is computationally intensive, taking about 30 seconds to generate a single 3D sample on an NVIDIA A100 GPU.
  • Comparison Scope: The authors mentioned that a full comparison with some very recent concurrent studies was difficult due to the lack of available code or models.
  • Future Directions: Promising next steps include scaling the method to even larger 3D volumes and exploring its utility for a wider variety of pathologies beyond lung and heart lesions.

Summary updated

Abstract

Patient data from real-world clinical practice often suffers from data scarcity and long-tail imbalances, leading to biased outcomes or algorithmic unfairness. This study addresses these challenges by generating lesion-containing image-segmentation pairs from lesion-free images. Previous efforts in medical imaging synthesis have struggled with separating lesion information from background, resulting in low-quality backgrounds and limited control over the synthetic output. Inspired by diffusion-based image inpainting, we propose LeFusion, a lesion-focused diffusion model. By redesigning the diffusion learning objectives to focus on lesion areas, we simplify the learning process and improve control over the output while preserving high-fidelity backgrounds by integrating forward-diffused background contexts into the reverse diffusion process. Additionally, we tackle two major challenges in lesion texture synthesis: 1) multi-peak and 2) multi-class lesions. We introduce two effective strategies: histogram-based texture control and multi-channel decomposition, enabling the controlled generation of high-quality lesions in difficult scenarios. Furthermore, we incorporate lesion mask diffusion, allowing control over lesion size, location, and boundary, thus increasing lesion diversity. Validated on 3D cardiac lesion MRI and lung nodule CT datasets, LeFusion-generated data significantly improves the performance of state-of-the-art segmentation models, including nnUNet and SwinUNETR. Code and model are available at https://github.com/M3DV/LeFusion.
BibTeX
@inproceedings{zhang2025lefusion,
  author = {Hantao Zhang and Yuhe Liu and Jiancheng Yang and Shouhong Wan and Xinyuan Wang and Wei Peng and Pascal Fua},
  title = {{LeFusion: Controllable Pathology Synthesis via Lesion-Focused Diffusion Models}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=3b9SKkRAKw},
}
Figure