The Swiss Army Knife for Image-to-Image Translation: Multi-Task Diffusion Models

Julia Wolleb, Robin Sandkühler, Florentin Bieder, Philippe C. CattinarXiv 2022 · DOI · arXiv

Abstract

Image-to-image translation traditionally requires task-specific models, often leading to training instability and poor generalization across diverse domains. This paper introduces a flexible framework utilizing a single pre-trained Denoising Diffusion Implicit Model (DDIM) to perform multiple translation tasks. By guiding the deterministic reverse diffusion process with gradients from external, task-specific networks—such as regression for continuous attributes or segmentation for spatial features—the method achieves precise manipulation while preserving the original image's structural integrity and identity. We demonstrate the efficacy of this "Swiss Army Knife" approach in facial aging and medical anomaly simulation, highlighting its potential as a modular tool for generative modeling.

Introduction

The goal of image-to-image translation is to modify specific characteristics of an image while maintaining "faithfulness" to the original content, such as background details and subject identity. Historically, Generative Adversarial Networks (GANs) have dominated this space. However, GANs are notoriously difficult to train, sensitive to hyperparameters, and typically require a unique model for every specific task.

Recent advances in diffusion models have provided a powerful alternative, offering superior image quality and diversity. Yet, these models often struggle with maintaining strict identity during translation or require complex manipulations of the latent space. This research proposes a modular framework where a fixed generative engine is steered by interchangeable task-specific "attachments." This approach allows a single pre-trained model to handle various tasks on the fly, ranging from facial aging to the simulation of tumor growth in medical scans.

Methodology

The core of the proposed method lies in the use of Denoising Diffusion Implicit Models (DDIMs), which allow for a deterministic mapping between image space and noise space. The process is divided into two primary phases:

1. Deterministic Encoding

The input image is converted into a noisy representation ($x_L$) using the DDIM forward process. Because this process is deterministic, the resulting noise "stores" the essential structural information and identity of the original image. This ensures that when the image is reconstructed, the non-target features remain stable.

2. Guided Denoising

During the reverse diffusion process (denoising), the model does not operate in a vacuum. Instead, it is guided by gradients from an external, task-specific model:

  • Regression Guidance: To change a continuous attribute like age, a separate regression model predicts the age of the current noisy image. The gradient of the loss between this prediction and the target age "pushes" the diffusion model to generate a version of the face that matches the target.
  • Segmentation Guidance: For tasks requiring spatial precision, such as "painting" a tumor into a brain scan, a segmentation model calculates the loss between its prediction and a user-defined target mask. This guides the diffusion model to manifest the anomaly only within the specified region.
The proposed multi-task diffusion framework architecture.

Crucially, the diffusion model acts as a general-purpose prior for "what images look like," while the external models act as "directors." This modularity means the core model never needs retraining; one simply swaps the guidance tool.

Datasets and Experimental Setup

The researchers evaluated the framework across two distinct and challenging domains:

  • Facial Photos: Utilizing a dataset of over 185,000 images of individuals aged 1 to 100. Images were normalized to a resolution of $128 \times 128$ pixels.
  • Medical Imaging (BRATS2020): This dataset contains 3D brain Magnetic Resonance (MR) scans of patients with tumors. The authors utilized 2D axial slices across four different MR sequences—T1, T1ce, T2, and FLAIR—at a resolution of $256 \times 256$.

Key Results and Insights

The experimental results underscore the versatility and precision of the guided diffusion approach:

  • Facial Aging and De-aging: The model successfully translated 40-year-old faces to target ages ranging from 10 to 80. Qualitatively, the outputs show realistic skin texture changes and structural aging while maintaining recognizable identity. The bidirectional nature of the guidance allows for both aging and de-aging using the same setup by simply changing the target value.
  • Medical Anomaly Simulation: On the BRATS2020 dataset, the model accurately scaled "relative tumor size." By providing a target ratio, the model could realistically grow or shrink tumors across all four MR sequences simultaneously, ensuring multi-modal consistency.
  • Spatial Control: Using segmentation guidance, the authors successfully inserted tumors into healthy brain slices at specific locations. This has significant implications for generating synthetic data to train anomaly detection algorithms where real-world data is scarce.
Example of high-fidelity facial translation output.

Discussion and Future Work

While the framework offers a powerful "Swiss Army Knife" for image translation, several limitations remain. The primary bottleneck is sampling speed; generating a single translated image takes between 73 and 102 seconds on an NVIDIA Quadro RTX 6000. While acceptable for research, this is significantly slower than the near-instantaneous inference of GANs.

Furthermore, the current work is limited to 2D slices. In medical contexts, 3D volumetric consistency is vital, and extending the framework to 3D data is a priority for future research. Finally, the "gradient scale" ($c$) remains a sensitive hyperparameter; finding the optimal balance between translation strength and image quality is essential for robust performance across different tasks.

Summary updated

Abstract

Recently, diffusion models were applied to a wide range of image analysis tasks. We build on a method for image-to-image translation using denoising diffusion implicit models and include a regression problem and a segmentation problem for guiding the image generation to the desired output. The main advantage of our approach is that the guidance during the denoising process is done by an external gradient. Consequently, the diffusion model does not need to be retrained for the different tasks on the same dataset. We apply our method to simulate the aging process on facial photos using a regression task, as well as on a brain magnetic resonance (MR) imaging dataset for the simulation of brain tumor growth. Furthermore, we use a segmentation model to inpaint tumors at the desired location in healthy slices of brain MR images. We achieve convincing results for all problems.
BibTeX
@misc{wolleb2022swiss,
  author = {Julia Wolleb and Robin Sandkühler and Florentin Bieder and Philippe C. Cattin},
  title = {{The Swiss Army Knife for Image-to-Image Translation: Multi-Task Diffusion Models}},
  year = {2022},
  eprint = {2204.02641},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2204.02641},
}
Figure