2D to 3D Medical Image Colorization

Aradhya Neeraj Mathur, Apoorv Khattar, Ojaswa SharmaWACV 2021

TL;DR

This paper introduces a novel framework for colorizing 3D medical images (like MRIs) using 2D photographic style examples. By combining modality conversion with neural style transfer and optimization-based colorization, they achieve realistic and structurally preserved colorizations, overcoming limitations of existing 2D methods and manual transfer functions.

What Problem Does This Paper Address?

Colorizing images, especially for visualization, has seen significant progress for 2D photographs. However, applying these techniques to 3D medical data, such as MRI scans, presents unique challenges. Medical images have different radiometric characteristics than photographs, and their 3D nature requires specialized handling. Existing methods for 3D visualization often rely on manual "transfer functions" (TFs) that map intensity values to colors, which are cumbersome and time-consuming to create. While automatic and semi-automatic TF synthesis exists, it's still an indirect approach. This paper tackles the problem of automatically generating realistic colorizations for 3D medical volumes, aiming to improve visualization capabilities and bypass the need for manual TF creation.

Methods

The proposed approach is a three-step process:

  1. MRI to Grayscale Conversion (Modality Conversion): Since direct colorization from MRI to color is problematic due to modality differences, the first step is to convert the input MRI volume into a corresponding grayscale "photographic" volume. This is achieved using a 3D Generative Adversarial Network (GAN). The GAN learns to predict how the tissue intensities from the MRI would appear under visible light, effectively creating a synthetic grayscale dataset that aligns with the target modality. This step is crucial for bridging the gap between MRI and photographic appearance.
  2. Style Transfer for Generating Hints: To guide the colorization process, stylized 2D "hints" are generated. This involves applying neural style transfer (based on Gatys et al.'s method) to selected slices from the grayscale volume using a 2D photographic style image. To mitigate issues like color bleeding and irregularities that arise from applying 2D methods to medical data, a total variation loss is incorporated to penalize color channel construction.
  3. Volume Colorization using Hints: The final step is to colorize the entire 3D volume using the generated 2D stylized slices as hints. This leverages an optimization-based colorization method (inspired by Levin et al.) extended to 3D. The colorization is performed in the YUV color space, where the Y channel is derived from the GAN-generated grayscale volume. The optimization ensures that neighboring voxels with similar intensities receive similar colors, preserving the underlying MRI structure while propagating the colors from the stylized hints. For handling large volumes, an upsampling step using SRGAN is employed due to hardware limitations on the maximum input size for the GAN.

Datasets

The primary dataset used is the Visible Human 2 dataset, which contains unregistered 16-bit MRI and 24-bit color cryo imaging data of the human head. The authors perform rigid registration between the MRI and cryo volumes to establish correspondence. They also generate approximately 3000 sub-volumes (32x32x32) for training the GAN. For the grayscale conversion step, they create a synthetic dataset by performing Poisson reconstruction on the MRI volume using the cryo volume as a boundary condition, serving as a "ground truth" for the grayscale appearance.

Key Insights and Ideas

  • Modality Conversion is Key: Directly colorizing MRI to photographic color is ill-posed. Converting MRI to a grayscale "photographic" representation first is a critical intermediate step that makes subsequent colorization feasible and more realistic.
  • 3D GANs for Medical Data: The use of 3D GANs is essential for handling the volumetric nature of medical data and maintaining structural correspondence, which is often lost in 2D-based approaches.
  • Style Transfer as "Hints": Neural style transfer, when adapted with additional constraints (like total variation loss), can effectively generate guiding "hints" for 3D colorization, even when the style image is from a different modality.
  • Optimization for Structure Preservation: Combining style transfer hints with an optimization-based colorization framework allows for the preservation of the original MRI structures while introducing new colors.
  • Bridging 2D and 3D: The framework effectively bridges the gap between 2D style transfer techniques and 3D volumetric data by using stylized 2D slices as hints for a 3D optimization process.

Key Results

  • The proposed method produces photorealistic 3D colorizations that maintain the structural integrity of the original MRI volumes.
  • Qualitative comparisons (Figures 3, 5, and 6) show that their method outperforms exemplar colorization and other style transfer methods on medical data, which often struggle with modality differences and structural preservation.
  • Quantitative comparisons (Table 1) show their method achieving the highest SSIM, PSNR, and lowest MSE among compared style transfer methods, indicating better structural preservation and color consistency. For example, their method achieved an SSIM of 0.319, PSNR of 9.72, and MSE of 0.144.
  • The colorization process is automated, eliminating the need for manual transfer functions.

Limitations and Future Work

  • Lack of Semantic Information: The method is limited by the lack of explicit semantic information for MRI volumes, which hinders context-aware stylization of hints.
  • Computational Resources: Processing large 3D volumes is computationally intensive, requiring upsampling techniques (like SRGAN) to handle hardware limitations.
  • Comparison Challenges: Evaluating medical image colorization is difficult due to the lack of standardized metrics and the inherent differences in modalities.
  • Future Work: The authors suggest that the framework is general enough to be applied to other 3D multi-modal data and opens up avenues for high-fidelity photorealistic visualization and colorization. Further exploration into incorporating semantic information could improve context-aware stylization.

Summary updated

BibTeX
@inproceedings{mathur2021d,
  author = {Aradhya Neeraj Mathur and Apoorv Khattar and Ojaswa Sharma},
  title = {{2D to 3D Medical Image Colorization}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2021},
}
Figure