DMK-SSN at ImageCLEF 2023 Medical: Controlling the Quality of Synthetic Medical Images Created via GANs using Machine Learning and Image Hashing Techniques

Shriram Manathattai Sathiyanarayanan, Dhivya Subburam, Shriram M. SathyaNarayanan, Bhavana Anand, Kavitha Srinivasan, Mohanavalli SubramaniamClef (working Notes) 2023

Introduction

As Generative Adversarial Networks (GANs) and Diffusion models become staples for augmenting scarce medical datasets, a critical question arises: do these synthetic images "leak" the private data of the real patients used to train them? This research explores the "fingerprints" left behind in synthetic lung CT scans, using a combination of Convolutional Neural Networks (CNNs), traditional computer vision (SIFT+KNN), and perceptual hashing to map artificial images back to their real-world sources.

Problem Statement

In the medical field, data is gold but privacy is paramount. Researchers often use GANs to generate synthetic datasets that mimic real pathologies, such as tuberculosis, to train diagnostic AI without compromising patient confidentiality. However, a persistent hypothesis suggests that these generative models don't just learn the abstract concept of a disease; they might "memorize" and replicate distinct characteristics of the specific real images used during training.

If a synthetic image is too similar to a real one, it poses a significant security and ethical risk, as the synthetic data would still be subject to the same strict privacy restrictions as the original patient data. The authors participated in the ImageCLEF 2023 Medical GANs Lab to determine if they could identify which real images were "used" versus "unused" in the creation of a synthetic dataset.

Methodology

The team from Sri Sivasubramaniya Nadar College of Engineering employed three distinct approaches to detect data leakage:

  1. Convolutional Neural Network (CNN): A relatively simple 3-layer architecture designed to capture spatial features. It was trained as a binary classifier to distinguish between "used" and "unused" images based on learned patterns.
  2. SIFT + KNN: A hybrid of traditional computer vision and machine learning. They used Scale-Invariant Feature Transform (SIFT) to identify robust keypoints in the images (invariant to rotation or scaling) and then used a K-Nearest Neighbors (KNN) classifier to categorize them.
  3. Perceptual Hashing (pHash): This was the most "forensic" approach. Unlike cryptographic hashes, perceptual hashes create a "fingerprint" based on the image's frequency content using Discrete Cosine Transform. By comparing the Hamming distance between the hashes of synthetic and real images, the researchers could look for near-matches.
Overview of the machine learning approaches (CNN and SIFT+KNN) used for classification.
Workflow for the image hashing and Hamming distance comparison approach.

Dataset and Experimental Setup

The study utilized the benchmarking dataset provided by ImageCLEF 2023, consisting of axial slices of 3D CT scans from lung tuberculosis patients.

  • Source Material: 8,000 axial slices.
  • Development Set: 500 synthetic images, 80 real images "used" for training, and 80 real images "not used."
  • Test Set: 10,000 synthetic images and 200 unlabeled real images.
  • Format: All images were $256 \times 256$ pixels in PNG format.

Key Insights

  • The "50" Threshold: Through pHash analysis, the authors discovered that a Hamming distance of approximately 50 served as a critical divider. A distance significantly below 50 strongly indicated that a real image was the "source" or "parent" for a specific synthetic output.
  • Memorization vs. Generalization: The ability to map synthetic images back to real counterparts confirms that generative models (in this case, Diffusion models) do indeed retain "fingerprints" of their training data, which has major implications for medical data de-identification.
  • Traditional vs. Deep Learning: Interestingly, the SIFT+KNN approach outperformed the CNN in terms of raw accuracy (0.61 vs 0.535), suggesting that local, discrete feature matching might be more effective for this specific "fingerprint" detection task than the global feature extraction of a standard CNN.

Results

Performance was evaluated using F1 scores and accuracy on the task of identifying "used" real images:

Method Accuracy Precision Recall F1 Score
CNN (Submission 1) 0.535 0.544 0.43 0.4804
SIFT + KNN (Submission 2) 0.61 0.512 0.4 0.449

While the CNN achieved a slightly better F1 score, the SIFT+KNN approach was more accurate overall. The pHash method proved invaluable for qualitative verification, successfully mapping specific synthetic files (e.g., generated_0480.png) to their real-world origins (e.g., used_0027.png).

Conclusion and Future Work

The results serve as a proof-of-concept, though the F1 scores (below 0.5) highlight the extreme difficulty of this task. The "fingerprints" are subtle and often obscured by the generative process. Future work will explore few-shot learning to detect fingerprints with fewer examples and cross-modality testing (MRI, Ultrasound) to determine if this behavior is consistent across different medical imaging types.

Summary updated

BibTeX
@inproceedings{sathiyanarayanan2023dmk,
  author = {Shriram Manathattai Sathiyanarayanan and Dhivya Subburam and Shriram M. SathyaNarayanan and Bhavana Anand and Kavitha Srinivasan and Mohanavalli Subramaniam},
  title = {{DMK-SSN at ImageCLEF 2023 Medical: Controlling the Quality of Synthetic Medical Images Created via GANs using Machine Learning and Image Hashing Techniques}},
  booktitle = {Clef (working Notes)},
  year = {2023},
}
Figure