Uncovering Hidden Subspaces in Video Diffusion Models Using Re-Identification

Mischa Dombrowski, Hadrien Reynaud, Bernhard KainzarXiv 2024 · DOI · arXiv

Introduction

Latent Video Diffusion Models (LVDMs) represent a significant advancement in generative AI, particularly for medical imaging where data scarcity and privacy are critical. However, synthetic datasets generated by these models often fail to match the performance of real-world data in downstream tasks. This paper investigates this "performance gap" and addresses the computational inefficiencies and privacy risks associated with high-resolution video generation. The authors hypothesize that LVDMs may only be learning a limited "subspace" of the training data diversity, leading to reduced model recall.

Methodology: Leveraging the Latent Space

The core innovation of this work is the shift from raw image-space analysis to the latent space of a Variational Autoencoder (VAE). By performing privacy-filtering and re-identification (Re-ID) tasks on compressed embeddings rather than high-resolution frames, the researchers achieved significant efficiency gains.

The proposed video generation, latent privacy modeling, and evaluation pipeline.

Latent Privacy Model $\mathcal{S}$

The researchers developed a Siamese neural network based on a ResNet-50 architecture to perform Re-ID. This model learns to distinguish whether two frames belong to the same video or different ones. This capability is used for two primary purposes:

  1. Privacy Filtering: Establishing a threshold ($P_{Max}$) to identify synthetic videos that are "too similar" to training samples, thereby preventing data leakage.
  2. Measuring Model Recall: Quantifying how much of the training data's diversity is actually captured by the generative model. If a training video has a synthetic "closest neighbor" within a certain confidence threshold, it is considered "learned."

Key Results and Insights

The transition to latent space analysis yielded a 60x speedup in training time (from 50 hours to 50 minutes) and reduced memory requirements by a factor of 64.

The "Subspace" Discovery

The most significant finding is the low "Model Recall." Across datasets like EchoNet-Dynamic and EchoNet-Pediatric, the models only successfully "learned" between 18.8% and 30.8% of the training videos. This suggests that current diffusion models compress training data into a narrow variety of outputs, explaining why synthetic datasets often lack the robustness of real ones.

Robustness of Latent Re-ID

The latent-based Re-ID model demonstrated exceptional performance and generalization. Unlike image-space models, which often struggle with domain shifts, the latent representations focus on essential anatomical features.

Confusion matrices for the Re-ID model performance in the latent space across three echocardiography datasets.

As shown in the confusion matrices above, the model achieves high accuracy in identifying video membership across different views (Dynamic, A4C, PSAX). Furthermore, the models maintained high AUC (0.99+) even when evaluated on entirely different medical datasets, proving their robustness.

Conclusion

This research highlights a fundamental limitation in current video diffusion models: they only explore a small fraction of the available data diversity. By utilizing the latent space for evaluation, the authors provide a computationally efficient framework for measuring privacy and recall. Future work aims to leverage these insights to expand the "hidden subspaces" covered by generative models, ultimately closing the performance gap between synthetic and real-world medical data.

Summary updated

Abstract

Latent Video Diffusion Models can easily deceive casual observers and domain experts alike thanks to the produced image quality and temporal consistency. Beyond entertainment, this creates opportunities around safe data sharing of fully synthetic datasets, which are crucial in healthcare, as well as other domains relying on sensitive personal information. However, privacy concerns with this approach have not fully been addressed yet, and models trained on synthetic data for specific downstream tasks still perform worse than those trained on real data. This discrepancy may be partly due to the sampling space being a subspace of the training videos, effectively reducing the training data size for downstream models. Additionally, the reduced temporal consistency when generating long videos could be a contributing factor. In this paper, we first show that training privacy-preserving models in latent space is computationally more efficient and generalize better. Furthermore, to investigate downstream degradation factors, we propose to use a re-identification model, previously employed as a privacy preservation filter. We demonstrate that it is sufficient to train this model on the latent space of the video generator. Subsequently, we use these models to evaluate the subspace covered by synthetic video datasets and thus introduce a new way to measure the faithfulness of generative machine learning models. We focus on a specific application in healthcare echocardiography to illustrate the effectiveness of our novel methods. Our findings indicate that only up to 30.8% of the training videos are learned in latent video diffusion models, which could explain the lack of performance when training downstream tasks on synthetic data.
BibTeX
@misc{dombrowski2024uncovering,
  author = {Mischa Dombrowski and Hadrien Reynaud and Bernhard Kainz},
  title = {{Uncovering Hidden Subspaces in Video Diffusion Models Using Re-Identification}},
  year = {2024},
  eprint = {2411.04956},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2411.04956},
}
Figure