Multi-Modal Latent Diffusion using Information Disentangling Autoencoders
Lists
Only you see this
Abstract
While real world data frequently comprises multiple distinct modalities, most generative AI operates primarily with a single modality. Moreover, existing multimodal generation works primarily tackle either text-image generation or conditional modality generation. We extend the latent diffusion framework into the multimodal image domain, facilitating consistent joint generation of two or more modalities. Based on the idea that two modalities comprise some shared information and some information unique to each modality, we develop a variational autoencoder architecture and training scheme to generate an effective latent space for diffusion generation. We demonstrate generation of both RGB & semantic segmentation and RGB & depth map modality pairs. We find our disentangled latent space structure to improve generative quality and ensure consistency between modalities.
BibTeX
@misc{siromani2025multi,
author = {Daniel Siromani and Pier Luigi Dragotti},
title = {{Multi-Modal Latent Diffusion using Information Disentangling Autoencoders}},
year = {2025},
doi = {10.1109/camsap66162.2025.11423982},
}Loading related content...