MANGO: Multimodal Attention-based Normalizing Flow Approach to Fusion Learning

Thanh-Dat Truong, Christophe Bobda, Nitin Agarwal, Khoa Luu

Multimodal learning—combining data from different sources like images, text, and depth maps—is the backbone of modern AI, from self-driving cars to advanced chatbots. However, most current "fusion" methods rely on Transformers to mix these signals implicitly. While effective, this "black box" approach makes it hard to understand how different modalities interact or to capture their complex underlying structures.

Researchers have introduced MANGO (Multimodal Attention-based Normalizing Flow), a new framework that moves away from implicit mixing toward explicit, interpretable, and tractable multimodal fusion.

TL;DR

MANGO leverages Normalizing Flows and a novel Invertible Cross-Attention (ICA) layer to explicitly model the joint distribution of multimodal data. By combining the expressiveness of attention with the mathematical rigor of flows, MANGO achieves state-of-the-art results in semantic segmentation, image translation, and genre classification.

What Problem Does This Paper Address?

Most multimodal models use "implicit fusion." They feed different data streams into a Transformer, which learns correlations through training on massive datasets. However, this has several drawbacks:

  1. Alignment-Agnosticism: The model might not fully exploit the specific alignments between, say, a pixel in an RGB image and a corresponding point in a depth map.
  2. Modality Dominance: If one modality (like text) is much stronger or more abundant, the model might ignore the others.
  3. Lack of Interpretability: It is difficult to "see" how the model decides to weigh one modality over another.

The authors propose using Normalizing Flows (NF) to solve this. NFs use a series of bijective (invertible) mappings to transform complex data into a simple probability distribution. This allows for explicit modeling of the data's likelihood. The challenge? Standard NFs use simple "coupling layers" that aren't powerful enough to capture the complex dependencies found in multimodal data.

Methods: The Secret Sauce of MANGO

The core innovation of MANGO is the Invertible Cross-Attention (ICA) layer. This layer brings the power of Transformer-style attention into the invertible world of Normalizing Flows.

The MANGO architecture uses a series of invertible blocks to map multimodal features into a latent space.

1. Invertible Cross-Attention (ICA)

To make attention invertible, the authors formulated it as an auto-regressive process. By using an upper triangular matrix during the attention calculation, they ensure that the transformation can be mathematically reversed. This allows the model to capture long-range dependencies across modalities while remaining a valid Normalizing Flow.

The ICA layer ensures that information from one partition is embedded into another in a way that can be perfectly undone during the inverse pass.

2. Three Partitioning Strategies

How you split the data determines what the attention "sees." MANGO introduces three ways to partition multimodal tokens:

  • Modality-to-Modality (MMCA): One modality (e.g., RGB) acts as the "query" to attend to the other (e.g., Depth).
  • Inter-Modality (IMCA): Tokens from both modalities are mixed into partitions, forcing the model to learn deeper cross-modal correlations.
  • Learnable Inter-Modality (LICA): Uses a learnable permutation matrix (optimized via LU decomposition) to find the most efficient way to mix modalities before attention.

3. Scaling with Latent Flows

Normalizing flows are notoriously memory-intensive. To handle high-dimensional data (like high-res images), MANGO uses a Latent Normalizing Flow approach. It first compresses the raw data into a lower-dimensional latent space using a pre-trained encoder and then performs the fusion flows in that compressed space.

Key Insights and Results

The researchers evaluated MANGO across three distinct tasks to prove its versatility: Semantic Segmentation (NYUDv2 and SUN RGB-D), Image-to-Image Translation (Taskonomy dataset), and Movie Genre Classification (MM-IMDB).

  • Attention + Invertibility is Possible: The paper proves that you don't have to sacrifice the expressiveness of attention to get the mathematical "tractability" of flows.
  • Explicit Modeling Wins: By explicitly modeling the joint distribution, MANGO prevents any single modality from dominating the fusion process, leading to more robust features.
  • State-of-the-Art Performance: MANGO consistently outperformed prior methods. On the NYUDv2 benchmark, it achieved a 59.2% mIoU, surpassing GeminiFusion (57.7%) and TokenFusion (54.2%). For movie classification, it achieved a 71.7% Micro F1 score, a significant jump over the previous best of 68.2%. It also produced significantly better FID scores in image translation tasks.

Limitations and Future Work

While MANGO is a major step forward, the authors note a few areas for improvement:

  • Objective Balancing: The model is trained with a dual loss (negative log-likelihood for the flow and a task-specific loss). Finding the perfect balance between these two remains an open research question.
  • Computational Cost: Although the latent flow approach helps, Normalizing Flows are still more computationally demanding than simple concatenation.

Summary updated

Abstract

Multimodal learning has gained much success in recent years. However, current multimodal fusion methods adopt the attention mechanism of Transformers to implicitly learn the underlying correlation of multimodal features. As a result, the multimodal model cannot capture the essential features of each modality, making it difficult to comprehend complex structures and correlations of multimodal inputs. This paper introduces a novel Multimodal Attention-based Normalizing Flow (MANGO) approach to developing explicit, interpretable, and tractable multimodal fusion learning. In particular, we propose a new Invertible Cross-Attention (ICA) layer to develop the Normalizing Flow-based Model for multimodal data. To efficiently capture the complex, underlying correlations in multimodal data in our proposed invertible cross-attention layer, we propose three new cross-attention mechanisms: Modality-to-Modality Cross-Attention (MMCA), Inter-Modality Cross-Attention (IMCA), and Learnable Inter-Modality Cross-Attention (LICA). Finally, we introduce a new Multimodal Attention-based Normalizing Flow to enable the scalability of our proposed method to high-dimensional multimodal data. Our experimental results on three different multimodal learning tasks, i.e., semantic segmentation, image-to-image translation, and movie genre classification, have illustrated the state-of-the-art (SoTA) performance of the proposed approach.
BibTeX
@inproceedings{truong2025mango,
  author = {Thanh-Dat Truong and Christophe Bobda and Nitin Agarwal and Khoa Luu},
  title = {{MANGO: Multimodal Attention-based Normalizing Flow Approach to Fusion Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=cd2MWwIIHu},
}
Figure