When building models that look, listen, and read simultaneously, the central architectural challenge is fusion: how do we combine distinct streams of information into a single representation?
The simplest approach is feature concatenation, where vectors from different modalities are glued end-to-end. However, as noted by Bezirganyan et al. in their work on M2-Mixer: A Multimodal Mixer with Multi-head Loss for Classification from Multimodal Data, simple concatenation lacks the expressive capacity to model nuanced, non-linear interactions between disparate data types. It forces subsequent deep layers to implicitly learn how a specific word in a caption relates to a specific patch of an image.
To capture these cross-modal dynamics explicitly, researchers turned to the outer product. By computing the outer product of modality vectors, a model creates a multi-dimensional tensor where every feature of every modality interacts multiplicatively with every other feature. Amir Zadeh et al. leveraged this to create the Tensor Fusion Network for Multimodal Sentiment Analysis, demonstrating that explicit tensor fusion can capture unimodal, bimodal, and trimodal dynamics simultaneously.
But this expressivity comes at a severe cost: a geometric explosion of parameters that quickly becomes computationally intractable.
The Geometry of Parameter Explosion
To understand why outer-product fusion scales so poorly, consider two modality representations: a textual feature vector $x \in \mathbb{R}^{d_1}$ and a visual feature vector $y \in \mathbb{R}^{d_2}$. Their outer product $Z = x \otimes y$ is a matrix of size $d_1 \times d_2$. If we wish to project this fused representation to a hidden layer of size $h$, we must multiply it by a weight tensor $W$ of size $h \times d_1 \times d_2$.
As Mai et al. point out in Divide, Conquer and Combine: Hierarchical Feature Fusion Network with Local and Global Perspectives for Multimodal Affective Computing, this creates a severe curse of dimensionality. If $d_1 = d_2 = 512$ and $h = 512$, the weight tensor requires over 134 million parameters for just a single fusion layer.
The bottleneck becomes catastrophic when scaling to three or more modalities (such as text, audio, and video). For $M$ modalities, the tensor size scales exponentially as $O(d^M)$. Qiu et al. observe in Deep Biological Pathway Informed Pathology-Genomic Multimodal Survival Prediction that modeling these high-order interactions creates an unmanageable parameter explosion that destabilizes training and induces severe overfitting, a challenge echoed in medical domains by the creators of MedFuseNet. Liang et al. characterize this in Learning Representations from Imperfect Time Series Data via Tensor Rank Regularization as a memory bottleneck that restricts full tensor fusion from being deployed in large-scale architectures.
Bypassing the Explosion with Low-Rank Factorization
The key insight that makes high-order multimodal fusion practical is that we do not need to explicitly compute the massive outer product tensor to compute its projection.
Instead, we can exploit the fact that the high-dimensional weight tensor $W$ contains significant redundancy and can be approximated using low-rank tensor decomposition. This breakthrough was introduced by Ying Shen et al. in Efficient Low-rank Multimodal Fusion With Modality-Specific Factors, a framework known as Low-rank Multimodal Fusion (LMF).
Mathematically, if the weight tensor $W$ can be decomposed into a sum of $R$ rank-1 tensors (where $R$ is the chosen rank), we can rewrite the tensor multiplication. For two modalities, instead of computing $y = W \cdot (x \otimes y)$, the decomposition allows us to distribute the multiplication:
$$y = \sum_{r=1}^R w_h^{(r)} \cdot \left( \langle w_x^{(r)}, x \rangle \cdot \langle w_y^{(r)}, y \rangle \right)$$
This algebraic rearrangement completely changes the computational complexity. Instead of computing an expensive outer product of size $d_1 \times d_2$ and multiplying it by a giant weight matrix, the model projects each modality vector independently into a lower-dimensional space using modality-specific factors ($w_x$ and $w_y$), and then performs a simple element-wise product.
By bypassing the explicit construction of the high-dimensional tensor, the computational and memory complexity drops from exponential ($O(d^M)$) to linear with respect to the number of modalities ($O(M \cdot d \cdot R)$). This allows the model to capture the exact same high-order cross-modal interactions with a fraction of the parameters, accelerating inference and mitigating overfitting.
A Taxonomy of Tensor Decompositions
Depending on how the interactions are structured, different tensor decomposition strategies offer distinct trade-offs between expressiveness and efficiency.
Block-Term Decomposition
Rather than decomposing a tensor into purely rank-1 vectors, Hedi Ben-younes et al. proposed BLOCK: Bilinear Superdiagonal Fusion for Visual Question Answering and Visual Relationship Detection. BLOCK uses block-term tensor decomposition to split the interaction tensor into a sum of low-rank blocks. This allows the model to vary the complexity of the interaction space, striking a "Goldilocks" balance that maintains high expressiveness while keeping the parameter count manageable.
Tucker and Tensor Ring Decompositions
For tasks requiring massive multi-way interactions, standard low-rank approximations can still struggle. Zhu et al. utilized Tucker decomposition in Multi-Modal Emotion Recognition Using Tensor Decomposition Fusion and Self-Supervised Multi-Tasking to factorize high-dimensional emotion tensors into a core tensor multiplied by modality matrices.
To scale even further, the authors of MMT: Multi-way Multi-modal Transformer for Multimodal Learning integrated Tensor Ring decomposition directly into a transformer architecture. This allows text, audio, and video tokens to interact simultaneously in a circular, matrix-product state representation, capturing complex many-to-many correlations without requiring stacked, resource-intensive cross-modal attention layers.
Temporal and Trilinear Extensions
Tensor bottlenecks also occur across time and when incorporating auxiliary data. Tao Jin et al. developed Dual Low-Rank Multimodal Fusion (Dual-LMF), which applies low-rank factorization to both the feature dimensions and the temporal dimensions simultaneously, capturing fine-grained cross-modal trajectories in real time. When a third explicit modality is introduced—such as integrating the answer space into visual question answering—Tuong Do et al. demonstrated in Compact Trilinear Interaction for Visual Question Answering that PARALIND tensor decomposition can compress trilinear interactions that would otherwise be completely impossible to train.
Alternative Paths to Scalability
While tensor factorization solves the mathematical bottleneck of outer products, researchers have explored complementary structural strategies to handle high-dimensional multimodal data:
- Sparse Pruning: Alexej Gossmann et al. developed a sparse machine learning model in Multimodal Sparse Classifier for Adolescent Brain Age Prediction that leverages numerical residual errors to prune redundant nodes entirely, preventing overfitting in high-dimensional neuroimaging data.
- Structural Disentanglement: Liu et al. introduced the Part-Whole Relational Fusion Towards Multi-Modal Scene Understanding framework, which uses Capsule Networks as a routing mechanism to separate shared cross-modal semantics from modality-specific details, bypassing the need for exhaustive interaction tensors across dense sensor suites like LiDAR, thermal, and RGB.
- Sequential Fusion: Rather than fusing all modalities at a single bottleneck, some architectures integrate data views sequentially. However, as noted by Wang et al. in SeqMvRL: A Sequential Fusion Framework for Multi-view Representation Learning, scaling sequential methods introduces a different kind of complexity: a combinatorial explosion in the search space for finding the optimal integration sequence.
Ultimately, whether through structural routing, sparse pruning, or low-rank factorization, the evolution of multimodal fusion reflects a singular shift: moving away from brute-force concatenation and explicit outer products toward mathematically compressed, interaction-aware architectures that scale efficiently.
Key Foundational Work
Multimodal neuroimaging computing: the workflows, methods, and platforms (2015) — contributed 1 concept elements to this direction.
Audio-Visual Classification and Fusion of Spontaneous Affective Data in Likelihood Space (2010) — contributed 1 concept elements to this direction.