When designing neural networks, human engineers lean heavily on symmetry and uniformity. Standard vision backbones or text transformers stack identical layers sequentially. However, when we try to fuse fundamentally different data types—such as high-frequency audio spectrograms, discrete text tokens, continuous electronic health records, or sparse LiDAR point clouds—this hand-crafted symmetry breaks down.
Historically, researchers relied on blunt heuristics like early fusion (concatenating raw inputs) or late fusion (averaging model outputs). But the most effective interactions happen in the middle, where features are partially abstracted. The problem is that the space of possible "middle fusion" architectures is combinatorially explosive. Which layer of a video network should talk to which layer of an audio network? Should they combine via addition, concatenation, or cross-attention?
As shown by Juan-Manuel Perez-Rua et al. in MFAS: Multimodal Fusion Architecture Search, the optimal architectures discovered by automated search are frequently highly asymmetrical and non-intuitive, utilizing irregular skip-connections that human designers routinely overlook. Automated multimodal fusion treats this architectural design not as a manual engineering task, but as an optimization problem solved via Neural Architecture Search (NAS).
Structuring the Fusion Search Space
To make this search tractable, the problem is broken down into a search space defined by two core questions: where to fuse (connectivity) and how to fuse (operations).
Instead of hand-engineering a fixed "fusion bridge," frameworks like the one proposed by Di Hu in SepFusion: Finding Optimal Fusion Structures for Visual Sound Separation define a set of primitive mathematical operations—such as spatial mapping, attention mechanisms, or bilinear pooling. The search algorithm then explores combinations of these primitives to find the ideal mathematical alignment between disparate feature vectors.
Crucially, the timing of these operations matters. In MUFASA: Multimodal Fusion Architecture Search for Electronic Health Records, Zhen Xu et al. demonstrated the power of a unified search space that simultaneously optimizes independent processing branches alongside a centralized fusion mechanism. Rather than forcing a fixed integration point, the network dynamically chooses between hybrid integration points across different network depths, allowing low-level structural features and high-level semantic features to merge exactly when their representations are most compatible.
Search Strategies and Efficiency
How do we navigate this massive search space without burning infinite compute? The field has converged on two primary algorithmic paradigms:
Differentiable Search (Gradient-Based)
By relaxing the discrete choice of fusion operations into a continuous, weighted combination, the architecture can be optimized using standard backpropagation. For instance, Peng Sun et al. utilized a differentiable search in Deep RGB-D Saliency Detection With Depth-Sensitive Attention and Automatic Multi-Modal Fusion to select from a predefined set of multi-scale fusion and spatial restoration operations. Similarly, Mengmeng Ma applied a differentiable fusion depth search in Are Multimodal Transformers Robust to Missing Modality? to let the model learn the exact layer where modality interaction should occur, enhancing robustness when one stream goes missing.
Discrete and Evolutionary Search
When gradients cannot easily flow across highly heterogeneous streams, evolutionary algorithms offer a robust alternative. In AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures, Michael S Ryoo et al. used an evolutionary algorithm guided by learnable connection weights to discover intricate multi-stream connectivity patterns across varying temporal resolutions.
To mitigate the high computational cost of evaluating populations of networks, Zhihua Cui et al. introduced A Fast Neural Architecture Search Method for Multi-Modal Classification via Knowledge Sharing. Their approach allows individuals in an evolutionary population to inherit pre-trained weights from a shared "supernet," drastically reducing training time while still identifying highly specialized fusion architectures.
The Modality-Agnostic Advantage
The true power of automated multimodal fusion lies in its generality. Because the search space is built on primitive mathematical operations rather than domain-specific assumptions, the same underlying framework can adapt to entirely different sensor combinations.
As demonstrated in the work on SepFusion: Finding Optimal Fusion Structures for Visual Sound Separation, a modality-agnostic fusion framework can optimize interactions between audio and raw video frames, human skeletons, or motion spectra without requiring manual redesign. By letting data-driven optimization dictate the flow of cross-modal information, automated fusion bypasses human cognitive biases, consistently unlocking superior predictive accuracy across diverse multimodal tasks.
Key Foundational Work
FusionNet: Incorporating Shape and Texture for Abnormality Detection in 3D Abdominal CT Scans (2019) — contributed 2 concept elements to this direction.
MFAS: Multimodal Fusion Architecture Search. (2019) — contributed 2 concept elements to this direction.
Bayesian optimization on graph-structured search spaces: Optimizing deep multimodal fusion architectures (2018) — contributed 1 concept elements to this direction.