When a model combines two sensors, say RGB video and depth, or images and text, one of the most consequential design choices is where the two streams meet. The streams can merge at the input, at the final predictions, or somewhere in between, in what is called intermediate or mid-level fusion. Architecture search methods automate this choice, but they are expensive: MFAS took 603.6 GPU-hours on NTU RGB+D, and BM-NAS still took 38.6 GPU-hours. This proposal starts from a structural fact those methods share. Both freeze the single-sensor networks, often called backbones, and search only the small fusion network on top. Once the backbones are frozen, every candidate architecture reads from the same fixed set of intermediate features. One forward pass over a data subset can cache those features. Candidate fusion wirings could then be scored in closed form on the cache, with no candidate ever trained.
The core claim is that under frozen backbones, a closed-form, data-aware score computed on cached features ranks fusion architectures well enough to replace most of the search. The score is kernel ridge regression on random features, and it measures how much a set of layers from both streams adds over each stream alone. Search cost should drop from GPU-hours to minutes while final accuracy stays within seed noise of searched fusion. We also build the missing ground truth: a small tabular benchmark of fully trained fusion architectures. Without it, no fusion proxy's ranking quality can be measured.
Why this gap, and why this form of the idea
Two literatures meet here without having talked to each other. Work on cheap architecture search has produced scores that need no training. EZNAS evolves scoring formulas over statistics of untrained networks and reaches a Kendall rank correlation of 0.65 on NAS-Bench-201 (EZNAS). RoBoT combines several such scores through Bayesian optimization (RoBoT). Both papers, however, report that these scores struggle to separate the top 10% of architectures. EZNAS also notes that its scores are blind to cross-branch wiring, which is exactly what a fusion search space consists of. These papers constrain the design: a data-free score computed at random initialization is the wrong tool for choosing among connections between pretrained features.
Fusion search methods make up the second literature. MFAS scores its candidates with a recurrent surrogate plus weight sharing (MFAS). BM-NAS uses a differentiable two-level supernet (BM-NAS). MFH-NAS chains differentiable and evolutionary stages and reports its cost in GPU-days (MFH-NAS). MUFASA evolves both the backbones and the fusion network and trains every candidate (MUFASA). These papers motivate the proposal, because all of them pay training cost to rank candidates. None of them uses the fact that frozen backbones make the candidates' inputs fixed and cacheable.
The obvious idea at this intersection is to port zero-cost proxies to fusion graphs, as Gap 1 of the file suggests. Our proposal differs deliberately. The score here uses real data and real pretrained features, not random-initialization gradients. It is conditional: it scores what layer $j$ of modality B adds given the layers already taken from A. It is also fitted in closed form, so the only "training" is one linear solve per candidate on a cache of a few thousand examples.
Core Claim and Mechanism
For a candidate fusion architecture $a$, let $S_a$ be the set of backbone layers it reads from each modality. Let $\phi(S_a)$ be a random-feature map applied to the pooled cached activations of those layers. The random features give the score a fixed nonlinearity, a stand-in for the nonlinear fusion layers. We define the score as
$$s(a) = R^2_{\text{val}}\big(\phi(S_a)\big) - \max\big(R^2_{\text{val}}(\phi(S_a^{A})),\, R^2_{\text{val}}(\phi(S_a^{B}))\big),$$
The terms are defined as follows:
- $R^2_{\text{val}}$ is the held-out fit, or a classification analogue, of a ridge regressor onto the labels.
- $S_a^{A}$ and $S_a^{B}$ are the subsets of layers drawn from each modality.
The score therefore rewards complementarity rather than raw capacity. Scores are cheap in part because ridge solves for nested layer sets can reuse matrix factorizations. That reuse makes it feasible to score every candidate in an MFAS-style space instead of sampling it.
The score is expected to capture where fusion happens but not how. Operators such as attention or gated units are not modeled by a ridge fit. We therefore split the search. The score chooses the connection pattern. A short operator search then runs only on the top few connection patterns, reusing the BM-NAS cell primitives and training for a small fixed budget. Li et al. argue that a mismatch between the two modalities' latent distributions hurts generalization. As an optional second term we add their Wasserstein mismatch between the paired layers, computed on the same cache, and test whether it improves ranking.
Research Program
The first deliverable is the benchmark. We use the MFAS search space: which layer from each modality feeds each fusion layer, and which activation follows. On AV-MNIST and MM-IMDB we fully train about 500 uniformly sampled architectures each, with three seeds. On NTU RGB+D we train about 150. We release their accuracies, which makes this a small fusion analogue of NAS-Bench-201. Every ranking claim in the proposal is measured against this ground truth.
The second deliverable is the scorer and a study of when it fails. We measure Kendall rank correlation over all sampled architectures and over the top 10% separately, because the top 10% is where zero-cost proxies are known to break. We vary cache size, the number of random features, and the backbone. The third deliverable is end-to-end search: select by score, run the short operator search on the top-$k$ connection patterns, fully train the winner, and report accuracy against wall-clock search cost. A stretch goal targets the question MUFASA raises. Because caching is per backbone, we can try a handful of backbone depths per modality and ask whether cheap scoring makes a coarse joint search over backbones and fusion affordable. Rethinking Performance Estimation in Neural Architecture Search found that training epochs dominate ranking fidelity, so this stretch goal is also where the method is most likely to fail. We treat it as an open question.
Baselines
The naive reference points are late fusion, meaning score averaging of the two single-sensor models, and single-layer halfway fusion at the backbone midpoint with a 1×1 projection. The latter follows the halfway design that beat early and late fusion on KAIST (Liu et al.). Both use the same frozen backbones and the same fusion-head training recipe as every other method. The method has to at least match halfway fusion, or search of any kind is unjustified.
The strongest existing search methods are MFAS and BM-NAS. The reported numbers are 90.04% accuracy on NTU RGB+D for MFAS at 603.6 GPU-hours, and 90.48% for BM-NAS at 38.6 GPU-hours (BM-NAS). MFAS also reports 88.38% on AV-MNIST and a weighted F1 of 0.6250 on MM-IMDB (MFAS). We rerun BM-NAS with its public setup on our backbones and report our reproduction next to the cited figure. Beating it means matching its accuracy within seed standard deviation at under one-tenth of the search wall-clock, measured on the same GPU type.
The generic zero-cost proxies are synflow, NASWOT, snip and grad_norm applied to the fusion head, plus the RoBoT ensemble of them (RoBoT). We give RoBoT the same number of queries. The comparison is Kendall correlation on our benchmark.
Low-fidelity training is random search in which each candidate trains for one or two epochs on cached features, in the spirit of budgeted estimation. This is matched to the proxy on wall-clock time, not candidate count.
Uniform all-layer fusion is cross-stitch-style exchange at every block, following C3D-Stitch, and is compared on accuracy per fusion parameter.
Ablations isolate the key ingredient. The first removes the complementarity subtraction, leaving a raw joint fit. The second replaces random features with a linear map. The third adds or removes the coherence term. The fourth replaces the data-aware score with a data-free proxy on the same cache. That last swap tests the central bet directly: if it performs as well, the data are not what make the score work.
Why not just…?
Why not just fuse halfway and skip the search? Halfway fusion is a strong default, and on some datasets it may be close to optimal. But fixed-stage fusion trailed searched cross-stage fusion on M3FD, 85.3% versus 88.2% [email protected] (MFH-NAS), and MFAS found its best wirings to be asymmetric. Our benchmark will show, per dataset, where halfway fusion ranks among all sampled architectures. If it sits in the top few percent everywhere, we will report that search is unnecessary in this regime.
Why not just train each candidate for one epoch? This is the real threat, because short training uses the true operators and the true loss. It is also exactly what the low-fidelity baseline is, and we match it to the proxy on wall-clock time. Our expectation is that with cached features, one ridge solve costs far less than an epoch, so the proxy can cover the whole space while short training covers only a sample. If short training on the cache ranks better at equal time, the correct conclusion is that caching is the real contribution and the score is not. We would say so.
Why not just use existing zero-cost proxies or RoBoT? They are data-free, computed at initialization, and their authors report weak separation among top architectures and blindness to cross-branch wiring (EZNAS). The fourth ablation tests this on identical inputs.
Why not just put fusion units at every layer? C3D-Stitch and BAT (BAT) do well this way, but Adaptable Adapters found that per-layer selection matched full-data results while halving adapter count, and it improved the low-data score from 44.87 to 51.28 (Adaptable Adapters). We compare accuracy at matched fusion-parameter budgets.
Decisive Checks
Three results would settle the proposal:
- Top-10% ranking. Does the cached complementarity score reach a top-10% Kendall correlation clearly above RoBoT and above time-matched short training on at least two of the three benchmarks?
- End-to-end cost. Does search driven by the score match BM-NAS on NTU RGB+D within seed noise at under 4 GPU-hours including caching?
- The subtraction. Does removing the complementarity subtraction degrade ranking? If it does not, the score is only measuring capacity.
A negative result on the first check, with short training winning, would still be useful. It would redirect fusion search toward cheap training on cached features.
References
- EZNAS: Evolving Zero-Cost Proxies For Neural Architecture Scoring
- Robustifying and Boosting Training-Free Neural Architecture Search
- MFAS: Multimodal Fusion Architecture Search.
- BM-NAS: Bilevel Multimodal Neural Architecture Search
- MFH-NAS:A Hybrid Neural Architecture Search Framework for Multimodal Fusion Object Detection
- MUFASA: Multimodal Fusion Architecture Search for Electronic Health Records
- Supporting Multimodal Intermediate Fusion with Informatic Constraint and Distribution Coherence
- Rethinking Performance Estimation in Neural Architecture Search
- Multispectral Deep Neural Networks for Pedestrian Detection
- Analysis of Deep Fusion Strategies for Multi-Modal Gesture Recognition
- Bi-directional Adapter for Multimodal Tracking
- Adaptable Adapters