The Combinatorial Scaling Bottleneck in Multimodal Learning

15 concepts14 papers1 proposal

When we train a model on two modalities—such as pairing images with text captions—the architecture is straightforward. We project both inputs into a shared space or let them interact via cross-attention. However, real-world applications rarely stop at two modalities. A comprehensive AI assistant or an autonomous vehicle might simultaneously ingest video, audio, text, depth maps, thermal data, and localized sensor streams.

As the number of input modalities ($M$) grows, multimodal architectures hit a severe scaling wall. Instead of scaling linearly, the computational and memory costs explode combinatorially. This bottleneck manifests in two distinct ways: the quadratic cost of modeling pairwise interactions ($O(M^2)$) and the exponential cost of managing missing data combinations ($O(2^M)$).


The Quadratic Cost of Pairwise Interactions

The most intuitive way to fuse multiple data streams is to let every modality talk to every other modality. For instance, in video analysis, you might want to explicitly model how the visual frames align with the audio track, how the audio correlates with text transcripts, and how the text maps back to the visual motion.

If you build explicit channels or encoders for every pair of modalities, the number of required interaction pathways scales quadratically at $O(M^2)$.

DENSE PAIRWISE INTERACTION (O(M²))
   [Modality A] <---> [Modality B]
        ^                  ^
        |                  |
        v                  v
   [Modality C] <---> [Modality D]

Chen et al. demonstrated this challenge with Pairwise Modality Interaction (PMI), noting that exhaustive bilinear modeling across visual, motion, and audio streams creates a steep computational bottleneck. Similarly, Qayyum observed that architectures allocating specific encoders for every pair of data variables suffer from quadratic complexity growth, limiting their utility in high-dimensional datasets like multi-altitude climate arrays.

Even in self-supervised learning, aligning multiple modalities through pairwise contrastive losses or factorizing representations into shared and private latent spaces scales quadratically with the number of sensors, as detailed by Liu et al..

The Intuitive Fix: Graph Sparsification and Message Passing

To bypass this $O(M^2)$ bottleneck, researchers treat multimodal fusion as a sparse graph problem rather than a dense matrix problem. Instead of computing exhaustive all-to-all attention, we can treat modalities as nodes in a factor graph or an attention graph.

SPARSE GRAPH FUSION (O(M))
   [Modality A]       [Modality B]
         \                 /
          v               v
          [Message Passing]
          ^               ^
         /                 \
   [Modality C]       [Modality D]
  • Factor Graph Attention: Schwartz et al. introduced a framework that models diverse inputs as nodes in a factor graph, using message-passing principles to dynamically learn which modalities actually need to attend to one another.
  • Dynamic Edge Pruning: Yang et al. developed the Modal-Temporal Attention Graph (MTAG), which explicitly avoids the quadratic explosion of edge types by dynamically pruning unimportant cross-modal connections, achieving state-of-the-art performance with a fraction of the parameters.
  • Adaptive Sparse Attention: For real-time applications like wearable sensor arrays, Mohapatra et al. designed MAESTRO, which uses adaptive sparse attention to eliminate irrelevant pairwise interactions, keeping the computational footprint manageable even as the sensor count scales.

The Exponential Burden of Missing Modalities

The second, more severe bottleneck occurs when models must handle missing data at inference time. In real-world deployments, a camera might get obscured, an audio feed might drop out, or a sensor might fail.

If a system relies on "missing-aware" configurations—where the model changes its behavior or uses a specific prompt based on exactly which modalities are present—it faces an exponential complexity explosion of $O(2^M)$. For 3 modalities, there are $2^3 = 8$ possible combinations of available data. For 10 modalities, there are $1,024$.

M = 3 Modalities (2³ = 8 states)      M = 4 Modalities (2⁴ = 16 states)
   [A, B, C]                             [A, B, C, D]
   [A, B], [B, C], [A, C]                [A, B, C], [B, C, D], ...
   [A], [B], [C]                         [A, B], [B, C], ...
   [None]                                ...

Early on, Suzuki et al. identified this architectural difficulty when trying to ensure that every possible subset of inputs could accurately map to a shared generative latent space. When modern architectures use parameter-efficient prompt learning to adapt to missing data, creating unique configurations or prompts for every possible subset of available modalities quickly becomes impossible to scale (Jin et al.; Bolelli et al.).

The Intuitive Fix: Additive Decomposition

To reduce $O(2^M)$ exponential complexity to $O(M)$ linear complexity, we must decouple the model's parameters from the combinations of modalities and attach them strictly to individual modalities.

Jang et al. proposed a elegant solution to this problem. Instead of training a unique prompt for every combination of missing data, they introduced modality-specific prompts that are simply summed together.

COMBINATORIAL PROMPTING (O(2ᴹ))       ADDITIVE PROMPTING (O(M))
   State: [A + B] -> Prompt_AB           State: [A + B] -> Prompt_A + Prompt_B
   State: [B + C] -> Prompt_BC           State: [B + C] -> Prompt_B + Prompt_C

If Modality A and Modality B are present, the model sums Prompt_A + Prompt_B. If Modality B drops out, it simply uses Prompt_A. This mathematical formulation completely bypasses the combinatorial explosion: the system only needs to maintain $M$ distinct prompt structures, yet it can seamlessly handle all $2^M$ possible missing-data states during inference.


The Challenge of Higher-Order Interactions and Interpretability

True multimodal reasoning isn't just additive; it often relies on non-additive synergies where the combination of multiple signals changes the meaning entirely (e.g., a sarcastic tone of voice reversing the sentiment of a text phrase).

However, Wörtwein et al. discovered that because modeling true non-additive bimodal and trimodal interactions scales exponentially, many standard multimodal models quietly default to simple additive fusion. They "cheat" by optimizing for individual modalities rather than learning complex cross-modal relationships.

This creates a massive hurdle for interpretability. If we want to audit a model to see if it is actually performing cross-modal reasoning, we have to isolate the exact contribution of these interactions.

Unfortunately, post-hoc explanation tools hit the exact same combinatorial wall:

  • Disentangled Explanations: Frameworks like DIME (Liang et al.) attempt to separate a model's prediction into unimodal contributions and cross-modal interaction terms, but computing these distinct spaces becomes computationally prohibitive beyond two or three inputs.
  • Shapley Interactions: Metrics like InterSHAP (Wenderoth et al.), which leverage the Shapley Interaction Index to quantify cross-modal dependencies, inherently face exponential computational complexity as the number of modalities increases.

Structuring Interactions via Specialized Experts

To capture these complex interactions without triggering a parameter explosion, modern architectures are turning to specialized Mixture-of-Experts (MoE) designs.

Instead of letting a single dense network attempt to learn all combinations, the $I^2MoE$ framework breaks down multimodal fusion into distinct, weakly supervised "interaction experts." Individual experts are forced to specialize in isolating unique, redundant, or synergistic information. By routing inputs only to relevant interaction experts, the architecture provides clear, sample-level explanations of cross-modal dynamics while keeping the underlying parameter scaling under control.

Key Foundational Work

Joint Multimodal Learning with Deep Generative Models (2017) — contributed 1 concept elements to this direction.
Factor Graph Attention. (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_quadratic_modality_scaling_e4dd630f,
  title = {quadratic modality scaling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/quadratic-modality-scaling/rdr_e4dd630f83734dd388af9efaadc66d53},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_quadratic_modality_scaling_e4dd630f_v1,
  title = {quadratic modality scaling},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/quadratic-modality-scaling/rdr_e4dd630f83734dd388af9efaadc66d53/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake