Vision-language models increasingly feed a language model with features from several image encoders at once. One encoder may be good at semantics (CLIP), another at spatial structure (DINOv2), a third at text in images (ConvNeXt). These features must be squeezed into a fixed budget of "visual tokens," the vectors the language model reads alongside words. Today that squeezing is done in one of three ways: concatenation, which bloats and conflicts; pooling with learned queries, which may quietly ignore whole encoders or image regions; or post-hoc pruning of encoders, which discards their specialties. We propose instead to compute a Wasserstein barycenter, a geometric average of point clouds that moves mass rather than blending coordinates. The barycenter is taken over the token sets of all encoders, and its support points serve as the visual tokens. Its defining property is a mass-balance constraint: every token of every encoder must be transported somewhere into the fused set. Because of that constraint, no encoder and no image region can be silently dropped.
In two sentences: treat each encoder's patch tokens as a cloud of points in a shared space, and replace the learned-query pooling step with an optimal-transport average whose transport plans must account for all of each encoder's mass. The scientific question is whether this enforced coverage, rather than extra parameters, explains when multi-encoder fusion helps on vision-centric tasks. The proposal includes the ablation that would show it does not.
Core Claim
Two papers motivate the gap. Investigating Redundancy in Multimodal Large Language Models with Multiple Vision Encoders masks encoders in Cambrian-1 and Eagle-X5. It finds that a two-encoder subset of Cambrian-1 8B beats the full four-encoder model by 1.7%, and that masking detrimental encoders gains up to 16% on some task categories. Its diagnostic for that (the conditional utilization rate) is post-hoc and requires training many encoder combinations. Cambrian-1 fuses up to four encoders into 576 tokens with its Spatial Vision Aggregator, which uses learnable latent queries plus cross-attention under a spatial prior. Nothing in that design guarantees that each encoder or region contributes.
Our claim is that a large part of the "conflict" between encoders comes from the fusion operator, not the encoders. Softmax attention normalizes only over the tokens each query attends to. A query can therefore collapse onto one dominant encoder, which is the same shortcut behavior that Wasserstein Modality Alignment documents between modalities in multimodal transformers. A barycenter adds the missing constraint on the other side of the plan: each source token's mass must be fully assigned. We predict three things. First, barycentric fusion should degrade less than concatenation or latent-query pooling when a weak or redundant encoder is added. Second, it should help most on counting and spatial-relation questions, where coverage matters. Third, if fusion is replaced by the same operator without the source-side constraint, the advantage should disappear.
What is genuinely new relative to existing OT fusion
The obvious "barycenter for multimodal fusion" idea already exists in narrower forms. Multimodal Variational Autoencoder: A Barycentric View fuses unimodal Gaussian posteriors with the closed-form Bures-Wasserstein barycenter. OTKGE fuses static image, text and structure vectors of knowledge-graph entities by a barycenter and reports a mean reciprocal rank of 0.843 on FB-IMG. Both average one vector or one Gaussian per modality. Neither handles variable-size token sets, spatial position, a fixed token budget, or a downstream autoregressive language model.
Unsupervised Multilingual Alignment using Wasserstein Barycenter supplies the closest method. It alternates between learning per-language maps into a shared space and computing a barycenter "virtual pivot" there, and avoids the distortion of choosing one language as pivot. We reuse that alternating structure. We replace orthogonal maps with learned linear maps (encoders differ in dimension), and we make the barycenter a differentiable layer trained end to end through the language model's loss.
Video-LLaVA argues for aligning heterogeneous visual inputs before a single shared projector. Our layer is an "align before projection" step, but the alignment is a transport average rather than a jointly pretrained encoder.
Research Program
Each encoder $m \in \{1,\dots,M\}$ yields patch tokens $x_{m,i}$ at image positions $p_{m,i}$, with uniform mass $1/N_m$ over its $N_m$ tokens. A learned linear map $A_m$ sends tokens into a shared $d$-dimensional space. The fused output is a set of $K$ support points $y_k$ with fixed grid positions $q_k$, each carrying mass $1/K$. $K$ equals the visual token budget, such as 144 or 576. The layer solves
$$\min_{\{y_k\}} \sum_{m=1}^{M} w_m \, \mathrm{OT}_\varepsilon\Big(\tfrac{1}{N_m}\textstyle\sum_i \delta_{(A_m x_{m,i},\,p_{m,i})},\ \tfrac{1}{K}\sum_k \delta_{(y_k,\,q_k)}\Big),$$
The terms in this objective are defined as follows:
- $\mathrm{OT}_\varepsilon$ is entropically regularized optimal transport.
- Its ground cost is $\|A_m x - y\|^2 + \lambda \|p - q\|^2$, so that mass prefers to move between nearby image locations.
- $w_m$ are per-image encoder weights produced by a small gating network from pooled features.
Following the free-support barycenter iterations described in Computational Optimal Transport, we alternate two steps. First, a few Sinkhorn iterations produce plans $P_m$ whose row sums are $1/N_m$ and whose column sums are $1/K$. Second, each support point is updated by the closed-form step $y_k \leftarrow K\sum_m w_m \sum_i P_m[i,k]\, A_m x_{m,i}$. We unroll a small fixed number of these steps (around 3 outer and 10 Sinkhorn) and backpropagate through them. The resulting $K$ tokens pass through a standard two-layer MLP projector into the language model.
The design has two deliberate dials. The first is the source-side marginal: making it unbalanced, with a penalty $\tau$ on unassigned mass, interpolates continuously between full coverage and attention-like selection. The second is the gate $w_m$: it gives a trained, per-image analogue of encoder masking, which the redundancy paper could only do post hoc.
The layer adds only $\sum_m d \cdot d_m$ map parameters plus a tiny gate. Cambrian's aggregator, by contrast, carries learned queries and multiple cross-attention blocks. Sinkhorn on a few thousand by a few hundred matrices is small next to a language model forward pass, but we will measure throughput rather than assert it. The overhead of iterative OT is a known concern (OTKGE). If it bites, a sliced variant such as Fixed Support Tree-Sliced Wasserstein Barycenter is the fallback.
The setting is a small open language model (a ~1–4B Qwen3-family model, the scale at which Deep Pre-Alignment for VLMs reports results) with CLIP, DINOv2 and ConvNeXt encoders. We use the standard two-stage recipe: train the connector with encoders and language model frozen, then instruction-tune. All arms share one fixed public data mixture. We also report a stage-one-only, language-model-frozen variant, which isolates the connector and sidesteps the forgetting documented by Deep Pre-Alignment.
Baselines
The naive reference point is channel concatenation followed by an MLP. Each encoder's feature map is resampled to a common grid of $K$ tokens and the channels are stacked, as in the concatenation-style fusion studied for Eagle-X5 and Cambrian-1 in Investigating Redundancy. It is the most dangerous simple competitor because the encoders' grids are already spatially registered. If position alone solves correspondence, transport adds nothing. We also run the best single encoder with the same token budget, since the redundancy paper shows subsets can beat full stacks. The comparison counts as won only if the barycenter beats both at equal $K$.
The strongest existing method is Cambrian-1's Spatial Vision Aggregator (Cambrian-1). We reimplement it at the same token budget, with the same encoders, data and number of training steps. Its parameter count is matched by widening our MLP projector. Each arm gets the same small learning-rate sweep. For the reference number, the file reports for Cambrian-1 8B that a two-encoder subset surpasses the four-encoder model by 1.7% on the 15-benchmark suite of Investigating Redundancy. No reported absolute CV-Bench score was found on Lacuna for the aggregator. Because our models will be smaller and trained on less data, published absolutes are not directly comparable, and we compare against our own reimplementation. Beating it means a higher CV-Bench average with no loss on the general suite. Because the 1.7% figure is relative, we also apply the redundancy paper's own test: we add a fourth, weak encoder and check whether accuracy stays flat or rises.
A token-reduction reference is ATP-LLaVA. It retains 98.1% of LLaVA-1.5 performance while cutting visual tokens from 576 to 144. That is the bar for our $K=144$ setting on its benchmarks: GQA, VQA v2, MMBench, MME, POPE, SEED-Bench and ScienceQA.
The key ablation keeps everything identical but removes the source-side marginal. The plan is normalized only over source tokens for each support point, which makes it a softmax-attention pooling with a distance kernel. A second ablation sweeps the unbalanced penalty $\tau$. A third replaces the transport cost with k-means assignment of pooled tokens. Together these isolate whether mass balance is the active ingredient.
Why not just…?
Why not just concatenate channels on a shared grid? This is the real threat, and it may win when encoders share resolution and patch layout. The proposal's own position term $\lambda\|p-q\|^2$ makes transport close to the identity in that case. We therefore test two regimes. In the first, encoders share a grid. In the second, they run at different native resolutions and token counts (e.g., a high-resolution ConvNeXt against a 336-pixel CLIP), where resampling for concatenation blurs one encoder. If concatenation matches the barycenter in both regimes, the transport machinery is unnecessary and we will say so.
Why not just use learned-query cross-attention, which is more flexible? Flexibility is precisely what lets a query collapse onto one encoder. Wasserstein Modality Alignment shows this kind of over-reliance on a dominant modality in multimodal transformers. The redundancy paper's finding that adding encoders can hurt Cambrian-1 is consistent with the fusion step failing to use them. The decisive evidence will be the marginal-removal ablation plus a per-encoder mass audit, which reports how much of each encoder's attention or transport mass reaches the output tokens. If attention already distributes mass evenly and still matches us, the coverage hypothesis is wrong.
Why not just prune encoders, or prune tokens? Encoder pruning via utilization rates requires training many encoder subsets and then fixes one choice for all inputs. The same paper reports that specialized encoders exceed 90% utilization on OCR and charts but matter little elsewhere, which argues for per-image weighting rather than removal. Our gate $w_m$ tests exactly that, and we will check whether its learned weights correlate with the published per-task utilization. Token pruning à la ATP-LLaVA is strong at 144 tokens. However, that paper reports early layers are sensitive to pruning, and hard dropping cannot merge information from several encoders into one token. We compare at matched $K$, and concede the point if pruning ties.
Why not just a single-encoder barycenter, i.e., k-means token merging? With one encoder and uniform weights, a free-support barycenter is close to clustering. That is why the contribution is the multi-encoder coupling, not compression per se. The k-means ablation measures this directly.
Proposed Experiments and Decisive Checks
The primary benchmark is CV-Bench from Cambrian-1, broken down by counting, 2D relations, depth order and 3D distance. These categories are where coverage should matter. CV-Bench is also built to resist language-only solving, a known weakness of other benchmarks according to the same paper. We additionally run every benchmark with the image removed and report the gap, so that gains attributable to linguistic priors are visible. The secondary benchmarks are MMMU (MMMU), POPE, GQA, and as much of the 15-benchmark redundancy suite as compute allows. For runs that unfreeze the language model, we report MATH-500 as a forgetting diagnostic. The reference points there are Deep Pre-Alignment's report of a base score of 84.8 dropping to 36.4 for a standard 4B baseline and 54.2 with their method. We also report inference throughput and peak memory against the concatenation and aggregator arms.
Three results would decide the proposal. First, the barycenter beats both concatenation and the reimplemented aggregator on CV-Bench at equal tokens and parameters, with the gain concentrated in counting and spatial categories. Second, removing the source-side marginal erases most of that gain, while increasing $\tau$ toward unbalanced transport moves results smoothly toward the attention arm. Third, adding a weak fourth encoder lowers accuracy for concatenation and latent-query pooling, as the redundancy paper found, but leaves the barycenter flat or improved, with the gate down-weighting that encoder.
If the first holds but the second does not, the benefit comes from the shared-space maps rather than transport. That would still be useful, but it is a different paper. If concatenation ties in the mismatched-resolution regime, the idea should be abandoned.
References
- Investigating Redundancy in Multimodal Large Language Models with Multiple Vision Encoders
- Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
- Wasserstein Modality Alignment Makes Your Multimodal Transformer More Robust
- Multimodal Variational Autoencoder: A Barycentric View
- OTKGE: Multi-modal Knowledge Graph Embeddings via Optimal Transport
- Unsupervised Multilingual Alignment using Wasserstein Barycenter
- Video-LLaVA: Learning United Visual Representation by Alignment Before Projection
- Computational Optimal Transport
- Fixed Support Tree-Sliced Wasserstein Barycenter
- Deep Pre-Alignment for VLMs
- ATP-LLaVA: Adaptive Token Pruning for Large Vision Language Models
- MMMU: A Massive Multi-Discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI