Models such as ImageBind and LanguageBind build one embedding space for many kinds of data: audio, depth, infrared, video, and text. Each non-pivot modality is trained to match a "pivot" encoder (images or text) with a contrastive loss, which rewards the model for picking out the true partner of each example from the other items in a batch. The per-modality losses are then simply added together. This works because the pivot is frozen: each modality trains its own encoder, and the adaptive optimizer rescales each encoder's gradients, so how the losses are weighted barely matters. Once the pivot is shared and trained, whether through full fine-tuning, low-rank adapters, or one transformer that serves every modality, the losses compete for the same parameters. Adding them up then hands control of the pivot to whichever pairing produces the largest residual gradient. In practice that is often the noisiest pairing, such as audio or motion sensors against video, because its loss never saturates.
We propose to treat this as a multi-objective optimization problem and to normalize each pairing's loss by the fraction of its achievable gap still open, rather than by its raw value. The idea rests on a fact that the classical normalization literature lacks. For a contrastive or cross-entropy loss, the worst reasonable value is known exactly: chance level, $\log B$ for batch size $B$. The only anchor that must be estimated is the best achievable value for each pairing, which is limited by how noisy its pairs are. Because the worst-case anchor is fixed, the range used for normalization never shrinks. This sidesteps both the chicken-and-egg problem of estimating the worst point on the trade-off surface and the "over-stretching" failure that afflicts running min-max normalization.
Core Claim
Let $L_k(\theta)$ be the validation-tracked contrastive loss between modality $k$ and the pivot under shared parameters $\theta$. Let $L_k^*$ be the lowest loss pairing $k$ reaches when it alone is allowed to train the pivot. Define the normalized coordinate and scalarized objective
$$z_k(\theta) = \frac{L_k(\theta) - L_k^*}{\log B - L_k^*}, \qquad S_p(\theta;\omega) = \Big(\sum_k \omega_k\, z_k(\theta)^p\Big)^{1/p},$$
where $\omega_k$ are user preference weights and $p$ trades average progress ($p=1$) against worst-case progress ($p=\infty$). The pivot's own image-text loss is included as one of the $k$, so that preserving the pivot is an explicit objective rather than an afterthought.
The claim is that training a shared pivot on $S_p$ yields a better and more controllable set of trade-offs across modalities than uniform summation or existing gradient-balancing rules. "Better" means higher worst-case zero-shot accuracy across pairings at equal compute, and a wider, more evenly spread Pareto front as $\omega$ varies. The key ingredient is the estimated ideal point $L_k^*$. Without it, $p=1$ collapses to uniform weighting with a fixed per-pairing constant. The proposal succeeds only if the gap-based anchors beat both raw-loss and running-range normalization.
Where This Sits in Prior Work
The normalization side supplies the method and its known failure modes. On the effect of normalization in MOEA/D shows that normalizing by running population ranges rescues evolutionary search on badly scaled problems. On 10-objective WFG9, hypervolume rises from 0.4897 to 0.8855. The same paper shows that the observed range collapses as the population converges, over-stretching that dimension unless the smoothing constant is large. Our fixed chance-level anchor is a direct answer to that constraint. Boundary Decomposition for Nadir Objective Vector Estimation and its discrete counterpart BDNC spend a bilevel search on locating the worst point of the Pareto front. We argue that for likelihood-style losses this point is given in closed form, so the estimation budget should go to the ideal point instead. By the classical definition, the ideal point is found by solving one single-objective problem per objective, which is exactly our specialist runs. COPA normalizes heterogeneous metrics through empirical CDFs and aggregates them with a weighted $p$-norm, but only for post-hoc model selection. We borrow its $p$-norm aggregation for the training loss. We deliberately do not use its rank transform, because ranks discard the size of gaps and training needs gradients.
The alignment side supplies the application regime and the evidence that balance matters. ImageBind freezes an image pivot and reaches 66.9% zero-shot on ESC-50. LanguageBind switches to a frozen text pivot and reaches 91.8%. Both avoid pivot competition by freezing the pivot, and both use fixed loss weights. OmniBind learns routing weights over specialist encoders but does not normalize the per-pairing objectives. How do Cross-View and Cross-Modal Alignment Affect Representations motivates the proposal. It shows that forcing an image encoder to match 3D geometry strips texture detail, a genuine trade-off between two objectives that the authors handle with a hand-set weight and a linear adapter. We turn that trade-off into a front that can be traced by varying $\omega$.
The closest existing work is MMPareto. It balances multimodal and unimodal classification losses in a fused classifier, using Pareto directions when gradients conflict and a rescaling step that keeps the update at least as large as the uniform one. Its central negative result also constrains us: textbook Pareto gradient methods underperform plain averaging because they suppress useful gradient noise. MMPareto differs from our proposal in three ways. It balances gradients by conflict geometry, whereas we normalize objectives by an estimated gap. It works on fused task classifiers with two or three losses, whereas we train a shared embedding pivot across four or more pairings. Its rescaling factor is heuristic, whereas our anchors have a definition. Post-hoc alignment (Canonicalizing Multimodal Contrastive Representation Learning, Multi-Way Representation Alignment) plays a supporting role. We test whether a closed-form alignment fitted on frozen features predicts $L_k^*$ cheaply, which would remove the need for specialist runs.
Research Program
The first step builds the testbed at a scale one lab can afford. We use a small shared-pivot binding model: an OpenCLIP-sized image-text pivot trained with low-rank adapters, plus separate light encoders for audio, depth, infrared, and video. Training uses a fixed subset of VIDAL-10M, the paired corpus from LanguageBind. The pivot's image-text loss is one of the objectives. A second testbed is the CREMA-D audio-visual setup of MMPareto, where cross-entropy against $C$ classes has the known worst-case value $\log C$. This lets us compare directly against reported numbers.
The second step estimates the ideal points. For each pairing we run a short specialist training in which only that pairing's loss and the pivot loss update the shared parameters. We record the minimum validation loss as $L_k^*$. We also fit two cheap proxies: a ridge regression and an orthogonal map from frozen modality features to frozen pivot features, following the Procrustes recipe, evaluated for contrastive loss. We measure how well these proxies rank and predict the specialist anchors. If they predict well, the method needs no extra training runs.
The third step trains on $S_p$ for $p \in \{1, 2, \infty\}$ over a grid of preference weights and traces the resulting front of per-pairing zero-shot metrics. We also add MMPareto-style norm preservation, which rescales each update to at least the uniform-sum norm. This checks whether worst-case focus at $p=\infty$ reproduces the Pareto Paradox by starving the optimizer of gradient noise.
The fourth step is analysis. For each pairing we track the share of pivot-gradient norm it contributes and its normalized coordinate $z_k$ over training. The aim is to test the mechanism directly: under uniform summation, noisy pairings should dominate pivot updates while contributing little closable gap.
Baselines
The naive reference is uniform summation of all pairing losses with the pivot unfrozen. This is the ImageBind/LanguageBind loss applied to a trainable pivot, run with identical data, steps, optimizer, and a learning-rate sweep of the same size as ours. A second naive reference freezes the pivot entirely, as those papers do. It is the natural "no competition" point, and every normalized method must be judged against it on the pivot's own metrics.
The strongest existing method is MMPareto, which is also the most directly relevant: it is the only method in the file designed for multimodal loss competition on shared parameters. On the 2+-pairing binding testbed we run it by applying its conflict test and rescaling across all pairing gradients on the shared pivot parameters, with the same tuning budget. On CREMA-D the reported MMPareto multimodal accuracy is 75.13%, against 71.10% for uniform averaging, 72.01% for G-Blending, and 69.19% for OGM-GE (MMPareto). Beating it there means exceeding 75.13% under our reproduction of its protocol. For the binding testbed, beating it means higher worst-case pairing accuracy at equal compute without lowering the pivot's own zero-shot accuracy.
The ablation that isolates the key ingredient keeps the same scalarization but replaces the specialist ideal point with two alternatives. The first is $L_k^* = 0$, which normalizes by $\log B$ alone. The second is the running minimum and maximum of each loss, which is the MOEA/D-style dynamic range. If gap anchors do not beat both, the contribution reduces to the choice of $p$. For ESC-50 as an external reference, the file reports 66.9% for ImageBind and 91.8% for LanguageBind. These come from far larger training runs, so they serve as ceilings rather than targets.
Why not just…?
Why not just keep the pivot frozen, as ImageBind and LanguageBind do? This is the most serious threat. If a frozen pivot matches our method on every emergent pairing, the loss-competition problem does not exist in practice, and this proposal becomes an evaluation contribution. The file contains no reported result showing that unfreezing the pivot helps emergent pairings, so we cannot argue this from prior work. Our frozen-pivot baseline settles it. The proposal is justified only if some trained-pivot configuration dominates the frozen one on at least one pairing without losing pivot accuracy, and if uniform summation fails to find that configuration.
Why not just rely on Adam, since it already normalizes gradient scale? For disjoint encoders this is essentially correct, and it is why we restrict the claim to shared parameters. On the shared pivot, Adam's statistics are pooled across all pairings, so it rescales their sum and not their shares. The gradient-share analysis tests this directly by measuring whether uniform summation lets high-residual pairings dominate pivot updates under Adam.
Why not just divide each loss by its running value, or learn the weights? Running-value normalization is the dynamic min-max rule. Ishibuchi et al. show it over-stretches objectives whose range has collapsed. In our setting, a converged easy pairing would be amplified exactly when it needs attention least. It is included as an ablation. Learned weights would face the Pareto Paradox from MMPareto, which shows that "balanced" weighting can hurt by removing useful noise. We therefore include the norm-preserving variant and report whether it is needed.
Why not just align frozen encoders afterwards with Procrustes? Post-hoc maps are cheap, but on audio-to-vision alignment GPA reaches about 15% retrieval and GCPA 29% (Multi-Way Representation Alignment), far below trained binding. We use such maps only as proxies for the ideal point, not as a competitor for final alignment quality, and we report their retrieval as a lower reference.
Proposed Experiments
On the binding testbed, zero-shot evaluation covers ESC-50 (audio), LLVIP (infrared), NYU-D (depth), MSR-VTT text-video recall at 1, and pivot image-text zero-shot on CIFAR-100, Caltech-101, and DTD. We summarize each trained model in two ways. The first is its worst-case normalized accuracy across tasks, normalized between chance and the specialist run. The second is the hypervolume of the front traced by the $\omega$ grid in those normalized coordinates, with COPA-style percentile aggregation reported alongside for model selection. All runs use three seeds and matched compute. The cost of the specialist runs is charged to our method, and we also report the proxy-anchor variant with that cost removed.
Three outcomes would be decisive. First, gap-normalized $S_\infty$, with or without norm preservation, should raise worst-case normalized accuracy over uniform summation and MMPareto by more than seed variance, while staying within seed variance of the frozen pivot on pivot tasks. Second, the gap-anchor variant should beat both the $\log B$-only and running-range ablations. If it does not, the ideal-point estimate is not doing the work. Third, the Procrustes or ridge proxy should rank the pairings' $L_k^*$ in the same order as the specialist runs, making the method nearly free. On CREMA-D, the same normalization, using a $\log C$ worst-case anchor and a unimodal-specialist ideal, should match or exceed 75.13%. Falling short there while succeeding on binding would suggest the benefit is specific to shared-pivot embedding training. Failing everywhere against the frozen pivot would be a clean negative result: under current recipes, freezing the pivot is the right way to avoid objective competition.
References
- On the effect of normalization in MOEA/D for multi-objective and many-objective optimization
- Boundary Decomposition for Nadir Objective Vector Estimation
- Boundary Decomposition for Finding Nadir Objective Vector in Multi-Objective Discrete Optimization
- COPA: Comparing the incomparable in multi-objective model evaluation
- ImageBind One Embedding Space to Bind Them All
- LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment
- OmniBind: Large-scale Omni Multimodal Representation via Binding Spaces
- How do Cross-View and Cross-Modal Alignment Affect Representations in Contrastive Learning?
- MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistance
- Canonicalizing Multimodal Contrastive Representation Learning
- Multi-Way Representation Alignment