Many scientific models are simulators. You can run them forward to produce data, but you cannot write down the probability of the data given the parameters. Simulation-based inference sidesteps this by training a neural network on simulated pairs of parameters and data, so that the network outputs the posterior distribution over parameters for any observation. Increasingly the observation comes in several forms at once. A cardiovascular model might be observed through a blood-pressure waveform and two optical pulse signals. A decision-making model might be observed through reaction times and EEG recordings. When one network learns from all streams jointly, it tends to latch onto whichever stream is easiest to exploit and neglect the others. Work on multimodal classification calls this "dominant modality bias." Those methods rebalance training, but they face an unsolved question: how much should each modality contribute? Forcing equal contributions hurts, because modalities genuinely carry different amounts of information.
Our proposal starts from one fact about simulation-based inference. In this setting the right amount of contribution can be checked. Adding data can never, on average, make the true posterior over any parameter less concentrated. So a correctly trained joint model must predict every parameter at least as well as a model that sees only one modality. Whenever a cheap single-modality predictor beats the joint model on some parameter, the joint model has failed to use information it was given. We turn that violation into a training signal. It specifies which encoder to boost, for which parameters, and when to stop. The result is a balancing method whose balance point comes from the inference problem itself rather than from a tuned parity heuristic.
Core Claim
Let $x = (x^{(1)}, \dots, x^{(M)})$ be an observation with $M$ modalities and $\theta_j$ the $j$-th simulator parameter. For the true posterior, the expected negative log posterior density of $\theta_j$ satisfies the standard "conditioning reduces entropy" inequality:
$$ \mathbb{E}\left[-\log p(\theta_j \mid x)\right] \le \mathbb{E}\left[-\log p(\theta_j \mid x^{(m)})\right] \quad \text{for every modality } m, $$
where the expectation is over simulated pairs $(\theta, x)$.
We train small one-dimensional density heads. For each modality $m$ and parameter $j$, a head $q_m(\theta_j \mid x^{(m)})$ reads only encoder $m$. A matching head $q_{\text{joint}}(\theta_j \mid x)$ reads the fused embedding. From these we define a dominance gap on held-out simulations:
$$ g_{m,j} = \widehat{\mathbb{E}}\left[-\log q_{\text{joint}}(\theta_j \mid x)\right] - \widehat{\mathbb{E}}\left[-\log q_m(\theta_j \mid x^{(m)})\right]. $$
A positive $g_{m,j}$, beyond a noise tolerance, is direct evidence that the fused representation discards modality $m$'s information about $\theta_j$. The method scales up the gradient flowing into encoder $m$ and its fusion pathway in proportion to $\sum_j \max(0, g_{m,j} - \tau)$, where $\tau$ is a tolerance estimated from held-out variance. It slows the encoders whose information is already fully used. Once every gap is non-positive, modulation switches off.
The claim is that, at practical simulation budgets, this beats unmodulated multimodal posterior estimation on posterior accuracy and calibration. Gains should concentrate on the parameters informed mainly by the weaker modality. The method should also match or beat classification-derived balancing rules, because those rules aim at the wrong target.
Relation to Prior Work
Three papers motivate the gap. Fuse It or Lose It introduces multimodal neural posterior estimation (MultiNPE). It compares early, late and hybrid fusion of modality-specific summary encoders, and finds early fusion highly sensitive to which modality acts as the query. It treats this purely as an architecture choice and trains with plain joint backpropagation, plus missing-data masks. Simulation-based Inference for Cardiovascular Models shows that different pulse signals constrain different physiological parameters: digital PPG constrains vascular resistance and pulse wave velocity, while radial PPG constrains arterial diameter. That is exactly the parameter-specific asymmetry our gap is built to detect. FUSE uses "multimodal" to mean the split between parameter and observation streams, plus multi-peaked posteriors. It trains with standard flow matching and does not balance heterogeneous observation streams. We do not compete with its simulator-steered sampling; we target the amortized regime.
The method is borrowed from the modality-balancing literature and modified. Classifier-guided Gradient Modulation (CGGM) uses lightweight unimodal auxiliary heads and their relative improvement rates to rescale and redirect gradients, and it extends balancing to regression and segmentation. We reuse the auxiliary-head idea but replace classifiers with density heads. We also replace relative improvement rate with the dominance gap, which has a known sign under correct inference. See-Saw Modality Balance (BALGRAD) projects conflicting gradients. It also reports that bias persists even with frozen pretrained encoders, which constrains our design: balancing must act at the fusion layer as well, not only on encoders. Adaptive Unimodal Regulation penalizes dominant encoders based on performance disparity, which is the closest analogue to our slow-down arm.
BalanceBenchmark plays two roles. It frames the open "relative balance point" problem that the dominance gap answers. It also reports that optimization-level methods can cost up to 2.5 times more FLOPs, a constraint we respect by using one-dimensional heads rather than extra masked forward passes. None of these papers operates on a probabilistic posterior objective, and none has an information-theoretic stopping criterion.
Research Program
The first stage is diagnostic, and it can falsify the whole project. We train vanilla late- and hybrid-fusion MultiNPE on the cardiovascular task, the reaction-time-plus-EEG task, and the 10-dimensional synthetic task from Fuse It or Lose It. Alongside, we train detached per-modality and joint marginal heads, which do not affect training. We then log $g_{m,j}$ across training and across simulation budgets. If the gaps are never significantly positive at realistic budgets, the greedy-learner story does not transfer to posterior estimation, and we will report that as the finding.
We also build one controlled simulator with a known analytic posterior. In it, a low-dimensional "easy" modality constrains $\theta_1$. A high-dimensional, nonlinearly encoded "hard" modality alone constrains $\theta_2$ and weakly constrains $\theta_1$. The difficulty of the hard modality can be dialed up or down. This lets us measure imbalance against the exact posterior using the classifier two-sample test from Benchmarking Simulation-Based Inference.
The second stage builds the method on top of the MultiNPE architecture. Each modality-specific encoder gets a one-dimensional mixture density head per parameter, trained with gradients into that encoder. The fused embedding gets matching heads. Gaps are estimated on a rolling held-out buffer of simulations, so that overfitting of the single-modality heads cannot manufacture false violations. The tolerance $\tau$ is set from the bootstrap standard error of the gap. The modulation coefficient multiplies the gradient at encoder $m$'s output and at its fusion input.
For models where fusion-layer conflict turns out to matter, we optionally add a BALGRAD-style projection, but only as an ablation. The main flow loss (a normalizing flow or flow matching objective, as in BayesFlow 2.0) is unchanged. The heads are discarded at inference, so amortized speed is preserved.
The third stage is analysis. We check whether the parameters with the largest positive gaps under vanilla training are the ones whose posteriors improve. For the cardiovascular model, we predict these include arterial diameter, informed by radial PPG. We also test whether the benefit shrinks as the simulation budget grows, which would mark the effect as a data-efficiency phenomenon.
Baselines
The do-nothing reference is vanilla MultiNPE with hybrid and late fusion, trained with unmodulated joint backpropagation exactly as in Fuse It or Lose It. Its reported numbers anchor the comparison: a maximum mean discrepancy of 0.23 for hybrid fusion on the 10-dimensional synthetic benchmark (versus 1.89 for a single source), and a posterior contraction of 0.90 on the cardiovascular task (versus about 0.55 for single-source models). For the reaction-time-plus-EEG task we found no reported result on Lacuna, so we will rerun the baseline ourselves. Beating this reference means a lower discrepancy, or higher contraction with a calibration error no worse, at equal simulation budget.
The strongest existing competitor is CGGM (Classifier-guided Gradient Modulation) transplanted to posterior estimation. It uses the same density auxiliary heads, but its modulation is driven by relative improvement rate and its directional loss, as published. It is the strongest competitor because it already handles non-classification losses with cheap auxiliary heads and reports the best results in its settings, such as 73.94 average Dice on BraTS 2021 against 71.80 for the previous best. Those numbers are not SBI numbers; they only establish it as the method to beat. We also include an OGM-style convergence-rate rule, since BalanceBenchmark groups it with the optimization-level methods.
Fairness is enforced by construction. Every method uses the same summary and flow architectures, identical simulation sets, and the same auxiliary-head capacity. Each gets the same number of hyperparameter trials, chosen on a shared validation split of simulations, and the same wall-clock budget, with FLOPs reported.
Two further arms complete the comparison. The key ablation keeps our exact auxiliary heads and gap estimates but replaces the gap-driven trigger with a parity target. This isolates the claim that the target, not the heads, matters. The last arm is a missing-modality dropout baseline at a tuned rate, as MultiNPE already supports. Winning over CGGM means statistically significant improvement across seeds, with the gains located on weak-modality parameters.
Why not just…?
Why not just drop modalities at random during training? Random dropout forces each encoder to be useful on its own. That can indirectly repair neglect, and it is a real threat. Fuse It or Lose It uses masking for robustness to missing sensors, but it does not test whether masking fixes competition when all modalities are present. BalanceBenchmark finds that such feed-forward methods often miss the relative balance point. Dropout is also blind to which parameter is neglected, and it spends capacity learning single-modality posteriors nobody asked for. We tune the dropout rate as generously as our own method. If it closes the positive gaps equally well, we will say so. The dominance gap then still serves as the diagnostic that tells practitioners when dropout is needed.
Why not just pretrain each encoder alone and freeze it? See-Saw Modality Balance reports that dominant modality bias reappears in the fusion layers even with frozen pretrained encoders. Freezing also forbids learning cross-modal features. In Fuse It or Lose It, hybrid cross-attention fusion needed exactly such features to outperform the alternatives. We include this arm; the dominance gap on the fused heads will show whether fusion still discards information.
Why not just add single-modality losses with fixed weights? This is the closest cheap alternative, and our parity-free ablation shares its heads. BALGRAD's authors report that static reweighting cannot track imbalance that shifts during training, and CGGM observes that one modality saturates earlier than the others. Our design goes further: it switches the auxiliary pressure off once the information inequality holds, so it cannot over-regularize late in training. We will grid-search fixed weights with the same trial budget. If a single fixed weight matches our method on every task, the dynamic trigger is unnecessary and the contribution shrinks to the diagnostic.
Why not just simulate more data or train longer? If imbalance is only a transient of optimization, a larger budget should erase it. For cheap simulators this may well be true. Our budget sweep, spanning a few thousand to over a hundred thousand simulations, directly measures the budget at which vanilla training closes the gaps. The method's value is the simulations it saves below that point. That matters most for expensive simulators like the hydrodynamic solver behind colliding-wind binaries, whose authors trained on 40,000 simulations.
Proposed Experiments
The benchmarks are the controlled two-modality simulator with an analytic posterior, the three tasks from Fuse It or Lose It, and the in-silico cardiovascular data of Simulation-based Inference for Cardiovascular Models, with transfer to MIMIC-III PulseDB as a stress test. Metrics are chosen to match what each task allows:
- The classifier two-sample test and maximum mean discrepancy against reference posteriors, where references exist.
- Posterior contraction and simulation-based calibration error everywhere.
- Per-parameter held-out negative log posterior.
- Accuracy when the dominant modality is corrupted at test time.
- Training FLOPs and memory.
Each configuration runs with five seeds and three simulation budgets.
Three checks are decisive. First, on the controlled simulator, vanilla training should show a positive gap on $\theta_2$ that grows with the hard modality's difficulty, and our method should drive the two-sample test on $\theta_2$ toward chance while leaving $\theta_1$ unchanged. Second, on the cardiovascular task, improvements should concentrate on the parameters that radial PPG uniquely informs, not be spread uniformly across parameters. Third, the parity-target ablation should do worse than the gap-driven trigger on at least the tasks where modalities carry unequal information. That would show the computable balance point, not merely the extra heads, is what helps.
If the first diagnostic stage finds no significant positive gaps at realistic budgets, the correct conclusion is that multimodal posterior estimation does not suffer greedy learning in practice. That negative result would still be useful to the field.
References
- Fuse It or Lose It: Deep Fusion for Multimodal Simulation-Based Inference
- Simulation-based Inference for Cardiovascular Models
- FUSE: FK-Steered Multi-Modal Flow Matching for Efficient Simulation-Based Posterior Estimation
- Classifier-guided Gradient Modulation for Enhanced Multimodal Learning
- See-Saw Modality Balance: See Gradient, and Sew Impaired Vision-Language Balance to Mitigate Dominant Modality Bias
- Adaptive Unimodal Regulation for Balanced Multimodal Information Acquisition
- BalanceBenchmark: A Survey for Multimodal Imbalance Learning
- Benchmarking Simulation-Based Inference
- BayesFlow 2.0: Multi-Backend Amortized Bayesian Inference in Python
- Amortized Simulation-Based Inference of Colliding-Wind Binaries from Short, Noisy Image Time Series