Keeping Imputed Modalities On-Manifold When Streams Drift Out of Sync

Multimodal models often have to fill in a missing input stream, such as the video when the camera drops out or the audio when a microphone fails. One fast way to do this is to learn how the streams line up in time and then build the missing stream as a weighted average of the stream that is still present. This works when the streams are roughly in sync. When speech leads or lags the visible action by more than the model expects, the learned weights spread across the wrong moments. Our hypothesis is that the averaged features then fall outside the region of feature space where real features live, so the downstream classifier sees inputs unlike anything it was trained on. We propose to test this hypothesis directly. If it holds, we fix it with an adversarial critic: a small second network that tries to tell imputed features from real ones, while the imputation model is trained to fool it.

In two sentences: averaging-based imputation under temporal misalignment produces blurred features that fall off the data manifold. A critic that pushes imputed features toward the real feature distribution, conditioned on what was observed, should recover accuracy exactly in the high-offset regime where the averaging assumption breaks. The contribution is the diagnosis plus one targeted fix. The note also commits to comparisons that could show a cheaper loss does just as well.

Core Claim

MM-Align is the method we build on. When all modalities are present during training, it computes an optimal transport plan between them. Optimal transport is a soft matching that assigns each time step of one stream to weighted time steps of another, and MM-Align restricts it to a local time window. A recurrent "fitter" network learns to predict this plan from the stream that is still available. At test time the missing stream is imputed as the plan-weighted sum of the available features. The paper reports a 6.6% to 9.3% accuracy gain over unimodal baselines on CMU-MOSI and a 3x to 4x speedup over generative sequence-to-sequence imputation. Two weaknesses matter here. The imputed codes are convex combinations with no constraint on their distribution, and the fixed window assumes the streams are nearly synchronous.

The claim is that these weaknesses interact. Narrative offsets, meaning lags of seconds between what is said and what is shown, push the true correspondence outside the window. The fitter then produces diffuse plans, and diffuse plans average distinct moments into features that no real sample would produce. A distribution constraint on the imputed codes should help most precisely here. It should give little or no benefit when the streams are in sync. This is a sharp, falsifiable prediction about where the gain should and should not appear.

The constraint comes from the adversarial latent regularization line of work. ARGA trains a discriminator to separate graph embeddings from samples of a Gaussian prior. On PubMed link prediction it reaches 97.1% average precision against 94.7% for the variational graph autoencoder. SCENIR and DSVB use the same device to keep retrieval and fMRI latent spaces compact and free of empty regions. All three regularize a single, synchronous modality toward a fixed prior, and that is the part we change. Adversarial Network Embedding found that the shape of the prior barely matters. This motivates our choice of target: the empirical distribution of real features for the missing modality, rather than a Gaussian. It also motivates an ablation that tests whether the specific target matters at all. The research file raises a related concern. A unimodal isotropic prior could flatten exactly the lag structure that carries cross-modal meaning. Matching to real features, and conditioning on the observed stream, avoids imposing that geometry.

Research Program

The work has three stages.

Stage 1: measure the failure. We run a published-configuration MM-Align on unaligned feature sequences from CMU-MOSI, CMU-MOSEI and MELD. We inject controlled offsets δ between the available streams at test time and again at train/test distribution shift, sweeping δ from zero up to several times the transport window radius. For each δ we record three quantities. The first is accuracy. The second is plan entropy, which tells us how diffuse the predicted plans are. The third is the off-manifold distance of imputed features, defined as the mean k-nearest-neighbour distance from each imputed feature to a held-out set of real features of the same modality, normalised by the same distance among real features. The stage succeeds if accuracy loss tracks off-manifold distance more closely than it tracks plan error alone.

Stage 2: add the critic. The critic $D$ receives a short window of imputed bottleneck vectors $\hat{z}_m$ for the missing modality $m$, together with a pooled summary $s$ of the observed streams. It must separate these from real pairs $(z_m, s)$ drawn from full-modality training samples. The fitter and imputation path minimize

$$\mathcal{L} = \mathcal{L}_{\text{task}} + \lambda_{\text{OT}}\,\mathcal{L}_{\text{plan}} - \lambda_{\text{adv}}\,\mathbb{E}\left[\log D(\hat{z}_m, s)\right].$$

Here $\mathcal{L}_{\text{task}}$ is the sentiment or emotion loss, and $\mathcal{L}_{\text{plan}}$ is MM-Align's loss fitting the predicted plan to the solver's plan. $\lambda_{\text{OT}}$ and $\lambda_{\text{adv}}$ are scalar weights.

Conditioning on $s$ matters. Without it, the critic only asks whether a feature looks like some real video frame. With it, the critic asks whether the feature is a plausible match for this particular utterance. This blocks the degenerate solution of imputing a generic typical frame.

Minimax training is known to be unstable, so we keep the critic small and apply it to low-dimensional bottleneck vectors. We also use a gradient penalty and report variance across seeds. Sentiment clips are short, which avoids the scaling problem the file flags for long audio-video token streams.

Stage 3: identify the active ingredient. We compare three versions of the critic. The marginal critic drops $s$. The Gaussian-prior critic follows ARGA. The conditional critic is the full method. Together these show whether the gain comes from manifold matching specifically, or from generic latent smoothness as the prior-insensitivity result in Adversarial Network Embedding might suggest.

Baselines

Naive reference. The do-nothing point is the unimodal model trained on the available modalities with the missing stream zero-filled. The file reports MM-Align's margin over this kind of baseline on CMU-MOSI, so it anchors the scale of any improvement.

Strongest existing method. This is MM-Align itself, rerun by us under identical feature extractors, splits, surviving rates and seeds. The file gives only MM-Align's relative gain of 6.6–9.3% on CMU-MOSI. It contains no reported absolute accuracy, so our rerun supplies the absolute number. No reported result was found on Lacuna for MM-Align on CMU-MOSEI or MELD in numeric form. Beating MM-Align means a statistically reliable gain over five seeds in accuracy and F1 at large δ, with no loss at δ = 0. A gain only at δ = 0 would contradict our mechanism and would not count.

Generative imputation. We also rerun Modal-Trans, a generative imputation baseline that MM-Align compared against. It tests whether a model that generates the missing stream directly avoids the off-manifold problem by construction.

Cheaper fixes. Three stronger, cheaper competitors are each tuned with the same hyperparameter budget as our method, the same number of trials and the same compute:

  • MM-Align plus an L2 regression loss to the true missing features during training.
  • MM-Align plus a maximum mean discrepancy penalty, a closed-form distance between feature distributions, against real features.
  • MM-Align with random-offset augmentation and a widened window.

Ablation. The marginal-critic and Gaussian-prior critic described in Stage 3 isolate whether conditioning and the choice of target are what matter.

Why not just…?

Why not just regress imputed features onto the true ones during training? Regression learns the conditional mean. The mean is on-manifold only when the correct match is unambiguous. Under large offsets several moments are plausible matches, and their mean is exactly the blurred average we are trying to avoid. If the conditional distribution is in fact nearly unimodal on these datasets, regression will match the critic, and our δ-sweep will show it. In that case we would report regression as the better recipe.

Why not use a closed-form distribution penalty such as maximum mean discrepancy, or a variational KL term, instead of an adversary? These penalties are stable and are a real threat. ARGA's advantage over the variational graph autoencoder (97.1% vs 94.7% AP on PubMed) shows that adversarial matching can beat KL, but that result is on static graphs. It does not cover sequence imputation. The maximum mean discrepancy baseline runs at equal tuning budget. If it ties, we would conclude that the active ingredient is distribution matching, not adversarial training, and recommend the cheaper loss.

Why not just train with random offsets and a wider transport window? This attacks the cause directly, and it may be enough. A wider window, though, admits more candidate matches and can increase plan diffuseness. That would make the averaging problem worse rather than fixing it. We test augmentation alone, the critic alone, and both together. If augmentation alone closes the gap at all δ, the critic is unnecessary.

Why not impute with a diffusion model as DDL-CXR does? DDL-CXR generates up-to-date chest X-ray latents from electronic health record trajectories and reaches 0.822 mortality AUROC, against 0.815 for the record-only baseline. It does keep samples on-manifold. However, it requires iterative sampling, it is sensitive to its noise schedule, and it targets a one-directional image-from-time-series setting. These costs undercut the speed that motivates MM-Align. We include Modal-Trans as the generative comparison and report wall-clock cost alongside accuracy.

Proposed Experiments and Decisive Checks

We run on CMU-MOSI, CMU-MOSEI and MELD across MM-Align's surviving-rate protocol, crossed with the offset sweep δ. For every method we report accuracy, F1, plan entropy, off-manifold distance and inference time.

The first decisive check is Stage 1. If off-manifold distance of MM-Align's imputations does not grow with δ, or does not predict the accuracy drop, the premise fails and we stop.

The second is the shape of the gain. The full method should help increasingly as δ grows past the window radius, and should be neutral near δ = 0.

The third is the ablation ordering. We expect the conditional critic to beat the marginal critic, which should beat the Gaussian-prior critic. If all three tie, the benefit is generic smoothing rather than manifold matching. If the maximum mean discrepancy penalty ties the conditional critic, the adversary is not worth its instability.

Finally, we test transfer. As an exploratory analysis, without a quantitative claim, we take the trained imputer and apply it to egocentric clips in the style of DAVE. The question is whether imputed audio under visual-only input still drifts toward sounds associated with visible objects. That drift is the positive-association failure DAVE documents, where Gemini 1.5 Pro scores about 1% on sound absence detection. The probe asks whether keeping imputed features close to the real distribution, conditioned on the observed stream, dampens that bias.