Keeping the Tokens That Only Matter Together: Synergy-Aware Token Pruning Distilled from a Dense Multimodal Teacher

Multimodal transformers that read video, audio and text together are expensive. Attention cost grows with the square of the combined sequence length, and video and audio contribute many frames that carry little signal. The standard remedy is token pruning: score each token, keep the top few, and send only those into the costly fusion layers. This works on average. It has a blind spot, though. A neutral face, or a flat tone of voice, looks useless when judged alone, yet it can be the very cue that turns a positive sentence into sarcasm. Signals of this kind are called synergistic: they carry information only in combination with another modality. Every pruning score we know of is computed from tokens judged one at a time or by their agreement with other modalities, so it is structurally inclined to throw such signals away.

We propose to learn the pruning score from a large, unpruned multimodal "teacher" model. For each token, the target is how much the teacher's prediction changes when that token is masked while all other modalities stay present. A small, cheap scorer is distilled from these targets, which are computed offline. At inference the scorer decides which tokens a compact "student" transformer gets to see, and the student is itself distilled from the same teacher. The claim is specific and testable. At the same compute budget, a scorer trained on the teacher's joint masking effects keeps synergistic tokens that consistency-based pruning discards. The gap should be largest on tasks where meaning depends on cross-modal combination, such as sarcasm and humor, and on a controlled XOR-style task, where it should be dramatic.

Why this gap is real

Two results in the file pull against each other. ConsMSA projects every text, audio and video token into a shared sentiment-distribution space and scores it by confidence and by the agreement (Jensen–Shannon divergence) of its local mini-prediction with tokens from its own and other modalities. It then discards low scorers before cross-modal fusion. It reports that keeping 80% of tokens on MOSI beats keeping all of them, and that pruning 90% gives a 2.03x speedup on BERT-based backbones and 2.16x on RoBERTa-based ones. The More, the Merrier (ConFu) proves that pairwise objectives cannot capture XOR-like dependencies, in which a single modality carries zero information about the label. It handles them with compositional "two modalities fused, aligned to a third" objectives.

Put the two together and the conclusion follows. A token whose mini-prediction is near-uniform on its own gets low confidence. Its agreement with other modalities is also uninformative, because there is nothing to agree with. That token is exactly the kind ConFu says must be kept. ConsMSA also reports degradation on the rare extreme-sentiment classes. That failure is consistent with, though it does not prove, a scorer that favors "typical" tokens. ConsMSA therefore motivates the proposal, and ConFu supplies the reason to expect it to fail in a predictable place.

The method borrows from two distillation papers. MTDVocaLiST compresses an audio-visual transformer by 83.52% by matching the teacher's cross-attention maps and value-relation matrices rather than only its outputs. We reuse that behavioral distillation for the student, extending it from two modalities to three. MIND distills from unimodal teachers and leaves cross-modal interactions unsupervised. Our teacher is jointly multimodal precisely so that its interactions can be measured and transferred. Our use of distillation also differs from both papers. They distill a student's representations. We additionally distill a decision about which inputs to compute on, which neither paper does.

Core mechanism

Let $f$ be the dense teacher, trained with modality dropout so that it behaves sensibly when inputs are masked. Let $x$ be a full trimodal input and $x_{\setminus t}$ the same input with token $t$ replaced by a learned mask embedding. Masking with a learned embedding rather than zeros follows the finding in Learning Contrastive Multimodal Fusion with Improved Modality Dropout that learned absent-modality embeddings beat zero-padding. Write $x^{(m)}$ for the input with every modality except $m$ (the modality of $t$) masked. We define the joint effect and its synergistic part:

$$e_t = D\big(f(x),\, f(x_{\setminus t})\big), \qquad s_t = e_t - D\big(f(x^{(m)}),\, f(x^{(m)}_{\setminus t})\big),$$

Here $D$ is the Jensen–Shannon divergence between predictive distributions. The quantity $e_t$ measures how much the token matters with all modalities present. The quantity $s_t$ is the portion of that effect that disappears once the other modalities are removed, so a large $s_t$ means the token matters only in combination.

The scorer is trained to rank tokens by $e_t$. We use $s_t$ only as an analysis tool, to check whether the scorer keeps synergistic tokens. The scorer is a two-layer network that sees each token plus a pooled summary of the other modalities. The cross-modal context is essential, since a token's joint effect depends on what the other streams contain. It is trained with a listwise ranking loss against $e_t$.

Computing $e_t$ exactly costs one teacher forward pass per token. This is feasible offline on MOSI-scale sequences, but we also test a cheaper estimate that masks contiguous chunks and spreads the effect over the chunk's tokens. The teacher is never run at inference.

The student is a small fusion transformer. It sees only the top-$k$ tokens the scorer selects, and it is trained on task labels, on the teacher's soft outputs, and on cross-attention distribution matching in the manner of MTDVocaLiST, restricted to the kept tokens. Following MTDVocaLiST, uncertainty weighting balances the loss terms, which avoids hand-tuning their weights.

Research Program

The work has three stages.

The first stage builds the teacher and the attribution data. We train a dense all-to-all trimodal transformer with modality dropout on each benchmark and compute $e_t$ and $s_t$ for every training token. Before any pruning, this stage answers a descriptive question: how much of the teacher's token-level effect is synergistic on MOSI, MOSEI, CH-SIMS, MUStARD and UR-FUNNY? If $s_t$ is negligible everywhere on the real benchmarks, the proposal's premise is weak for those benchmarks, and we will report that.

The second stage trains the scorer and the student at keep-rates of 50%, 20% and 10%.

The third stage tests whether the approach survives a fourth modality. We add a synthetic fourth stream with a planted pairwise XOR dependency, giving a controlled $M=4$ check. Pruning cost should grow linearly in tokens rather than in modality subsets, which is the scaling problem ConFu and Deep-HOSeq leave open. We do not claim real four-modality benchmark results, because the file contains no such benchmark with reported numbers.

Baselines

The naive reference point is the distilled student with no pruning. It uses the same architecture and the same teacher distillation but sees all tokens, and it defines the accuracy ceiling and the FLOPs we aim to cut. Random top-$k$ pruning at each keep-rate is the floor any learned scorer must clear.

The strongest existing method is ConsMSA. The file contains no absolute MOSI or MOSEI accuracies for it. Its reported figures are relative: Acc-3 improved by 1.31% and Acc-5 by 1.39% over the previous best on CH-SIMS, the 80%-beats-100% result on MOSI, and the 2.03x/2.16x speedups at 90% pruning. We will therefore re-run ConsMSA's scoring in our pipeline. We use the same backbone features, the same student, the same keep-rates and the same tuning budget (an equal number of hyperparameter trials per method), and we report accuracy against measured FLOPs and wall-clock latency.

Beating ConsMSA means two things. First, the accuracy-versus-FLOPs curve must be equal or higher across the three keep-rates on MOSI, MOSEI and CH-SIMS. Second, accuracy must be clearly higher at 10–20% keep on MUStARD and UR-FUNNY, where synergy should matter most, with no reported result found on Lacuna for ConsMSA on those two datasets. For orientation on MOSEI, the file reports Graph-MFN at 76.9% binary accuracy (CMU-MOSEI). That is an old reference, not the bar.

LoCoMT reduces cost differently. It restricts each attention head to within-modality or cross-modality views, and it reports GFLOPs reduced by up to 81% on MedVidCL for a 1% F1 loss. We compare against it at matched FLOPs on MOSEI and test whether the two methods compose, since head views and token pruning are orthogonal.

Two ablations isolate the key ingredient. The first replaces the target $e_t$ with the masked-modality effect $D(f(x^{(m)}), f(x^{(m)}_{\setminus t}))$, a marginal, teacher-derived score; any remaining gain then comes from the joint, cross-modal signal rather than from simply having a teacher. The second keeps our scorer but drops the student's behavioral distillation, which separates "choosing the right tokens" from "training the student better."

Why not just…?

Why not just prune by the teacher's attention weights? Attention mass is free to compute and is the most common importance proxy, so it is the most serious threat. Its weakness is that attention reflects where the model looks, not what changes its answer. A token can draw little attention mass and still flip a prediction through one decisive cross-attention head. We run attention-mass pruning, taken from the teacher's cross-modal heads, as a full baseline. If it matches our scorer on MUStARD and on the synthetic XOR task, the costly masking targets are unnecessary, and we would recommend attention-based distillation of the scorer instead.

Why not just learn the top-$k$ selection end to end with a differentiable relaxation and the task loss? End-to-end selection, for example with Gumbel-softmax (the device MODES uses to mask latent dimensions), optimizes exactly what we care about. It must, however, discover synergistic pairs from gradient signal on small datasets: MOSI has 2,199 clips. Gradients through one token of an XOR pair are weak while its partner is being dropped. We include this baseline with the same student and budget. Our prediction is that it performs comparably on MOSEI but worse on MUStARD and XOR. If it matches us everywhere, the teacher's role reduces to ordinary output distillation.

Why not just prune less aggressively and distill the outputs? At 50% keep, every scorer may perform about equally, and logit distillation may recover the rest. This is plausible, which is why we report full curves rather than a single operating point. The claim lives at the 10–20% keep-rates, where ConsMSA's speedups come from. If the curves converge there as well, the method has no practical value.

Decisive checks

Three results would settle the question.

The first is the synthetic XOR task, at three and four modalities. Here ConsMSA-style and marginal scorers should fall to chance at low keep-rates, while the joint-effect scorer stays near the dense student. If it does not, the mechanism is wrong.

The second is an analysis on MUStARD and UR-FUNNY. We bin kept tokens by the teacher's $s_t$ and check whether our scorer retains high-synergy tokens at a higher rate than ConsMSA, and whether retaining them explains the accuracy gap. Examples whose errors flip between the two methods should have markedly higher total $s_t$ than examples where both methods succeed.

The third is the full accuracy-versus-latency curves on MOSI, MOSEI and CH-SIMS, with the scorer's own cost included. A gain that vanishes once scorer latency is counted is not a gain.

We would also report the chunked-attribution approximation against exact $e_t$. If the approximation degrades the scorer, the offline cost limits how far the method scales to long video.