When Sound Carries the Evidence: Oracle-Tested Audio-Visual Token Selection

Models that watch and listen to video, often called omni-modal language models, turn a 20-second clip into more than 20,000 input tokens. Every practical system therefore throws most of those tokens away before reasoning. The methods that decide what to keep almost always let vision decide. They keep visual patches that look salient, then keep the audio that matches those patches. Benchmarks built so that the answer depends on sound show that these models fail when the decisive evidence is acoustic: off-screen speech, a change in background noise, or a shift in the music. We propose to settle whether this "audio neglect" is a failure of selection, meaning the right evidence is discarded or ignored before reasoning, or a failure of representation, meaning the audio encoder never turned music and environmental sound into something the language model can use. We then propose to fix the part that turns out to be broken.

The diagnostic comes from text summarization. There, researchers showed that supplying human-chosen keywords improves summaries far more than letting the model pick its own, which locates the bottleneck in filtering rather than in writing. We carry that "oracle" test over to audio-visual reasoning by handing the model human-marked evidence spans in both modalities. The method rests on a specific hypothesis. Relevance scores computed for audio tokens and for visual tokens are not directly comparable, because the two modalities sit in separate regions of the model's internal representation space. Any single ranked cut across modalities therefore starves the one whose scores run low. Our selector ranks tokens within each modality, splits the token budget across modalities per question, and always keeps the co-occurring tokens of the other modality when it keeps a segment.

Core Claim

Selection in omni-modal models fails for two reasons. First, cross-modal relevance scores are incommensurable, so one global ranking or a vision-anchored selector under-retains audio. Second, pruning within short local windows breaks the time alignment between what is seen and what is heard. If this is right, three things should follow on audio-dependent questions. Oracle evidence should close a large share of the gap to humans. A within-modality, budget-split, synchrony-preserving selector should recover a measurable part of that oracle gain. And the gain should be largest where vision-first selectors are worst. If instead oracle evidence barely helps on music and ambient-sound questions, the bottleneck is representational, and selection research is the wrong lever for those categories. That is a useful negative result in its own right.

Where This Sits

The file contains a clear chain of evidence for the gap. OmniVideoBench built 1,000 questions over 628 videos, each with step-by-step reasoning chains that record whether vision or audio supplies each step. It found near-random accuracy for many open models, a collapse when audio is removed, and every model below 50% on background and music understanding. The benchmark diagnoses the problem but proposes no mechanism. We use its chains as the scaffold for oracle annotation.

The closest method is OmniSIFT. It prunes video by spatial and temporal saliency, then uses the retained visual tokens to choose audio tokens, and trains the discrete top-K choice with a straight-through gradient estimator. Its own ablation shows that an audio-only selector does worse, so it argues against naively letting audio lead. We take that result as a constraint: audio should not replace vision as the anchor. Our design differs from OmniSIFT in three ways. Neither modality is the anchor. Ranking happens globally over the whole video rather than inside 2-second chunks. The budget adapts per question rather than staying fixed, which answers two of OmniSIFT's stated limitations.

SMART likewise uses visual captions as the query that decides when auxiliary streams matter. It also converts all modalities to text, which its own analysis concedes loses acoustic texture. That loss matters most exactly where OmniVideoBench finds models weakest.

The scoring signal comes from vision-only work. FlexSelect showed that attention from the question to visual tokens at an intermediate layer of the host model ranks relevance better than external retrievers, and that a 0.5B model can learn to imitate this ranking. We extend that signal to audio tokens and test whether it can be compared across modalities.

Two further papers constrain the ranking design. Threading Keyframe with Narratives showed that pure top-K selection collapses onto redundant moments, so a diversity term is needed. LongVU found that language-aligned features are poor at detecting redundancy; we therefore keep redundancy pruning separate from relevance ranking.

The cross-modal gap literature supplies the hypothesis. Learning Robust Visual-Semantic Embeddings showed that pairwise contrastive training aligns matched pairs but leaves the overall distributions of each modality as separate clusters. Extending Multi-modal Contrastive Representations showed that a single projection layer cannot close the gap within a modality while also fitting an external embedding space. If audio and visual tokens occupy separate clusters, their attention scores from a shared query have different scales, and a single global top-K is biased by construction.

Research Program

Oracle evidence spans. We annotate time-stamped evidence spans for every OmniVideoBench question, one set for audio and one for visual. Annotators follow the benchmark's per-step modality labels, and a second annotator checks each span. We release the spans. With them we run the oracle test: the host model receives only the tokens inside the oracle spans, plus a thin uniform sample for context, at the same token budget as the selectors. The selection gap for a method is oracle accuracy minus that method's accuracy, reported per task type.

Measuring the scoring bias. At a reference layer chosen with FlexSelect's layer-search procedure, we record query-to-token attention for every audio and visual token. We then compare the share of each modality kept by a global top-K against that modality's share of the oracle evidence. A systematic shortfall for audio that follows the scale difference between the two score distributions would be direct evidence for the incommensurability hypothesis.

The selector. Let $a_i$ be the attention score of token $i$ in modality $m$ at the reference layer. We replace $a_i$ with its percentile rank $r_i$ among that modality's tokens in the same video, which removes scale differences between modalities. A small head reads the question and a coarse pooled summary of the video. From these it predicts budget fractions $\beta_m$ that sum to one, and modality $m$ keeps its top $\beta_m B$ tokens by $r_i$, where $B$ is the total budget. Selection runs over the whole video and uses the diversity-constrained greedy procedure of Nar-KFC rather than raw top-K. A paired-retention rule then keeps a few tokens from the other modality in the same time window, so that sight and sound for each selected moment stay aligned. The budget head is trained on the AVoCaDO dataset (107K synchronized audio-visual instruction pairs) to maximize answer likelihood, using OmniSIFT's straight-through estimator. The host model stays frozen. OmniVideoBench is used only for evaluation.

Branch if selection is not the bottleneck. If oracle spans fail to help on the music and background-sound categories, the problem is representational. We then test the cheapest known fix: tuning only the normalization parameters inside attention blocks on AVoCaDO. Tuning LayerNorm in Attention reports that this outperforms LoRA on image-text tasks, but it has never been tried on audio or video.

Baselines

Do nothing. The naive reference is the full-token host, Qwen2.5-Omni-7B, run on all benchmarks with no pruning, alongside uniform token sampling at our budget. For the full-token host, OmniSIFT reports 49.7% on WorldSense. We found no reported result on Lacuna for Qwen2.5-Omni-7B on OmniVideoBench, so we measure it ourselves. The largest open reference on OmniVideoBench is Qwen3-Omni-30B-A3B at 38.40% (OmniVideoBench).

OmniSIFT. This is the strongest existing method. It is the only asymmetric audio-visual selector trained end to end on the same host and the same fine-tuning data. It reports 49.9% on WorldSense while keeping 25% of tokens. We found no reported result on Lacuna for OmniSIFT on OmniVideoBench or DailyOmni. We rerun it with the released recipe on AVoCaDO, at the same 25% budget, on the same host, with the same tuning budget as our selector. Beating it means higher accuracy on OmniVideoBench audio-dependent questions and on WorldSense at equal budget and comparable latency, with no loss on vision-dominant task types.

Keep all audio, prune video with FlexSelect. Audio tokens are far fewer than visual ones, so this baseline tests whether audio selection matters at all. It is matched on total token count.

Ablations. Three ablations isolate the key ingredients. The first uses a single global top-K on raw attention scores with no percentile normalization; this directly tests incommensurability. The second fixes $\beta_m$ to the average split in the training data, which tests whether per-question allocation matters. The third removes paired retention, which tests the synchrony argument.

Why not just…?

Why not just keep all the audio and prune only video? This is the most serious threat. Since audio is cheap, it may well match our method at equal budget. The oracle study settles it partly. If audio-dependent questions still fail when all audio is present, the missing piece is either visual evidence that the sound points to, or representation. Our selector adds value over this baseline only if it recovers visual moments tied to audio cues. We will report honestly if it does not, and then narrow the contribution to audio-guided visual selection.

Why not just transcribe and caption the audio, as SMART does? For speech this may be competitive, and we include a WhisperX-plus-captioner textualization baseline. But SMART itself acknowledges that textualization loses acoustic texture. Music and ambience are the categories where OmniVideoBench finds every model below 50%, so we expect textualization to trail on exactly those. A per-category comparison will show whether it does.

Why not just fine-tune the whole model on audio-visual data with LoRA? If audio neglect were purely representational, fine-tuning would be the right tool, and selection would be irrelevant. The oracle branch tests exactly this. We run the LoRA and LayerNorm-tuning arms on AVoCaDO with a matched compute budget, both with and without our selector, and check whether the two gains add up or overlap.

Decisive Checks

Evaluation uses OmniVideoBench, broken down by its 13 task types, together with WorldSense and DailyOmni. Metrics are accuracy, selection gap, evidence recall against the oracle spans, and the accuracy drop when audio is removed. We also report latency and peak memory.

The proposal stands or falls on three results. First, oracle spans must close a substantial fraction of the gap to the 82.69% human baseline on audio-dependent questions. Second, the global raw-score ablation must under-retain audio relative to oracle evidence share in the way the scale difference predicts. Third, the full selector must beat OmniSIFT and keep-all-audio at equal budget, with the largest margins on background and music questions and on questions whose evidence spans lie far apart in time. If the first check fails, the paper becomes a documented case that omni-modal audio neglect is representational, and the normalization-tuning arm becomes the main contribution.