Learning When to Caption the Audio

Models that watch video and listen to its soundtrack often ignore the sound. Their audio features sit in a different region of the shared embedding space from the text and vision features, so the language model effectively never uses them. The most reliable fix is to have a separate audio model write a caption of the soundtrack, which puts the audio into words the language model already understands. The fix works, but it is slow. On the OmniCVR retrieval benchmark, a vision-language embedding model reaches 12.4% Recall@1 on audio-centric queries when audio is fed as raw features, and 77.2% when audio is first captioned. Captioning costs about 1.77x the latency of the vision-only model, and it discards precise timing (OmniCVR).

We propose to learn, for each short time segment of a video, which of three forms its audio should take: dropped entirely, passed as a few compressed feature tokens, or captioned into timestamped text. A small trained selector makes this discrete choice, and it is trained end to end through the downstream task under an explicit latency cost. The claim is that most segments need no caption, and that a learned per-segment selector can recover most of the accuracy of always captioning at close to vision-only cost. Heuristic gates, such as audio loudness or a classifier that asks whether the query mentions sound, cannot do the same at matched latency, because the need for audio varies within a clip and not just between queries.

Core Claim and What Is New

The three input directions play different roles here. Differentiable discrete routing is the mechanism: a component that makes hard, categorical choices yet can still be trained by gradient descent, using Gumbel-Softmax (a noisy, temperature-controlled softmax that approaches a one-hot choice) or a straight-through estimator (hard choice forward, smooth proxy gradient backward). Temporal audio-video streams are the setting. The cross-modal representation gap is both the thing being managed and what we measure: the selector's decisions become a per-segment readout of where latent audio features are "good enough" and where they must be converted to language.

The closest existing work takes one of three stances, and none learns the choice we propose. OmniSIFT prunes audio tokens with a straight-through top-K selector guided by vision, reaching 49.9% on WorldSense with 25% of tokens against 49.7% for the full-token Qwen2.5-Omni-7B. It keeps a fixed budget inside 2-second chunks, and it never changes how audio is represented, only how many latent tokens survive. Dynamic Multimodal Fusion uses Gumbel-Softmax to select which modality branches run, cutting 46.5% of compute on CMU-MOSEI. It works on static or slice-aggregated inputs and names long temporal streams as future work. OmniCVR's audio-as-text approach applies captioning uniformly.

We differ from all three in two ways. Our selector chooses between representation types that sit on opposite sides of the modality gap, rather than between subsets of tokens or encoders. It also chooses per time segment, so a clip can caption the three seconds containing a siren and drop the rest.

Research Program

We split each clip into fixed segments of about two seconds, matching OmniSIFT's chunking so comparisons are fair. For every segment $t$ we have three candidate audio representations:

  1. Skip: no tokens.
  2. Latent: a small number of audio-encoder tokens compressed by the model's existing projector.
  3. Caption: a short caption from Qwen2-Audio, prefixed with a segment timestamp token so timing survives.

The selector is a two-layer network. It reads the pooled latent audio features of the segment, the pooled visual features, and the query embedding. All three are already computed, so the selector adds negligible cost.

Training. The obstacle is that caption generation is autoregressive and not differentiable. We sidestep it by precomputing captions for every training segment offline. Routing then becomes a discrete choice among three fixed token sequences, and Gumbel-Softmax with a straight-through hard forward pass applies directly. At inference, captions are generated only for segments the selector picks. The objective is

$$ \mathcal{L} = \mathcal{L}_{\text{task}} + \lambda \sum_{t} \sum_{k \in \{\text{skip},\text{latent},\text{caption}\}} p_{t,k}\, c_k, $$

where:

  • $\mathcal{L}_{\text{task}}$ is the contrastive retrieval loss or the question-answering loss;
  • $p_{t,k}$ is the selector's probability of choice $k$ at segment $t$;
  • $c_k$ is the measured wall-clock cost of that choice, dominated for captions by generation time;
  • $\lambda$ sets the trade-off. We sweep it to trace an accuracy-latency curve rather than report one point.

Stable training. AR-Net needed three training stages to stop its routing policy from collapsing, so we use a simpler two-phase schedule. First we train the downstream model with routes sampled uniformly at random, so it learns to consume all three representations. Then we train the selector with temperature annealing and the cost term. We will report collapse rates across seeds.

Analysis. For each segment we compute a gap score: how much the task margin improves when that segment's latent audio is replaced by its caption. We then test whether the selector's caption decisions track this score. If they do, the selector gives a cheap, learned map of where the audio-text gap actually bites. The map is itself a measurement the current literature does not have.

Baselines

The naive reference points are the two fixed policies at the ends of the curve.

Latent-only. Audio always goes in as latent tokens. This is VLM2Vec as reported in OmniCVR: 38.44% overall R@1 and 12.4% on audio-centric queries. That paper's ablation also reports 13.6% R@1 for latent audio tokens against 32.7% with text descriptions.

Always-caption. The strongest existing method on this question is AudioVLM2Vec from the same paper, at 66.98% overall and 77.2% audio-centric R@1 with the roughly 1.77x latency penalty. We rerun both fixed policies on our hardware with the same backbone, training triplets and tuning budget, and report latency measured the same way.

OmniSIFT-style selection. We run OmniSIFT's vision-guided fixed-budget audio token selection on the same backbone. It tests whether pruning latent tokens alone, without ever captioning, closes the gap. On WorldSense its reported 49.9% is the number to beat.

Heuristic gates. These are the real threats. One gate captions a segment when its audio energy or voice-activity score exceeds a threshold. Another captions every segment of a query when a text classifier predicts the query is audio-related. Each threshold is tuned on the validation split to hit the same latency as our selector.

Ablations. Four ablations isolate the key ingredient:

  • a per-clip rather than per-segment selector;
  • a selector without query input;
  • a soft mixture that weights all three representations instead of choosing one, which by construction saves no generation time;
  • a per-segment oracle that picks the cheapest route not hurting the task margin, which gives an upper bound.

What counts as success. The selector must reach at least 95% of AudioVLM2Vec's overall R@1 at no more than 1.2x vision-only latency. It must also beat the best heuristic gate by a clear margin at equal latency on the integrated split.

Why not just…?

Why not just always caption the audio? Always captioning is the accuracy ceiling we are chasing. OmniCVR itself reports the 1.77x latency and loss of timing. If the accuracy-latency curve shows no point meaningfully cheaper than always-caption at similar accuracy, the proposal fails, and we will say so.

Why not just gate on audio loudness or on the query text? This is the most serious alternative. On OmniCVR's audio-centric split (20% of queries), a query classifier may do nearly as well as a learned selector, because the need for audio is decided by the query. Our claim therefore rests on the integrated split (57.2% of queries) and on WorldSense and DailyOmni, where only some moments of a clip carry decisive sound. There, per-segment decisions using the audio features should beat both gates. If they do not at matched latency, the learned selector is not justified and the heuristic should be adopted.

Why not just fix the latent audio features so captions are unnecessary? Cheaper adaptation such as tuning LayerNorm in attention, or shared codebooks such as Cross-Modal Discrete Representation Learning and UniCode, might strengthen the latent path. We include a LayerNorm-tuned latent path as a baseline and also as the latent option inside the selector. If it closes the gap, the selector should learn to stop captioning, which is a legitimate outcome that the gap-score analysis would expose. The 13.6% versus 32.7% ablation suggests the gap is large enough that this is unlikely, but the experiment decides it.

Why not just prune more aggressively, as OmniSIFT does? Pruning chooses which latent tokens to keep. It cannot turn an audio event the language model ignores into one it attends to. The OmniSIFT baseline tests this directly on the audio-centric split.

Proposed Experiments

Retrieval (OmniCVR). Using the released OmniCVR data, we train on the 50,000 triplets and evaluate R@1 on vision-centric, audio-centric and integrated splits, with a VLM2Vec-family backbone. Wall-clock latency is measured on the 2,000-candidate gallery and 5,000 queries.

Question answering (WorldSense, DailyOmni). We attach the selector to Qwen2.5-Omni-7B and report accuracy against tokens, latency and peak memory. DailyOmni has no reported result on Lacuna, so we establish the full-token and OmniSIFT numbers ourselves.

Timing test. OmniCVR notes that captioning loses temporal precision. We construct probe questions from DailyOmni and WorldSense clips that ask when a sound occurs, and compare timestamped per-segment captions against whole-clip captions.

Decisive checks. Three results settle the proposal:

  1. The learned selector's Pareto curve dominates both heuristic gates on the integrated split and on WorldSense.
  2. Per-segment routing beats per-clip routing at equal latency.
  3. Caption decisions correlate with the measured per-segment gap score.

If the first check fails, the idea reduces to a heuristic and we will report it as such.