Deciding What Sounds at Coarse Scale: Retrospective Audio Grounding for Audio-Visual Segmentation

Audio-visual segmentation is the task of outlining, pixel by pixel, the object in a video that is making the sound. Current systems inject audio into the image features at every resolution of the decoder, the part of the network that upsamples a coarse feature map back to a full-resolution mask. The dense version compares every pixel with the audio at every scale. It is slow, and it tends to "bleed": if three people sit together and one speaks, every pixel that looks like a person matches speech audio, so all three get segmented. The cheap version replaces this with one global audio vector that reweights feature channels. It is fast, but by its authors' own account it loses thin structures and cannot tell identical sources apart.

We propose splitting the job by scale. Only the coarsest level of the decoder decides which object is sounding. At that level each cell sees a whole object together with its motion over time, and the grid is small enough that audio attention costs almost nothing. The finer levels never look at the audio at all. Instead, each high-resolution pixel queries a small window of the coarse, audio-informed map beneath it and asks "which sounding-or-silent object do I belong to?" This is a retrospective, fine-to-coarse lookup of the kind that sharpened boundaries in point-cloud and ImageNet backbones. The bet is that bleeding and cost both come from letting fine-scale, texture-driven features match audio directly. If that is right, routing all audio evidence through one coarse decision should give sharper masks, fewer silent-distractor false positives, and roughly linear cost at the same time.

Core Claim

Cross-modal attention belongs at exactly one scale: the coarsest one, where features encode object instances rather than textures. Resolution should be recovered by gated local attention from fine visual features into that coarse audio-grounded map. The resulting decoder, which we call Coarse-Anchored Retrospective Decoding (CARD), should beat channel gating on fine structure and multi-instance disambiguation. It should match dense multi-scale fusion on accuracy at a fraction of the cost. The contribution is not "add cross-attention to a pyramid", which several systems already do. It is the testable claim that where audio enters the hierarchy determines distractor bleeding, together with a decoder built on that claim.

Relation to Prior Work

Three audio-visual segmentation systems define the design space, and all three put audio into several decoder levels.

Audio-Visual Segmentation introduced the task, the AVSBench benchmark and the TPAVI module. TPAVI computes dense pixel-to-audio attention independently at multiple pyramid scales. It motivates this work, because the file records that it suffers exactly the spatial distractor ambiguity and flicker we target. AVSegFormer adds an audio-guided channel mixer over the multi-scale pyramid and a decoder with 300 audio-conditioned learnable queries. It is the accuracy reference, but it carries roughly seven times the parameters of mobile models. LightAVSeg argues that dense attention is unnecessary and uses a refined global audio state to gate visual channels as the image is upsampled. It constrains our design, since any spatial audio attention must stay cheap. Its admitted failures on thin limbs and identical concurrent sources are the gap we aim at.

None of the three lets fine decoder levels query coarse levels. CARD differs from all of them by removing audio from fine levels entirely.

RAVS attacks bleeding directly. It clusters visual features with density-peaks clustering, contrasts clusters that respond to the audio against those that do not, and adds a Dirichlet uncertainty head for transition frames. We share its diagnosis but not its mechanism. RAVS groups pixels with an external, hyperparameter-sensitive clustering step on top of fixed features. CARD obtains object-level grouping for free from the coarse level of the decoder and propagates it by learned attention.

The mechanism comes from two unimodal lines of work. Retro-FPN lets fine point-cloud levels pull semantics from coarser levels through local cross-attention over K nearest neighbours, combined with a GRU-like semantic gate. It gains up to 4.3 mIoU on S3DIS for 0.08M parameters. Cross-Layer Retrospective Retrieving via Layer Attention shows that later layers querying earlier ones beats blind skip connections on COCO. We reuse the local retrospective query and the gate. What we add is that the coarse map being queried carries an audio-derived sounding decision, so the lookup transfers cross-modal identity rather than only semantics.

The gate follows Gated Cross Attention for drug-target interaction, which found that ungated cross-attention washes out the original features over training and that a learned gate with Sparsemax preserves them. GateFusion supports multi-depth gated fusion and a penalty on visual false positives during silence. It operates at the level of whole bounding boxes for active speaker detection with no spatial pyramid, so it does not answer the dense question. We borrow its silence penalty as an optional loss.

Research Program

The decoder has three pieces. We build it on the same encoders as the baselines: PVTv2 or SeaFormer for vision, and VGGish for audio.

Coarse audio grounding. At the coarsest level (stride 32), visual tokens from all frames of the clip attend to audio tokens, and audio attends back. This yields audio-conditioned coarse features. A one-channel "sounding" logit is supervised by the downsampled ground-truth mask. With a 224-pixel input this is about 49 tokens per frame, so dense bidirectional attention is affordable even on mobile hardware. The attention includes the temporal axis, because a mouth moving in sync with speech is the cue that separates a speaking person from a silent one. That cue is visible only across frames.

Retrospective local lookup. At each finer level $l$, a fine feature $f_i$ at pixel $i$ forms a query. Keys and values come from the $k \times k$ window $\mathcal{N}(i)$ of coarse audio-grounded features around $i$'s projected location. The update is

$$ \tilde f_i = g_i \odot \sum_{j \in \mathcal{N}(i)} \mathrm{sparsemax}_j\!\left(\frac{q(f_i)^\top k(c_j)}{\sqrt d}\right) v(c_j) + (1-g_i)\odot f_i, $$

where $c_j$ are coarse features, $q, k, v$ are linear maps, $d$ is the key dimension, and $g_i = \sigma(W[f_i; \bar c_i])$ is a gate computed from the fine feature and the window mean $\bar c_i$. The cost per level is linear in pixel count times $k^2$. Sparsemax lets a boundary pixel commit to one coarse cell, which keeps it from averaging a sounding object with its silent neighbour. We fix $k$ (3 or 5) rather than adapt it, because image grids have uniform density and the variable-density concern raised against Retro-FPN's fixed K does not apply here.

Diagnostic analysis. Before building anything, we test the premise on released TPAVI and AVSegFormer checkpoints. We measure, separately at each pyramid scale, how much of the audio-attention mass falls on silent instances of the sounding class. If bleeding is not concentrated at fine scales, the core claim is wrong, and we report that rather than proceed on it.

Baselines

Naive reference. The do-nothing reference is a visual-only decoder on the same backbone, trained on the same masks with no audio. It reveals how much of each benchmark is solvable by saliency alone. It also sets the floor for silent-distractor false positives, since a visual-only model segments every instance of the salient class.

Accuracy tier: AVSegFormer. AVSegFormer is the strongest existing method. It reports 82.06 mIoU and 89.9 F-score on AVSBench S4 and 36.66 mIoU and 42.0 F-score on AVSS with PVTv2. For MS3 the file gives only its margin over iGAN (+15.47 mIoU), not an absolute PVTv2 value, so we will rerun it. The file lists 49.5 mIoU on MS3 for the ResNet-50 variant. We will train it from its released configuration with our data pipeline and the same epochs and augmentation, and report both its published numbers and our reproduction. CARD beats it if it matches S4 mIoU within reproduction noise while exceeding it on MS3, AVSS and VPO-MSMI, at clearly lower parameters and latency.

Efficient tier: LightAVSeg. LightAVSeg reports 50.4 mIoU on MS3 with 20.5M parameters and 163.4 ms on a Snapdragon 8 Elite CPU. We will run CARD on the same SeaFormer backbone and measure latency on the same class of device. Beating it means higher MS3 and AVSS mIoU at no more than about 20% extra latency, plus a measurable gain on thin-structure boundary F-score. Thin structures are the failure its authors concede.

Original dense baseline: TPAVI. Audio-Visual Segmentation reports 78.74 mIoU on S4 and 54.00 on MS3. It is the direct test of "audio at every scale versus audio at one scale" under a matched backbone.

Distractor tier: RAVS. RAVS is the specialist for distractor robustness. The file gives only relative gains for it (+4.2% J&F on AVS-Semantic, +11.5% on VPO-MSMI) and no absolute value, so we will rerun its released code on VPO-MSMI and AVSS with the same backbone.

Ablations. Four ablations isolate the key ingredient.

  • Audio-agnostic lookup. The retrospective lookup queries a coarse map that has had no audio attention, and audio instead enters by LightAVSeg-style channel gating at every level.
  • Coarse attention plus FPN. Audio attends only at the coarse level, but the result is propagated by plain FPN upsampling and addition, with no lookup.
  • Audio at all scales. Audio attention is also added at fine levels, which tests whether fine-scale audio hurts.
  • Softmax and no gate. Sparsemax is replaced by softmax, and the gate is removed.

Why not just…?

Why not just bolt Retro-FPN onto an existing audio-visual decoder? This is the most serious threat. If retrospective lookup improves boundaries for purely visual reasons, CARD's gains could be ordinary boundary sharpening with nothing to do with audio. The audio-agnostic-lookup ablation settles it: same lookup, same cost, but audio enters through channel gating. We predict that this variant closes most of the boundary-F gap to CARD but little of the silent-distractor gap. If it closes both, the scale-placement claim fails, and the honest contribution shrinks to "Retro-FPN transfers to audio-visual segmentation."

Why not just run TPAVI-style attention at the coarsest level and upsample with a standard FPN? This is cheaper still and may capture most of the disambiguation benefit. The evidence against blind lateral connections comes from other domains. Retro-FPN showed that standard FPNs leave semantic ambiguity at boundaries, and MRLA showed layer attention beating skip connections by 3 to 4 AP on COCO. Neither result was on audio-visual data, so the coarse-attention-plus-FPN ablation is run directly. We expect it to match CARD on per-object disambiguation but lose on boundary F and thin structures, especially where a sounding and a silent object touch.

Why not just scale up AVSegFormer's query decoder or backbone? Its learnable queries already disentangle multiple sources, and more capacity might absorb the bleeding problem. But the file notes that its query efficacy drops in single-source scenes, and its cost is already the obstacle to deployment. We compare at matched FLOPs and also report its full-size numbers. If a bigger AVSegFormer dominates CARD on VPO-MSMI at similar cost, our efficiency argument stands but the accuracy argument does not, and we will say so.

Why not just use RAVS's clustering? Density-peaks clustering is an external, hyperparameter-sensitive step that adds substantial compute, according to the file. CARD is meant to obtain the same grouping from the coarse decoder level at negligible cost. Running both on VPO-MSMI under a matched backbone shows directly whether learned coarse grouping matches explicit clustering.

Proposed Experiments

The benchmarks are AVSBench S4, MS3 and AVSS, plus VPO-MSMI, the synthetic benchmark with multiple same-class instances. We report mIoU and F-score (J&F on AVSS where the baselines use it), boundary F-score in a thin band around mask edges, parameters, FLOPs and mobile latency. We add a silent-distractor false-positive rate: predicted mask area falling on ground-truth instances of the sounding class that are not sounding. This is computed on VPO-MSMI, whose synthetic construction provides instance identity, and on MS3 frames where instance masks can be derived. We also add a temporal flicker score: the frame-to-frame change in predicted area at frames where the ground-truth sounding state does not change. Every model is trained with three seeds, and differences are reported with their spread.

Three checks are decisive.

Per-scale bleeding diagnostic. On TPAVI and AVSegFormer, attention mass on silent distractors should rise at finer scales. If it does not, the premise fails.

Audio-agnostic-lookup ablation. CARD should cut the silent-distractor false-positive rate substantially relative to both this ablation and LightAVSeg, while matching or beating them on mIoU.

Audio-at-all-scales ablation. Adding fine-scale audio attention back into CARD should increase distractor false positives despite the extra capacity. This is the most direct evidence that the placement of audio, not the amount of attention, is what matters.

If all three hold at the efficient tier's latency budget, the result is a cheap decoder and an explanation of why dense multi-scale fusion bleeds. If only the accuracy gains hold, the work still yields a sound ablation map of where audio should enter a segmentation pyramid.