Audio-visual segmentation asks a model to outline, pixel by pixel, the objects in a video that are making sound. Hand-drawn masks are expensive, so weakly supervised versions train only from clip-level category tags. They work tolerably when one object is sounding. They break down when several objects sound at once, or when a silent object of a plausible class sits next to the one that is sounding. Current systems match pixels to audio through short-window (10–30 ms) spectral embeddings such as VGGish. Those embeddings say what is sounding but carry almost nothing about how the sound unfolds over hundreds of milliseconds. This proposal uses that slower structure directly. A strumming hand, moving lips, or a drumstick produces sound whose loudness, within each frequency band, rises and falls at 1–16 Hz. The visible motion of that same object rises and falls at the same rates. Measuring how well each image patch's motion rhythm matches each audio band's loudness rhythm gives a localization signal per region and per source. It costs no labels and can tell apart two co-occurring candidates that semantic embeddings cannot separate.
The idea in two sentences: speech research has long argued that 10–30 ms frames are blind to the slow modulations that carry syllables and rhythm, and that long-window sub-band envelopes recover them. We take those envelopes out of speech recognition and correlate them with per-patch visual motion to produce pseudo-masks that are seeded per source, which is exactly where weakly supervised audio-visual segmentation currently fails.
Core Claim
Our claim is that, in the 1–16 Hz modulation band, the coherence between long-window sub-band audio envelopes and per-patch visual motion envelopes is a strong spatial cue under weak supervision. We expect it to matter most in multi-source and co-occurrence scenes. When added to a weakly supervised audio-visual pipeline, it should raise pseudo-mask and final mask quality on AVSBench MS3 well beyond what a stronger semantic audio encoder or a broadband audio–motion correlation achieves. It should also leave single-source performance roughly unchanged.
The motivation comes from two places. On the audio side, Hilbert envelope based spectro-temporal features for phoneme recognition in telephone speech uses Frequency Domain Linear Prediction (FDLP), which fits an all-pole model to the temporal envelope of each frequency band over windows of hundreds of milliseconds. It shows that these envelopes survive channel distortion that ruins short-frame features. Tandem representations of spectral envelope and modulation frequency features for ASR extracts modulation spectra from 200 ms envelope segments and reports 27.5% phoneme error rate on TIMIT. Far-Field Speech Recognition Using Multivariate Autoregressive Models argues that 1–16 Hz envelope trajectories stay stable under reverberation. Modulation spectrogram features for improved speaker diarization uses modulation features to distinguish voices.
Together these papers supply our audio representation, but every one of them is audio-only. None asks whether those rhythms are visible in pixels. On the segmentation side, Weakly-Supervised Audio-Visual Segmentation (WS-AVS) motivates the work. It aligns multi-scale image regions with a VGGish audio embedding through multiple-instance contrastive learning, reaches 34.13% mIoU on AVSBench S4, and explicitly lists overlapping multi-source scenes and coarse boundaries as unsolved.
What Is Different From the Closest Work
The obvious combination is to swap WS-AVS's VGGish front end for FDLP or modulation features. We do not propose that. A better audio embedding still yields one global vector that is compared to regions by semantic similarity. It cannot tell which of two guitars is playing, or whether the speaking person is the left or right one.
Our cue is relational and temporal. It is computed between each patch's own motion time series and each audio band, so it needs no category knowledge at all. This also separates us from the frequency-domain segmentation methods. Frequency-Domain Decomposition and Recomposition for Robust Audio-Visual Segmentation (FAVS) applies 2D FFTs to short-term feature maps and treats audio high frequencies as noise. It is fully supervised and never models envelope rhythm. FFR: Frequency Feature Rectification for Weakly Supervised Semantic Segmentation analyzes spatial frequencies of ViT tokens in still images and has no time axis.
Annotation-Free Audio-Visual Segmentation (SAMA-AVS) pairs static images with unrelated audio clips. By construction it cannot learn synchrony, and its authors note the resulting multi-source weakness. We reuse two things from the WSSS literature: affinity-based propagation from seeds, from Weakly Supervised Learning of Instance Segmentation With Inter-Pixel Relations (IRNet), and the contrastive alignment of WS-AVS as the semantic stream.
Research Program
The method has four parts: an audio side, a visual side, a coherence score that links them, and an assembly step that turns scores into masks.
Audio side. For each clip we compute sub-band temporal envelopes $e_b(t)$ for bands $b$ over sliding windows of about one second. The primary version uses FDLP with gain normalization, following Hilbert envelope based spectro-temporal features for phoneme recognition in telephone speech. Because the file flags FDLP's computational overhead, a cheaper version uses squared Hilbert envelopes of a mel-spaced filterbank. Comparing the two is itself an ablation.
Visual side. For each patch $p$ we form a motion envelope $m_p(t)$ from optical-flow magnitude, or from frame-to-frame change in frozen ViT patch features, sampled at the video frame rate. At 25–30 fps the visual Nyquist limit is about 12–15 Hz. That covers most of the target band, but it caps what can be matched, and we report results restricted to 1–12 Hz as well.
Coherence score. The core quantity is the magnitude-squared coherence between patch motion and band envelope:
$$C_{p,b}(f) = \frac{|S_{m_p e_b}(f)|^2}{S_{m_p m_p}(f)\, S_{e_b e_b}(f)},$$
Here $S_{xy}(f)$ is the cross-spectral density of signals $x$ and $y$ at modulation frequency $f$, estimated by Welch averaging within the window. Coherence ignores any fixed time lag between sound and motion, which makes it tolerant to small audio-video offsets. It is also scale-free, so a loud source does not dominate a quiet one.
Separating sources. To handle multiple sources, we factorize the band-by-time envelope matrix with non-negative matrix factorization into $K$ temporal components, each with its own band profile. Each component's activation time course is scored against every patch with the same coherence measure. The result is $K$ heatmaps, one per hypothesized source. Concurrent sounds that collide in the instantaneous spectrum often have different rhythms, which is the separability that the diarization and deepfake work exploit (Characterizing the temporal dynamics of universal speech representations for generalizable deepfake detection finds distinctive 7–12 Hz modulation artifacts in synthetic speech).
Assembling masks. Component heatmaps are matched to the clip's weak tags through the WS-AVS-style contrastive class maps, by intersection and Hungarian assignment. Patches whose maximum coherence falls below a threshold fall back to the semantic stream alone. This fallback is how we handle sound sources with no visible motion, such as loudspeakers and engines, which this cue cannot see. The surviving high-confidence regions become seeds. Following IRNet, we propagate them with learned inter-pixel affinities into pseudo-masks and train a student segmentation network on the result.
The analysis component is as important as the method. We will measure, per AVSBench category, how often sounding objects show significant coherence. This tells the community which kinds of sound sources the cue covers and which it does not.
Baselines
The naive reference point is the trivial baseline reported in WS-AVS, at 12.63% mIoU on S4. A second naive reference treats every moving patch as sounding: motion saliency with no audio. If that does nearly as well as our method, the audio side is contributing nothing.
The strongest weakly supervised competitor is WS-AVS itself, at 34.13% mIoU on S4 as reported in Weakly-Supervised Audio-Visual Segmentation. Its predecessor C²AM reports 30.87% in the same paper. For MS3 under weak supervision, we found no reported result on Lacuna for WS-AVS, so we will reproduce WS-AVS on MS3 ourselves. Our method is built on top of a reproduced WS-AVS with identical visual backbone, training clips, tag supervision, epochs, and hyperparameter search budget. That way any gain is attributable to the coherence stream. Beating it means a statistically reliable improvement over three seeds on MS3, with no loss larger than one point on S4.
A third baseline, strong and close to our own components, is WS-AVS with its VGGish encoder replaced by an FDLP or modulation-spectrum encoder fed the same one-second window. This isolates whether the gain comes from better audio features (Gap 1 in the research file) or from patch-level coherence (our claim).
Two ablations isolate the key ingredients. The first replaces sub-band envelopes with a single broadband loudness envelope; the second removes the source factorization.
As upper references, not competitors, we quote fully supervised results. Audio-Visual Segmentation reports 78.74% mIoU on S4 and 54.00% on MS3, and FAVS reports 74.4 Mean Jaccard on MS3. SAMA-AVS reports 81.53 mIoU on S4. It uses dense masks borrowed from image datasets, so it is a different supervision regime. We still run it on MS3 for comparison, using its released recipe.
Why not just…?
Why not just use a stronger or longer-context audio encoder in WS-AVS? This is the real threat for scenes where the sounding objects belong to different categories. Semantic matching may already suffice there, and we expect our gains to be small on such clips. Our claim is limited to same-category and co-occurring candidates, where any global audio vector is spatially uninformative. The FDLP-encoder baseline above, scored separately on a co-occurrence subset, settles this. If it matches our method on that subset, the coherence idea is not needed.
Why not just correlate overall loudness with motion? Broadband audio–visual synchrony is a simpler version of the same idea, and it may capture much of the single-source gain. In a mixture, though, the broadband envelope is a sum of sources. Every sounding object then correlates partially with it, which blurs the masks across objects, exactly the failure WS-AVS reports. The broadband ablation tests whether sub-band envelopes and factorization add anything. If they do not, we will report the simpler cue as the finding.
Why not just use SAM with audio adapters, or purely visual CAM refinement? SAMA-AVS shows that a foundation model with VGGish adapters is very strong on S4. However, it trains on static image–audio pairs and cannot observe synchrony. Purely visual refinement methods like IRNet and Causal Intervention for Weakly-Supervised Semantic Segmentation cannot tell a silent dog from a barking one. We test both on the co-occurrence subset. We also note that SAM can serve as our boundary refiner in place of IRNet affinities. That makes the cue complementary to foundation models rather than a replacement for them.
Why not just apply a 2D FFT to the spectrogram as FAVS does? A 2D FFT of a short-term spectrogram treats the time-frequency plane as an image. It has no link to any particular pixel's motion, and FAVS needs dense masks to train its routing. We include a variant that substitutes FAVS-style decomposed audio features into the semantic stream, as a further check.
Proposed Experiments
All main experiments use AVSBench S4, MS3, and AVSS, reporting mIoU or Mean Jaccard and a boundary F-score. The file reports AVSBench as 5,356 videos in one source and 4,932 in another, so we will fix and publish the exact split.
The first decisive check is pseudo-mask quality before any student training, measured against ground truth on annotated training frames. The coherence seeds must beat WS-AVS contrastive maps on MS3, or the method has no foundation.
The second check uses a co-occurrence subset of MS3 and AVSS. These are frames in which an annotated sounding object shares its category, or its usual visual context, with a visible non-sounding object. The predicted signature is a large gain here and a small gain elsewhere.
The third check is a controlled mixture test. We place two real AVSBench clips side by side and mix their audio. Ground-truth masks are known, and we can vary the rhythmic similarity of the two sources to see where factorization fails.
The fourth check tests robustness to reverberation. We convolve the test audio with simulated room impulse responses and check whether FDLP envelopes degrade more slowly than VGGish matching does, as the speech literature predicts.
As an audio-only diagnostic, we check that the envelope components localize events in time on DCASE2018 Task 4. There, Towards Duration Robust Weakly Supervised Sound Event Detection reports 32.52% event-based F1 and a 4.02% long-versus-short gap. Long windows risk smearing short events, so a widening of that gap would warn us that the one-second window must be multi-resolution.
The proposal fails cleanly in either of two cases: coherence seeds do not beat contrastive seeds on MS3 pseudo-masks, or the broadband ablation matches the full method on the mixture test.
References
- Hilbert envelope based spectro-temporal features for phoneme recognition in telephone speech
- Tandem representations of spectral envelope and modulation frequency features for ASR
- Far-Field Speech Recognition Using Multivariate Autoregressive Models
- Modulation spectrogram features for improved speaker diarization
- Weakly-Supervised Audio-Visual Segmentation
- Frequency-Domain Decomposition and Recomposition for Robust Audio-Visual Segmentation
- FFR: Frequency Feature Rectification for Weakly Supervised Semantic Segmentation
- Annotation-free Audio-Visual Segmentation
- Weakly Supervised Learning of Instance Segmentation With Inter-Pixel Relations
- Characterizing the Temporal Dynamics of Universal Speech Representations for Generalizable Deepfake Detection
- Audio-Visual Segmentation
- Causal Intervention for Weakly-Supervised Semantic Segmentation
- Towards Duration Robust Weakly Supervised Sound Event Detection