Audio-visual segmentation means outlining, pixel by pixel, the object in a video that is making the sound you hear: the violin that is playing rather than the silent one next to it. A natural way to get sharper outlines is iterative mask refinement, where the network repeatedly takes its own current mask back as input and corrects it. In video this has a known weakness. Nothing reliable says whether a given correction made the mask better or worse. Without a ground-truth check, the loop can lock onto its own mistakes, and each frame drifts differently, so outlines flicker. We propose to use estimated scene depth as that missing check. A correct mask of a sounding object usually separates pixels at one distance from pixels at another, so a refinement step should be kept only if it makes the inside and outside of the mask more distinct in depth. A second check requires that the step does not weaken the mask's dependence on the actual audio.
The obstacle is that single-image depth models are unstable over time. Their output is only defined up to a scale and offset, and these change from frame to frame. A depth-based check built on raw per-frame depth would therefore inherit that flicker. The proposal treats depth stabilization as a requirement of the gate, not as a goal in itself. Each frame's depth is aligned to the previous frame on background pixels with a two-parameter least-squares fit before the gate reads it. Our central question is whether this cheap alignment, or a temporally consistent video depth model, turns depth into a usable label-free acceptance test for mask refinement.
Core Claim
Iterative refinement of sounding-object masks in video becomes reliably beneficial when each update is accepted or rejected by two label-free tests. The first is a temporally aligned depth-separation test. The second is an audio-dependence test. Without temporal alignment, the depth test is corrupted by per-frame scale jitter and should help little or even hurt.
The closest existing work handles the pieces separately. Audio-Visual Segmentation via Depth-Guided Collaborative Modeling (DGCM-AVS) shows that depth from Depth Anything V2 helps separate sounding objects from silent distractors and reduces boundary leakage. However, it uses depth as a fixed per-frame feature in a single feedforward pass. It does not correct for temporal jitter and does not refine iteratively. SAM-REF introduces a gate that rejects refinement steps which lower IoU, but that gate depends on supervisory cues or confidence heuristics. We replace it with a geometric signal that needs no labels. Audio-sync Video Instance Editing with Granularity-Aware Mask Refiner refines masks iteratively in an audio-driven video setting, but it works purely in 2D appearance space. Layered Depth Refinement with Mask Guidance runs in the opposite direction: it uses a fixed mask to sharpen depth. We borrow its negative result and use the dependency the other way round, letting depth judge the mask.
Research Program
The base system is an existing audio-visual segmentation model that produces an initial mask $M^{(0)}_t$ for frame $t$. Over that model sits a lightweight refinement decoder that takes the frame, the current mask and the model's audio-conditioned features and proposes $M^{(k+1)}_t$. The refiner follows the mask-as-input-channel design of Iterative Instance Segmentation. Only this small decoder is re-run across iterations, which limits the cost multiplier that makes iterative refinement expensive in video.
Depth $D_t$ comes from a frozen monocular model. Before the gate uses it, we remove the per-frame scale and offset jitter by fitting
$$(s_t, b_t) = \arg\min_{s,b} \sum_{p \in B_t} \left(s\,D_t(p) + b - \tilde D_{t-1}(p)\right)^2,$$
where $B_t$ is the set of pixels outside a dilated version of the current mask (assumed mostly static background), $\tilde D_{t-1}$ is the previous frame's aligned depth, and $\tilde D_t = s_t D_t + b_t$. This addresses exactly the frame-to-frame affine drift that Video Depth Anything and Wang et al. identify as the source of depth flicker. The fit is closed-form and costs almost nothing.
The depth test scores a mask by how well it splits aligned depth in a thin band around its boundary:
$$G(M) = \frac{|\mu_{\text{in}} - \mu_{\text{out}}|}{\sigma_{\text{in}} + \sigma_{\text{out}}},$$
with $\mu$ and $\sigma$ the mean and spread of $\tilde D_t$ just inside and just outside the boundary.
The audio test scores a mask by $A(M)$, the base model's mean mask logit inside $M$ under the true audio minus the same quantity under audio taken from another clip. This guards against a failure documented in AViNet: audio-visual networks often learn to ignore audio entirely. A refinement step that pulled the mask toward a visually prominent but silent object would lower $A$.
An update is accepted if $G$ does not decrease and $A$ does not fall below a margin. If neither test is informative, the previous mask is kept. A typical uninformative case is a sounding object at nearly the same depth as its surroundings, such as a guitar held against a body. In that case $G$ of the initial mask falls below a threshold, and the gate defers instead of guessing.
We will run three analyses. First, a controlled study injects synthetic scale-and-offset jitter of increasing size into per-frame depth and measures how gate decisions and segmentation quality degrade with and without alignment. Second, we compare the closed-form alignment against simply substituting temporally consistent depth from Video Depth Anything or StableDPT. These video models smooth over time and may soften depth at object boundaries, which is exactly where the gate looks. Third, we stratify results by the initial depth contrast of the target object to find where the depth signal stops being useful.
Baselines
The naive reference is the base audio-visual segmentation model with no refinement. On AVSBench, the original Audio-Visual Segmentation paper reports 78.74% mIoU on the single-source subset (S4) and 54.00% mIoU on the multi-source subset (MS3). These are the reference numbers for the original architecture. We will rerun that model ourselves so that every arm shares the same initialization.
The strongest existing competitor is DGCM-AVS, because it already exploits depth for this exact task. The file reports only relative gains for it: a 10.2% relative Jaccard gain on the semantic subset (AVSS) and a 7.8% F-score gain on MS3 with a ResNet-50 backbone. No absolute mIoU for DGCM-AVS was found on Lacuna, so we will reproduce it. We also attach our gated refiner on top of it. Beating it means a statistically reliable gain in mIoU and F-score on MS3 and AVSS at matched backbone, with lower mask flicker, and with added compute limited to the refiner decoder.
The ablations isolate the key ingredient. One arm runs refinement for a fixed number of steps with no gate. One uses the base model's own confidence as the gate, which is the label-free version of the SAM-REF idea. One uses the depth gate on raw per-frame depth. One uses the depth gate on video-model depth without our alignment. One uses the depth gate without the audio test. All arms use the same training data and refiner size, and each gets the same hyperparameter search budget for its thresholds, tuned on a held-out split.
Why not just…?
Why not just swap Depth Anything V2 for Video Depth Anything inside DGCM-AVS? This is the most serious threat, and it is a direct arm of the study. If the swap alone matches the gated refiner, the iterative component adds nothing and we will report that. We expect it to fall short for two reasons. DGCM-AVS remains a single feedforward pass with no mechanism to undo a wrong mask. In addition, temporally regularized depth tends to blur boundaries during fast motion, a known limitation of temporal-gradient-style training, and boundaries are where segmentation errors live.
Why not just iterate more and let the refiner learn to stop? Ungated feedback loops can treat their own false positives as anchors. This is why SAM-REF needed a gate. The ungated arm and the confidence-gated arm measure directly how much of the gain comes from the depth and audio tests rather than from extra iterations.
Why not just feed depth into the refiner as an extra input channel? Layered Depth Refinement with Mask Guidance found that networks given a guidance channel often fail to respect its hard edges. DGCM-AVS found that naive concatenation of depth with appearance underperforms frequency-separated fusion. We include a concatenation arm anyway. Using depth as an external test rather than an input is the design choice under examination.
Why not just smooth masks over time, or propagate them with a video segmentation memory as in SAM2-LOVE? Smoothing reduces flicker but cannot fix a mask that is consistently wrong, such as a silent distractor. Memory propagation carries errors forward. We will run a SAM2-propagation arm with identical initial masks. SAM2-LOVE's reported 58.5% J&F on Ref-AVS is the reference point for that setting.
Proposed Experiments
The main evaluation is on AVSBench S4, MS3 and AVSS, reporting mIoU (Jaccard) and F-score on annotated frames. Temporal stability is measured on all frames as boundary jitter: the change in mask outline between consecutive frames, after discounting change predicted by ground-truth motion where annotations allow. A secondary evaluation on Ref-AVS tests longer temporal context, using J&F.
The decisive checks are three predictions, each of which can fail. First, the gate on raw per-frame depth should help less than the gate on aligned depth, and the gap should widen as injected jitter grows; if there is no gap, temporal depth inconsistency does not matter for this use. Second, removing the audio test should increase the drop in segmentation quality when audio is shuffled at test time; if it does not, the audio test is not doing work. Third, on MS3, which has multiple sources and the most silent-distractor confusion, the gated refiner should beat both DGCM-AVS with Video Depth Anything swapped in and the ungated refiner. Gains should concentrate in the high-depth-contrast stratum and vanish, without regressions, in the low-contrast stratum where the gate defers.
References
- Audio-Visual Segmentation via Depth-Guided Collaborative Modeling
- SAM-REF: Introducing Image-Prompt Synergy during Interaction for Detail Enhancement in the Segment Anything Model
- Audio-sync Video Instance Editing with Granularity-Aware Mask Refiner
- Layered Depth Refinement with Mask Guidance
- Iterative Instance Segmentation
- Video Depth Anything: Consistent Depth Estimation for Super-Long Videos
- BiFuse: Monocular 360 Depth Estimation via Bi-Projection Fusion
- AViNet: Diving Deep into Audio-Visual Saliency Prediction
- StableDPT: Temporal Stable Monocular Video Depth Estimation
- Audio-Visual Segmentation
- SAM2-LOVE: Segment Anything Model 2 in Language-aided Audio-Visual Scenes