Letting the Separated Sound Move the Box: Audio-Guided, Per-Source Box Refinement for Audio-Visual Separation

Systems that separate a mixed soundtrack into the sound of each visible instrument first run an object detector. They then treat its boxes as fixed: each box is cropped, turned into a query, and used to pull that instrument's sound out of the mix. If a box is loose, clipped, sitting on a silent instrument, or sitting on the wrong one of two identical violins, nothing downstream can fix it, and separation degrades. Object detection solved a similar fragility years ago with iterative box refinement, in which a box is nudged a little at each decoder layer rather than predicted in one shot. Separation has a signal that detection lacks for deciding where to nudge it: the partially separated sound itself. We propose a separator in which each source's box is refined over a few steps, driven by that source's current separated audio. Visual features are re-cropped from the corrected box at each step, and each source stops refining independently once further steps stop helping.

The cost constraint is central to the design. Across vision, language and multimodal models, almost all of the benefit of iterative refinement arrives in the first one or two passes, while cost grows linearly with each pass. We therefore design refinement so that the backbone runs once, each extra step touches only one source's box, and easy sources exit after a single step. Compute then goes to the minority of hard sources: duets of the same instrument, occluded players, and silent distractors. Our claim is that this produces a better quality-versus-latency trade-off than either a stronger static detector or a deeper fixed decoder. We also commit to reporting it honestly if it does not.

Core Claim

In query-based audio-visual separation, a source's spatial support should be a variable that the audio can move, not an input. Concretely, one round of audio-conditioned box correction with re-cropped visual features should raise separation quality on hard sources (same-class duets, silent distractors, poorly localized detections) by more than spending the same compute on extra mask-decoding layers over fixed boxes. Per-source early stopping should then recover most of the latency that a fixed number of refinement steps would cost.

Relation to Prior Work

The closest systems take the visual side as frozen. iQuery initializes "visually named" queries from detector boxes and decodes spectrogram masks through motion, audio and inter-query attention. It reports 11.17 dB SDR on MUSIC, but its decoder layers refine only the acoustic mask and never revisit the box. Cyclic Co-Learning (CCoL) is the nearest conceptual relative. It alternates between using visual grounding to guide separation and using separated audio to improve grounding, and it shows that static proposals on silent objects push earlier separators to negative SDR. Its boxes, however, come from a frozen Faster R-CNN, the cycle is a training-time curriculum rather than an inference procedure, and it fails on two instances of the same class. ContAV-Sep inherits the same dependence on an external detector, Detic. These three papers motivate the proposal: each documents failures that originate in fixed boxes.

The refinement mechanism itself comes from detection. DAB-DETR represents queries as 4D anchor boxes updated by residual offsets at each layer, and uses box width and height to shape attention. Deformable DETR introduced layer-wise box refinement with sparse sampling around reference points. FA-RPN shows that re-cropping features from corrected box coordinates at test time improves face detection at roughly 50 ms per image. D-FINE adds distillation from deep to shallow layers, which we reuse so that the first step is already good. All four are unimodal: their refinement is driven by visual evidence alone. Ours is driven by the separated audio of the specific source being refined.

The cost literature constrains the design. Hwang et al.'s lung segmentation work, DeFusionNET (gains flat after three iterations) and Deliberation Networks (little beyond two passes) all show steep diminishing returns. We therefore budget for one to three steps, not sixteen. iRe2f shows that much of the throughput loss of iterative refiners comes from re-encoding everything at each step, and recovers speed with lightweight residual updates; we adopt that principle for audio-visual separation by never re-running the backbone.

The closest work combining iteration and audio-visual separation is the Bottleneck Iterative Network (BIN). It applies a light fusion block a fixed $R$ times and cuts GPU inference time by 80% relative to RTFS-Net. BIN targets speech with face video, has no spatial localization, and uses a fixed iteration count. We differ on all three points.

Method

A shared video backbone and audio encoder run once per clip. Each candidate source $i$ starts from a detector box $b_i^{(0)}$. At step $t$, the model crops visual and motion features inside $b_i^{(t)}$, decodes a spectrogram mask $m_i^{(t)}$ for that source, and encodes the masked spectrogram into a separated-audio embedding $a_i^{(t)}$. A small refinement head then predicts a box offset and a stop probability. To do this, it attends from $a_i^{(t)}$ to visual features in a window around the current box: the region where the box is, plus a margin where it might need to go. The update follows DAB-DETR's residual form:

$$b_i^{(t+1)} = b_i^{(t)} + \Delta b\big(a_i^{(t)},\, F_{\mathrm{win}}(b_i^{(t)})\big),$$

where $F_{\mathrm{win}}$ is the visual feature window around the box. The same head emits a sounding/silent score, so a box on a non-playing instrument can be suppressed rather than separated.

Because the backbone is shared, the cost per clip is approximately

$$C \approx C_{\mathrm{backbone}} + c_{\mathrm{step}} \sum_i K_i,$$

where $K_i$ is the number of steps source $i$ takes and $c_{\mathrm{step}}$ is the cost of one crop, decode and refine step. Adaptive stopping is useful only if $\sum_i K_i$ falls well below $N K_{\max}$ for $N$ sources while quality holds. The halting head is supervised directly: during training, its target at step $t$ is whether step $t+1$ improved that source's SDR. This avoids ponder-cost tuning.

Training uses the standard mix-and-separate recipe from The Sound of Pixels with one addition. Detector boxes are randomly jittered, shrunk, shifted onto a neighbouring same-class instance, or swapped with silent objects, so the refiner learns corrections that real detectors require. Final-step masks and boxes are distilled into step one, following D-FINE's deep-to-shallow distillation, so that sources which exit early lose little. We build on the official iQuery codebase. Its mask decoder becomes our per-step decoder, which keeps comparisons tight.

Research Program

The work has three parts. The first is the model: iQuery plus a refinement head, re-cropping, and a halting head. The second is an evaluation suite that isolates hard cases. On MUSIC we will build subsets from existing data: same-class duets, mixtures containing a visible silent instrument (following CCoL's silent-distractor protocol), and a controlled box-corruption sweep in which detector boxes are perturbed by known amounts. These subsets let us measure SDR as a function of initial box error. The third is a cost analysis. For every method we will plot SDR against measured wall-clock latency per clip and against FLOPs on the same GPU, and record the distribution of $K_i$ per source to show where compute actually goes.

Baselines

The naive reference point is iQuery exactly as released, with frozen detector boxes and no box updates, retrained under our data pipeline. The strongest published number in our file on MUSIC is iQuery's 11.17 dB SDR (iQuery). For silent-distractor scenes, CCoL reports 9.07 SDR and 75.9% grounding accuracy (CCoL). We will reproduce iQuery and train every variant with the same backbone, detector, epochs and hyperparameter search budget. Beating it means a statistically reliable SDR gain on the full MUSIC test set, together with clearly larger gains on the same-class and silent-distractor subsets, at a latency increase we report explicitly.

The strongest competitor on this specific question is a compute-matched deeper iQuery. It adds mask-decoder layers over fixed boxes until its FLOPs and latency match our model's average, which tests whether moving the box matters or whether extra decoding alone explains the gain.

CCoL serves as the baseline for audio feeding back into grounding. We compare on its silent-distractor setting and on grounding accuracy, measured as whether the refined box lands on the sounding instrument.

Ablations isolate the key ingredients. In the first, the refinement head sees only visual features and not $a_i^{(t)}$; if this matches the full model, the audio feedback is not doing the work. In the second, boxes are refined but features are not re-cropped, which tests whether FA-RPN-style re-cropping matters. The third compares fixed $K=1, 2, 3$ against adaptive stopping. The fourth removes the jitter augmentation.

Why not just…?

Why not just use a stronger static detector such as D-FINE upstream? This is a real threat for the box-corruption sweep, since better boxes shrink the error that refinement corrects. It is a weak answer to silent distractors and same-class duets, however. A purely visual detector cannot tell which of two violins is playing or whether a visible instrument is silent; CCoL documents negative SDR for static-proposal separators in the silent case. We will settle this by running our refiner and the iQuery baseline both on Detic boxes and on boxes from a stronger detector. If the gain disappears with the stronger detector on every subset, including same-class and silent ones, the proposal fails and we will say so.

Why not just add decoder layers or iterate the audio queries over fixed boxes? iQuery already refines masks across layers, and TransAVS shows that layered decoding with separation losses resists collapse onto the dominant source. The compute-matched deeper iQuery baseline answers this directly. Our prediction is that the advantage appears specifically when the initial box is wrong, which shows up as a widening gap in the box-corruption sweep. With accurate boxes, the two should be close.

Why not just run a fixed one or two refinement steps and skip adaptive stopping? Given the diminishing-returns evidence, fixed $K=2$ may capture nearly everything. Adaptive stopping earns its place only if most sources exit at step one while hard ones keep going, so that $\sum_i K_i$ falls well below $2N$ at equal SDR. If fixed $K=2$ matches adaptive stopping on both axes, we drop the halting head and report the simpler method as the result.

Why not just use pixel masks, as audio-visual segmentation models do? Masks such as those from TransAVS or QDFormer are richer spatially. However, a box gives a four-number state that the audio can update cheaply and use to re-crop features, which keeps $c_{\mathrm{step}}$ small. A mask-refinement variant is a reasonable follow-up, but it changes the cost profile we are testing.

Proposed Experiments and Decisive Checks

We will evaluate SDR, SIR and SAR on MUSIC and MUSIC-21, and run the AVE separation setup used by iQuery. There is no reported per-subset SDR on Lacuna for MUSIC-21 or AVE separation from iQuery, so we will establish those baselines ourselves. On MUSIC we additionally report grounding accuracy on silent-distractor mixtures and SDR on same-class duets.

Three outcomes would be decisive. First, in the box-corruption sweep, the refiner's SDR should degrade much more slowly with initial box error than iQuery's and than the compute-matched deeper iQuery's. Second, the audio-blind refinement ablation should recover only part of the gain on same-class duets and silent distractors, because only audio can resolve those cases. Third, on the latency–SDR plot, the adaptive model should sit above every fixed-$K$ and deeper-decoder point at equal latency, and its $K_i$ histogram should show extra steps concentrated on the hard subsets.

If the first check fails, refinement is not doing anything. If only the second fails, we have a visual box refiner bolted onto a separator, which is a far less interesting result. If only the third fails, the contribution reduces to one step of audio-guided correction, and that is what we will report.