Choosing Among Segmented Objects: Auditable Audio-Visual Direction Finding with Class-Agnostic Masks

Many systems combine a camera with a microphone array to work out where a sound is coming from. Most of them fail in two ways. They only "see" objects that a trained detector recognizes, and they merge sound and vision inside a neural network that nobody can inspect. On the STARSS23 benchmark of real spatial recordings, adding a person-centric object detector halves the angular error for human sounds. Overall error, though, barely moves, from 66.9° to 60.0°, because doors, taps and phones are invisible to the detector (STARSS23). We propose a different approach. A class-agnostic segmenter, a model like Segment Anything that outlines every object in an image without knowing what any of them are, supplies a short list of candidate source regions. With a 360° camera placed next to the array, each outlined region is a patch of directions on the sphere. Finding the direction of arrival (DOA) then reduces to picking which region is sounding, or declaring that the source is off-camera. That choice is made by a linear scoring rule over a dozen named features, most of them measures of array evidence inside each region.

The point is not that a linear rule beats a deep network. The point is that it gives class-agnostic visual coverage and an exact, per-decision split of "how much of this answer came from the microphones versus the pixels," and it should cost little accuracy in doing so. That split can then be used to flag likely errors. In two sentences: segment everything, score each segment with a transparent sum of acoustic and visual evidence, and output the winning segment's acoustically refined direction. Because the sum is linear, every localization comes with an exact acoustic-versus-visual audit trail.

Core Claim

Our hypothesis is that, with the sound-event detector held fixed, mask-candidate selection with a linear head will lower class-dependent localization error on STARSS23 below the published audio-visual system. The largest gains should come on the non-human classes where detector boxes give nothing. A linear head should also land within a small margin of a nonlinear head trained on the same features. A third claim matters most for the interpretability direction: the visual share of each decision's attribution should predict when that decision is wrong. If it does, the linear bottleneck becomes a usable abstention signal and is more than decoration.

Mechanism

For each video frame, an automatic mask generator from the Segment Anything family, or SAM 3 for comparison, produces masks on the equirectangular 360° image. In that image, pixel coordinates map directly to azimuth and elevation. Each mask $m$ defines a set of grid directions $\Omega_m$. From the multichannel short-time covariance we compute standard spatial spectra over the direction grid, SRP-PHAT and narrowband MUSIC averaged over frequency. For each mask we then pool these spectra into features. The features are spectrum mass inside $\Omega_m$ normalized by the total, peak value inside $\Omega_m$, and the distance from the mask to the global acoustic peak. Beside these acoustic features sit a few visual ones: mask objectness, frame-to-frame motion energy inside the mask, and text-image similarity between the mask crop and the class name output by the fixed sound-event detector. A null candidate, "source not in any mask," gets its own feature vector, built mainly from the acoustic mass outside all masks. The score is

$$s_m = \sum_i w_i\, x_{m,i}, \qquad p(m) = \frac{e^{s_m}}{\sum_{m'} e^{s_{m'}}},$$

where $x_{m,i}$ is feature $i$ of candidate $m$ and $w_i$ is a learned weight shared across candidates. The output direction is the acoustic peak restricted to the chosen mask, or the global acoustic peak for the null candidate. Each decision splits exactly into an acoustic term and a visual term, $s_m = s_m^{\text{ac}} + s_m^{\text{vis}}$. The weights are fit by multinomial logistic regression on the STARSS23 development split, with the ground-truth source assigned to the mask containing its annotated direction.

Each line of prior work supplies one ingredient and leaves out another. The linear-bottleneck idea of making the decision an exact sum of named contributions comes from concept-bottleneck work such as Concept Bottleneck Language Models For Protein Design and COLOR. In our case the "concepts" are physical array measurements tied to image regions, which also keeps the concept extractor free of learned leakage. We reuse the class-agnostic segmenter as is. Part-whole ambiguity, which SAPNet++ identifies, is handled by passing masks at several granularities as separate candidates and letting the acoustic features decide. The acoustic side is deliberately classical. The tetrahedral array has only four microphones, so we do not claim super-resolution. We use subspace methods only as feature extractors, following the MUSIC formulation surveyed for robot audition (/work/a-survey-on-sound-source-localization-in-robotics).

The closest existing systems differ in one place each. Multi-target DoA Estimation with an Audio-visual Fusion Mechanism fuses array cross-correlation with face boxes through an MLP reliability network, which ties it to faces and makes its decision unauditable. VP-SelDoA selects a target sound using a global CLIP embedding of an exemplar image, with cross-attention over binaural phase and level features, and never localizes the object in the scene image. Annotation-free Audio-Visual Segmentation and Audio-Visual Instance Segmentation segment sounding objects from monaural audio. They therefore cannot use the phase differences that separate two identical instruments. Physically Informed Spatial Regularization constrains an audio network with covariance physics but uses no vision. Its finding that simple algebraic steering models beat measured-response models in strong reverberation constrains us: we compute features from algebraic steering vectors only.

Research Program

The first stage builds the candidate pipeline on STARSS23 dev video and checks a coverage ceiling. This is the fraction of annotated source directions that fall inside some mask, reported per class. If coverage is low for some classes, such as footsteps, which come from a small, moving region near the floor, those classes cap the achievable gain, and we report that. The second stage fits the linear head and its nonlinear twins on identical features. The third stage is the audit study, described below. Distortion in equirectangular images is a known risk for segmenters trained on perspective photos. As a fallback, we run the segmenter on cube-map faces and merge masks across faces.

Baselines

The naive reference point is the acoustic peak alone, our null candidate always chosen. It is equivalent to running SRP-PHAT or MUSIC on the detector's active frames. The published audio-only STARSS23 baseline sits beside it, with a reported $LE_{CD}$ of 66.9° in microphone format (STARSS23).

The strongest existing competitor on this exact question is the STARSS23 audio-visual system, a convolutional-recurrent network fed with YOLOX detections. It reports an overall $LE_{CD}$ of 60.0° and 28.9° on the human classes, against 58.4° audio-only. It is the right target because it uses the same data, array and camera, and fuses vision with array audio. To keep the comparison fair, we keep its sound-event detection output fixed and swap only the localization stage. The same development/evaluation split applies to every system, and each gets an equal tuning budget of the same number of validation evaluations. Beating it means overall $LE_{CD}$ below 60.0° at equal or better $LR_{CD}$, with improvement on the non-human classes. No per-class non-human number is reported, so we recompute it from the released baseline.

As a second reference on a different array, we use the SSLR speaker dataset. There MLP-AVAW reports an MAE of 4.22° against 4.63° audio-only (Multi-target DoA Estimation with an Audio-visual Fusion Mechanism). There the bar is simply to match it with a linear head.

Three ablations isolate the key ingredients. First, YOLOX boxes replace masks inside our own linear head, which separates the value of class-agnostic coverage from the value of the head. Second, a uniform superpixel or grid partition replaces masks, which tests whether object boundaries matter or whether any region pooling would do. Third, the visual features are removed, and then the acoustic features are removed.

Why not just…?

Why not just train a better audio-only SELD network with physical regularization? This is a real threat on in-domain error. The inverse-Wishart regularizer brings $LE_{CD}$ from 35.19° to 33.96° on its synthetic benchmark (Physically Informed Spatial Regularization), though that is not STARSS23. Audio alone, however, cannot use the camera to reject directional interferers, which STARSS23 contains. We will run a SALSA-Lite-style audio model on STARSS23 under the same budget. If it matches our method, the visual contribution is not worth its cost, and the audit share will show it as a near-zero visual weight.

Why not just keep the detector boxes and improve the detector? The published system already shows the limit: large gains on people, almost none overall, because the closed-set detector never fires on doors or taps (STARSS23). The box-swap ablation answers this directly within our own head, broken down per class.

Why not just prompt SAM 3 with the detected class name and output the centroid of the top mask? This text-prompted shortcut (SAM 3) is the most serious alternative, because it needs no acoustics beyond the class label. It should fail whenever two instances of the same class are visible, or whenever the sounding part is not what the noun phrase names. We will evaluate it as a baseline and report it separately on the subset of STARSS23 frames with multiple same-class instances. If it ties our method there, our acoustic features add nothing and the claim is falsified.

Why not just use an MLP or gradient-boosted trees over the same features? The capacity they add over a linear rule may matter, because feature interactions such as "motion only counts if acoustic mass is also high" are plausible. We will train both under the same budget. A gap of a few degrees at most would justify the linear head. A large gap would mean we must add explicit crossed features before the bottleneck, and we will report how many such features were needed.

Decisive Checks

Three results settle the proposal. The first is overall and non-human-class $LE_{CD}$ on STARSS23 against 60.0°, with coverage reported. The second is the multi-instance subset comparison against SAM 3 text prompting. The third is the audit test. We bin evaluation decisions by visual share, $|s^{\text{vis}}| / (|s^{\text{ac}}| + |s^{\text{vis}}|)$, and measure whether error rises with that share. We then check whether abstaining on high-visual-share decisions improves error at a fixed recall more than abstaining by the nonlinear model's softmax confidence does. If the visual share is no better than confidence at predicting failure, the interpretability benefit is nominal, and we will report the work as a coverage result only.