Sound event localization and detection (SELD) means telling what is sounding and from which direction, frame by frame, when several sources may overlap. Recent work lets a user add a new sound class without retraining, by supplying a few example clips or a text description. The network then matches its outputs against that reference. The catch is where the example clips come from. In a real deployment, a user records their own doorbell or kettle in their own home, with the TV on and people talking. Those recordings are mixtures, not clean exemplars. The best existing open-set system averages whole-clip audio embeddings of the examples. It finds that a single audio example does worse than a text prompt because single exemplars are noisy (Zero- and Few-shot Sound Event Localization and Detection).
We propose to build each class's reference vector, its prototype, from the localizer's own per-source outputs on the example recordings rather than from whole-clip embeddings. The text description is used only as a soft pointer that says which of the separated per-source outputs in each example belong to the new class. The model already splits each example recording into a few "tracks," one per active source. A short attention step, seeded by the text, decides which tracks to average. The resulting prototype lives in the same space as the query-time outputs and excludes the interfering sources. We test whether this holds across convolutional-recurrent, pure-Transformer, and state-space backbones, since prototype methods have been reported to be fragile on Transformers.
Core Claim
Few-shot SELD under interference is limited mainly by what gets averaged, not by how many shots are averaged. If support embeddings are taken per source, inside the localizer's output space, and filtered by a text anchor, then a few in-situ example recordings should beat both whole-clip mean pooling and text-only prompts. The gain should grow with the example recordings' polyphony (number of overlapping sources) and shrink as their signal-to-noise ratio (SNR) rises. If the gain does not track interference in this way, the claim is wrong.
Background and Positioning
The open-set starting point is embed-ACCDOA (Zero- and Few-shot Sound Event Localization and Detection). Its network outputs several tracks per frame. Each track has a semantic embedding and an activity-coupled direction vector, and the tracks are trained with permutation-invariant training (a loss that ignores the order of tracks). The track format builds on ACCDOA, where vector length encodes activity and vector direction encodes arrival angle. It also builds on Multi-ACCDOA, which adds multiple tracks per class to handle same-class overlap. embed-ACCDOA classifies tracks by cosine similarity to text embeddings or to mean-pooled example embeddings. We keep its architecture and loss unchanged and replace only the prototype construction step. That makes it the method we reuse, and its 1-shot-versus-text finding is the evidence that motivates the gap.
Prototypical Networks supplies the core rule: a class is the mean of its embedded examples, which is optimal under squared Euclidean distance. It also supplies the warning that a single mean assumes a tight, single-cluster class and is pulled by outliers. Interfering sources inside an example recording are exactly such outliers. Centroids Matching likewise reports that prototypes are sensitive to support-set composition.
The closest prior use of audio prototypes under interference is the Interference Eraser (Visual Sound Localization in the Wild by Cross-Modal Interference Erasing). It shifts mixed audio embeddings toward fixed category centroids to recover quiet sounds for 2D visual localization. Our setting differs in three ways. We build prototypes at test time from user recordings rather than from pre-computed training clusters. We output 3D directions rather than image regions. And we filter interference at the level of separated tracks rather than with a global correction vector.
The cross-modal attention component borrows from two sources. Gated Cross-Attention mixes a cross-attended context with the original features through a learned gate, which prevents one modality from overwhelming the other. Selection by semantic prompt, as in VP-SelDoA, uses a visual prompt to pick out a target sound inside a mixture. We do not use attention to fuse modalities inside the encoder. We use the text embedding only to select evidence among example tracks, and the final prototype is purely acoustic. This matters because text describes a category, while the in-room recording captures this doorbell through this room.
Architectural generalization enters as a constraint. Prototype-based consistency regularization gave large gains on convolutional segmentation backbones, but its authors flag Transformer feature geometry as a risk. PhaseCoder shows that pure-Transformer spatial encoders are now competitive, while being limited to one dominant speaker. Is Mamba Capable of In-Context Learning? shows that state-space models match Transformers on some tasks but degrade on long contexts. Our method only reads the network's output track embeddings and never regularizes hidden features. We therefore predict it should transfer across backbones where feature-level prototype regularizers may not. This is a testable prediction, not an assumption.
Research Program
At enrollment, the user gives a text description and $K$ example recordings, each of which may contain other sources. We run the trained, frozen SELD network on every example recording. This yields a set of track-frame outputs indexed by $j$. Each output has a semantic embedding $e_j$ and an activity $a_j$, where $a_j$ is the length of its direction vector. Let $t$ be the text embedding from the same contrastive language-audio space embed-ACCDOA trains against. The prototype is refined for a few steps, starting from $q_0 = t$:
$$w_j \propto a_j \exp(\langle q_r, e_j\rangle/\tau), \quad c_r = \sum_j w_j e_j, \quad q_{r+1} = \mathrm{normalize}(c_r + \lambda t)$$
In this update, $w_j$ is the weight on output $j$ (normalized to sum to one), $\tau$ is a temperature, $c_r$ is the prototype at step $r$, and $\lambda$ controls how strongly the text keeps pulling the estimate. The update is a soft k-means step with one cluster, seeded and anchored by text. The activity weight discards silent tracks. The similarity weight discards tracks that belong to interferers. Two instances of the target class in one recording both receive weight, which is the correct behavior.
At query time, tracks are classified against the final prototype exactly as in embed-ACCDOA. We tune $\tau$, $\lambda$, and the number of steps on validation classes that are disjoint from the test classes. We also train an optional learned variant, where a gated attention pooling layer in the style of the Gated Cross-Attention work replaces the fixed update and is trained episodically on base classes. We adopt this variant only if it beats the fixed update.
We also analyze prototype quality directly. For each novel class, we compute a reference prototype from clean, isolated, spatialized example recordings. We then measure the distance from each method's estimate to that reference while sweeping the example recordings' SNR from clean to −10 dB and their polyphony from one to three sources.
Baselines
The naive reference point is zero-shot text-only classification with embed-ACCDOA, which uses no example recordings at all. The file reports that zero-shot outperforms 1-shot audio, but gives no aggregate number for zero-shot. No reported result was found on Lacuna for embed-ACCDOA zero-shot E_SELD, so we will measure it.
The strongest existing method is embed-ACCDOA few-shot with mean-pooled clip-level example embeddings (Zero- and Few-shot Sound Event Localization and Detection). It is the strongest competitor because it already handles open-set classes, overlapping tracks, and 3D directions in one network. Its reported 10-shot aggregate SELD error is 0.589, against 0.483 for the fully supervised baseline, in that paper's STARSS23/TNSSE21 evaluation. Error is lower-is-better. We rerun it with the same trained network, the same example recordings, and the same per-class validation threshold search, which matters because that paper reports high threshold sensitivity. The only difference is the prototype rule. Beating it means lower SELD error at matched shot count on interfered example recordings, with non-overlapping confidence intervals across random enrollment draws. A further target is 5-shot in-situ performance at or below its 10-shot error.
The closed-set ceiling is Multi-ACCDOA trained with full labels, which reports E_SELD 0.264 on DCASE 2021 Task 3 (Multi-ACCDOA). It anchors how much few-shot enrollment leaves on the table.
Two ablations isolate the key ingredient. The first is "hard nearest track": in each example recording, keep only the single most active track closest to the text embedding, then average. If this matches the full method, the attention refinement is unnecessary, and we will report the simpler rule as the contribution. The second is "track mean without text": average all active tracks. It tests whether the text anchor, rather than track-level extraction alone, does the filtering. A third control uses clean example recordings, which separates the effect of interference from the effect of shot count.
Why not just…?
Why not just use more shots with plain mean pooling? Mean pooling already improves steadily with more shots, which is a real threat. The averaging argument from Prototypical Networks cuts both ways, though. Averaging removes random exemplar noise, but a persistent interferer, such as the same TV in every recording, adds a consistent bias that does not average out. Our experiments vary whether interferers are independent across example recordings or shared. We predict mean pooling closes the gap only in the independent case.
Why not just use the text prompt and skip example recordings? Text is clean but generic, and the reported zero-shot result still trails supervised training. The specific doorbell in a specific room is not described by "doorbell." The "text-only" baseline settles this directly. If our in-situ prototypes do not beat text-only at 3 or more shots, the case for enrollment collapses and we will say so.
Why not just separate the example recordings first and embed the separated target? This is a legitimate alternative. It needs a separator that knows the target, and the file reports separation quality only on music duets, with OneAVM reaching 7.38 SDR on MUSIC (A Unified Audio-Visual Learning Framework). It also notes degradation beyond two sources. We include a text-queried separate-then-embed pipeline as a baseline under the same tuning budget. Our argument is that the localizer's tracks already perform the needed separation, and they do so in the very space used at query time. The experiment will show whether an extra separator buys anything.
Why not just fine-tune the classification head on the few shots? Linear heads overfit badly in few-shot regimes (Prototypical Networks) and require target labels per track. In-situ example recordings do not provide those labels without exactly the track-selection step we propose. We still run head fine-tuning on our selected tracks as a sanity check.
Proposed Experiments
We train embed-ACCDOA-style models on synthetic spatial mixtures built from FSD50K. Following the zero-/few-shot SELD setup, we hold out classes for enrollment and evaluate on STARSS23 real recordings and TNSSE21. We use DCASE 2021 Task 3 for same-class-overlap analysis. Example recordings are synthesized with the same spatial pipeline, using controlled interferer SNR (clean, 10, 0, −10 dB), polyphony (1–3), and interferer sharing (independent or shared across shots), at $K \in \{1, 3, 5, 10\}$. Metrics are aggregate E_SELD, class-dependent localization error and recall, location-aware F-score at 20°, and prototype distance to the clean reference.
For the architecture study, we train three backbones with matched parameter count and data. The first is a convolutional-recurrent network like ACCDOA's RD3Net. The second is a pure Transformer encoder in the style of PhaseCoder. The third is a Mamba encoder. We report the gain of our prototype rule over mean pooling for each. As a contrast, we also add a feature-level prototype consistency regularizer in the style of Xu et al. to each backbone, to test whether that design is the one that breaks on Transformers.
Three outcomes are decisive. The first is the shape of the gain: it should be near zero on clean example recordings and grow with polyphony and with shared interferers. A flat gain would mean the method is not removing interference. The second is the hard-nearest-track ablation, which decides whether soft refinement earns its place. The third is backbone consistency: a gain that survives on the Transformer and state-space encoders supports the claim that output-space prototypes avoid the geometric fragility reported for feature-level prototype methods.
References
- Zero- and Few-Shot Sound Event Localization and Detection
- Accdoa: Activity-Coupled Cartesian Direction of Arrival Representation for Sound Event Localization And Detection
- Multi-ACCDOA: Localizing And Detecting Overlapping Sounds From The Same Class With Auxiliary Duplicating Permutation Invariant Training
- Prototypical Networks for Few-shot Learning
- Centroids Matching: an efficient Continual Learning approach operating in the embedding space
- Visual Sound Localization in the Wild by Cross-Modal Interference Erasing
- An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention
- VP-SelDoA: Visual-prompted Selective DoA Estimation of Target Sound via Semantic-Spatial Matching
- Semi-supervised Semantic Segmentation with Prototype-based Consistency Regularization
- PhaseCoder: Microphone Geometry-Agnostic Spatial Audio Understanding for Multimodal LLMs
- Is Mamba Capable of In-Context Learning?
- A Unified Audio-Visual Learning Framework for Localization, Separation, and Recognition