Hearing Absence: Masking-Aware Negative Pseudo-Labels for Overlapping Audio

Self-training lets a model label its own unlabeled data and then learn from those guesses. It is a natural fit for passive acoustic monitoring, where microphones record thousands of hours of overlapping animal calls that nobody will ever annotate. The most reliable version of self-training keeps only the guesses the model is stable about, measured by how much its prediction changes when parts of the network are randomly switched off. It also adds "negative pseudo-labels": confident statements that a class is absent. These are especially valuable when each recording contains only a few of hundreds of possible species.

Audio has a failure mode that images mostly lack. A loud sound can completely bury a quiet one in the same time and frequency region, which is called energetic masking. In that case the model is not wrong to hear nothing. It is consistently and stably confident that the quiet species is absent, so stability-based uncertainty raises no alarm. Feeding that confident "absent" back into training teaches the model to ignore quiet background callers, which are often the rare species monitoring exists to find. We propose separating confidently absent from unobservable here. We test this directly by asking the model whether it could have heard the class: we inject a known example of the class into the frame at a realistic loudness and see whether the model detects it. Negative pseudo-labels are kept only where the class was audible.

Core Claim

Stability-based uncertainty measures the model's ignorance about the decision rule. It does not measure whether the input contained enough signal to decide. In overlapping audio, the negative pseudo-labels that do the most damage come from masked regions where the model is stably and wrongly certain. We claim three things. First, stochastic-pass variance will not separate masked-absent from truly-absent frames. Second, an injection-based audibility test will. Third, gating negative pseudo-labels by audibility will improve recall on quiet and rare classes in soundscape self-training without losing the density benefit of negatives. This is a specific, falsifiable hypothesis about where uncertainty-aware filtering breaks when it leaves images. It is not a port of an image method to spectrograms.

Relation to Prior Work

The method we build on is Uncertainty-aware Pseudo-label Selection from In Defense of Pseudo-Labeling. It filters positive and negative pseudo-labels by Monte Carlo dropout variance, meaning prediction spread across repeated passes with random neurons dropped. On multi-label Pascal VOC it raises mAP from 27.44 to 34.22 over plain pseudo-labeling, largely through negatives. The efficient Bayesian last layer of Mitigating Confirmation Bias in Semi-supervised Learning via Efficient Bayesian Model Averaging gives the same kind of variance estimate at 2–5% overhead, and we reuse it to make filtering affordable. Both were evaluated only on images and video. Both treat low variance as permission to trust a label. Masking is precisely a case where that permission is wrongly granted.

Three audio papers attack overlap with semi-supervision, and each leaves the absence question unexamined. 3MAS masks unannotated classes out of the loss using dataset metadata. That avoids false negatives but discards all negative information and never estimates uncertainty. Hodge and Podge adapts MixMatch to multi-hot labels and enforces that a mixture's prediction equals the mix of its parts. That penalizes the model for missing events that mixing has made physically inaudible, the same masking issue in reverse. Selective Pseudo-labeling filters separated streams by matching against the mixture's ground-truth weak tags. That is unavailable on unlabeled soundscapes.

Two lines constrain our design. Improving Bird Classification with Unsupervised Sound Separation shows that separation front-ends help (SSW cmAP 0.304 to 0.341) but over-separate calls into fragments, so we do not make separation part of the core mechanism. Towards Deep Active Learning in Avian Bioacoustics shows that entropy-based querying on frozen Perch embeddings gains up to 15% over random, but it requires human labels and a frozen backbone. We target the remaining unlabeled pool and adapt the backbone. Finally, BirdSet documents the focal-to-soundscape gap: models trained on clean single-species recordings collapse on field recordings. That gap is our application setting.

Research Program

The system is a multi-label sound classifier with independent per-class sigmoid outputs and a Bayesian last layer. It is trained on labeled focal recordings and then self-trained on unlabeled soundscapes.

Positive pseudo-labels follow the uncertainty-aware recipe, with one audio-specific change. Variance is computed on predictions smoothed over short temporal segments rather than single frames, to avoid the flickering masks that frame-wise dropout produces. Following the observation in Liu et al. (see), stable low-confidence detections are retained rather than discarded.

The new component is the audibility test for negatives. For a segment $x$ and a candidate-absent class $c$, we draw a held-out labeled exemplar $s_c$ of that class and mix it in at a signal-to-noise ratio $\rho$ sampled from the class's empirical loudness distribution in field recordings. We then define audibility as the detection rate

$$a_c(x) = \Pr_{s_c,\rho}\big[\, f_c(x + g_\rho s_c) > \tau \,\big],$$

where $f_c$ is the model's score for class $c$, $g_\rho$ scales the exemplar to loudness $\rho$, and $\tau$ is the detection threshold. A negative pseudo-label for $c$ is emitted only if three conditions all hold: the original score is low, its variance is low, and $a_c(x)$ exceeds a threshold. Otherwise the class is left unsupervised for that segment, as in 3MAS masking. This keeps negatives exactly where the model demonstrably could have heard the class.

Running injections for every class on every segment would be expensive, so we amortize. Injection outcomes on a subset of segments become training targets for a small audibility head that reads the backbone's embedding and predicts $a_c(x)$ for all classes in one pass. The injection probe is then run only to periodically recalibrate this head. We also study which classes are worth probing at all, since high-prior classes near the threshold carry most of the risk.

Proposed Experiments

The primary setting is BirdSet. We train on the Xeno-Canto focal data. We self-train on unlabeled soundscape audio with labels withheld, holding out whole recording sites. We evaluate cmAP and mean AUROC on the HSN, PER and other soundscape test sets. We also report recall stratified by annotated call loudness relative to the segment, because the claim is specifically about quiet, overlapped calls.

A secondary setting is DCASE 2019 Task 4. It provides a weak/strong/unlabeled split and a published semi-supervised comparator, which lets us check that the effect is not bird-specific.

For mechanism diagnosis we use controlled synthetic mixtures built from isolated focal calls. Synthetic mixing of isolated calls is standard practice for overlapping bioacoustics, and these mixtures let us know exactly which sources are submerged and at what loudness.

Baselines

The naive reference is the supervised focal-trained model with no self-training. For a supervised reference point, the BirdSet paper reports a large-training ConvNeXt at mean AUROC 0.85 and cmAP 0.36 (BirdSet). We reproduce this configuration as our starting checkpoint so that all self-training variants share its backbone, data and compute.

Plain confidence-threshold pseudo-labeling uses fixed sigmoid thresholds for presence and absence. It is the cheapest competitor, and its threshold is tuned on the same validation sites with the same budget as ours.

The strongest existing competitor is the uncertainty-aware selection method of In Defense of Pseudo-Labeling, ported to audio with both positive and negative pseudo-labels and our temporal smoothing. We run it with the same Bayesian head, so the only difference from our method is the audibility gate. It is strongest because it already exploits dense negatives and already filters by uncertainty. No reported result was found on Lacuna for this method on any audio benchmark; its best multi-label figure in the file is 34.22 mAP on Pascal VOC.

We also run a 3MAS-style variant that never emits negatives for unlabeled segments (3MAS). This tests whether negatives are worth keeping at all.

On DCASE 2019 Task 4 we compare against Multi-Hot MixMatch, whose reported event-based F-score is 43.4% (Hodge and Podge). We re-run it on our backbone where possible and report both the published and re-run numbers.

Two ablations isolate the key ingredient. The first replaces the injection probe with a band-limited energy heuristic: the ratio of segment energy in class $c$'s typical frequency band to the dominant energy. The second uses the exact injection probe without the amortized head.

Beating the baselines means two things, and both are required. Our method must improve stratified recall on quiet calls over the uncertainty-aware port at matched precision. It must also not lose overall cmAP relative to the best of the uncertainty-aware port and the no-negatives variant, with differences assessed across at least three seeds and held-out sites.

Why not just…?

Why not just use a high confidence threshold? On images this collapses under confirmation bias. In the Bayesian averaging paper, standard pseudo-labeling falls to 8.2% CIFAR-100 accuracy at 100 labels versus 35.2% with variance filtering (Mitigating Confirmation Bias). On CIFAR-10, plain pseudo-labeling reaches 22.02% error versus 8.18% for uncertainty-aware selection (In Defense of Pseudo-Labeling). More to the point, a masked call yields a confidently low score, so any threshold on confidence will emit exactly the harmful negative. The threshold baseline will show whether this holds on audio.

Why not just port uncertainty-aware selection unchanged? This is the real threat. If dropout or Bayesian variance is already high on masked-absent segments, the audibility test adds nothing. The first decisive check measures this directly on synthetic mixtures. If variance separates masked-absent from truly-absent segments about as well as audibility does, we abandon the gate and report the port as sufficient.

Why not just drop negative pseudo-labels, as 3MAS effectively does? This removes masking errors but also removes the dense absence signal that drove most of the multi-label gain on Pascal VOC. Whether that loss matters for hundreds of species is an empirical question that the no-negatives baseline settles.

Why not just use a frequency-band energy rule instead of injecting calls? This could plausibly work for tonal species with narrow bands. That is why it is an ablation and not a straw man. The rule ignores what the model has actually learned to detect, for example calls that remain recognizable through partial masking via harmonics or temporal pattern. We expect the injection probe to win mainly on broadband and harmonically rich classes. If the heuristic matches it overall, we will recommend the heuristic as the cheaper method.

Why not just ask experts via active learning? Active learning gains up to 15% over random on HSN (Towards Deep Active Learning in Avian Bioacoustics), but annotating overlapping soundscapes is costly and does not scale to the unlabeled pool. The two approaches are complementary: audibility scores could also tell annotators where absence is unverifiable.

Decisive Checks

The first check is diagnostic. On synthetic mixtures with known submerged sources, we compare how well segment variance and audibility each distinguish masked-absent from truly-absent class–segment pairs, measured by AUROC. The proposal stands only if audibility is clearly better.

The second check is causal. On soundscape test sets, we count how many annotated-present calls would have received a negative pseudo-label under the uncertainty-aware port, and how many of those the gate removes.

The third check is end-to-end. We compare loudness-stratified recall and overall cmAP on held-out BirdSet sites against all baselines. We also report the amortized head's agreement with the exact probe and its wall-clock cost relative to dropout-based filtering.