Recorder-Invariant Acoustic Units for Unlabeled Soundscapes

Passive acoustic monitoring means leaving autonomous recorders in forests or oceans for months. It produces far more audio than experts can ever label (Overview of BirdCLEF 2021). Unwritten human languages face the same problem: speech exists, but nobody can afford to write it down. Zero-resource speech research tackled that problem by learning to "transcribe" audio into discovered, phoneme-like units without any text. Its central lesson was that a naive clustering of audio groups sounds by who is talking rather than what is said. Success came from explicitly removing speaker identity, using the free knowledge of which utterances share a speaker. This proposal argues that soundscapes have a direct analogue of the speaker problem: the recorder. Each device has its own gain, frequency response, noise floor and site background, so clusters of unlabeled field audio drift toward "which microphone" rather than "which call." Every deployment also records the analogue of the speaker label for free, namely the recorder and site ID.

We propose to learn discrete acoustic units from unlabeled soundscapes with a self-supervised audio encoder that is trained to forget recorder identity. We will measure success with a new label-light test borrowed from speech science, a bioacoustic minimal-pair ABX test, defined below. If the units are recorder-invariant, a handful of expert examples should be enough to search months of audio for a species, including species no supervised model was trained on. That would replace dense annotation with a few labeled queries.

Core Claim

Unit discovery on field audio fails mainly because of recorder and site nuisance, not because the encoder lacks capacity. Removing that nuisance with free deployment metadata, rather than with species labels, will give units that transfer across recorders and to unseen species better than clustering of existing self-supervised features. The new elements are three. First, the nuisance variable is the recorder rather than the speaker. Second, the variable is removed in a way that avoids the obvious confound, which is that different sites host different species. Third, the units are evaluated by across-recorder discrimination and few-shot retrieval rather than by supervised probes.

Relation to Prior Work

Four papers already discretize animal audio with speech-style self-supervised models, and they define what is not new here. AVES adapts HuBERT to animal audio. HuBERT is an encoder trained to predict k-means cluster IDs of masked audio frames. AVES shows that curated bioacoustic pre-training beats general audio, but it throws the clusters away and evaluates only with supervised probes. ISPA turns clusters of AVES or MFCC features into tokens and maps them onto human phonetic symbols. Its tokens lose accuracy relative to continuous features on Watkins marine mammals (0.69 versus 0.77), which is a warning that naive discretization discards information. Phonetic and Lexical Discovery of Canine Vocalization clusters HuBERT layers into dog "phones" and words, but it works on clean YouTube dog audio and relies on video for meaning. Deep audio embeddings for vocalisation clustering clusters autoencoder embeddings of syllables that were already isolated by hand. None of these confronts the recorder as a confound, and none works on continuous multi-recorder soundscapes. All four supply the method we start from, which is clustering the middle layers of a self-supervised encoder. Our contribution is what happens to those clusters under recorder shift.

Two papers motivate the choice of mechanism. Combining Adversarial Training and Disentangled Speech Representation separates speaker from content using a speaker classifier whose gradient is reversed, so the encoder learns to defeat it. This cut across-speaker ABX error by 14.6 points absolute in English and 16.8 in Indonesian on ZeroSpeech 2019. Unsupervised Acoustic Unit Discovery for Speech Synthesis achieves similar invariance by conditioning the decoder on speaker ID. Both assume human speakers and utterance groupings, and we transplant the idea to recorders.

Other work constrains the design. Crossing the Species Divide finds that the upper layers of speech models specialize to human phonetics, and that the noise-trained WavLM transfers best to field data. That finding tells us which layers to cluster and makes WavLM a required baseline. BIRB documents the drop from focal to passive recordings, which is the shift we target. Perch 2.0 is the strongest counter-position: supervised training on 1.54M labeled recordings beats self-supervised learning. We take that claim seriously and test where it stops holding. Perch 2.0 also trains to predict recording source, which is the opposite of our objective, so the contrast is clean.

Research Program

We start from AVES and from WavLM, both of which take 16 kHz audio. We continue their HuBERT-style masked-unit training on unlabeled soundscape audio from BirdSet and BIRB training regions. Cluster targets come from a middle layer, as the Crossing the Species Divide results suggest.

The key design problem is the confound. An adversary that predicts site from vocalization frames would push the encoder to merge species, because species composition itself differs by site. We therefore apply the recorder adversary only to background frames. These are low-novelty, low-energy stretches selected by a fixed energy and spectral-flux detector. Channel signatures such as gain, frequency response and noise floor are shared by background and call frames, so removing them from background frames should remove them from call frames too. Species identity, meanwhile, never enters the adversary's input.

The training objective is

$$ \mathcal{L}(\theta,\phi) = \mathcal{L}_{\text{MP}}(\theta) - \lambda\, \mathcal{L}_{\text{rec}}(\phi;\theta, \mathcal{B}). $$

Here $\theta$ are the encoder weights, and $\mathcal{L}_{\text{MP}}$ is the masked-unit prediction loss. $\phi$ are the weights of a recorder classifier, and $\mathcal{L}_{\text{rec}}$ is its cross-entropy on pooled encoder states from the background frame set $\mathcal{B}$. $\lambda$ sets the strength of the reversal. We will compare this against the second speech-style option, decoder conditioning on recorder ID in the spirit of Eloff et al.

The second ingredient comes from the transcription-bottleneck literature. In soundscapes, cluster targets are noisy. Following the finding in Self-Training for End-to-End Speech Recognition that pseudo-labels on noisy audio must be filtered and ensembled, we run k-means with several seeds. We then mask out of $\mathcal{L}_{\text{MP}}$ any frames where the runs disagree. Agreement is measured after matching cluster labels across runs, by analogy with the ensemble-spread uncertainty in DISCO. Kahn et al.'s method needs a text language model, so it cannot be copied directly. Agreement across clusterings is our text-free replacement for their confidence score. This ingredient is secondary and gets its own ablation.

Two products come out of this training. One is a unit transcript of every soundscape. The other is query-by-example search: given a few exemplar clips, we rank soundscape windows by DTW distance over unit posteriors. DTW, dynamic time warping, aligns two sequences that play at different speeds.

We also introduce a bioacoustic ABX test. In an ABX trial, A and B are windows containing different species recorded on recorder 1, and X contains A's species recorded on recorder 2. The representation scores correct if X is closer to A than to B under DTW with cosine distance, following Schatz et al.. The paired score, recorder-across-species, measures how much channel information remains, and lower discrimination is better there. Both scores need only the existing sparse evaluation labels, never training labels. To control the site confound in evaluation, we restrict trials to species that were detected at both recorders.

Baselines

The naive reference is classical features: MFCCs clustered by k-means, scored with the same ABX and retrieval protocols. The BEANS results show such shallow pipelines are competitive on small curated classification tasks but weak on continuous detection, so beating them on soundscapes is necessary but not impressive.

The strongest unsupervised competitor is k-means on frozen middle-layer features of AVES and of WavLM. This is essentially what ISPA-F and the canine study do. Its units cost nothing extra, and WavLM's noise-robust pre-training already gives some field robustness. We cluster it with the same number of units, on the same unlabeled audio, using the layer chosen on a held-out validation region.

The strongest supervised competitor is Perch 2.0 embeddings used for the same exemplar-based retrieval. The file reports BirdSet ROC-AUC 0.908 and BEANS mAP 0.504 for Perch 2.0. On the BIRB retrieval task itself, the reported reference is the Conformer S baseline in BIRB: 0.75 cROC-AUC on passive soundscapes, falling within roughly 0.65–0.80 for held-out species. No reported result found on Lacuna for Perch 2.0 on BIRB, and none for any bioacoustic ABX test, so we measure both ourselves.

All self-supervised arms get the same unlabeled hours and GPU budget, and an equal hyperparameter search judged on a validation region. Beating Perch 2.0 overall is not the goal. Our method counts as better only if it beats Perch 2.0 on held-out species or non-bird taxa, and beats the AVES/WavLM clustering baseline everywhere on across-recorder ABX and held-out-recorder retrieval.

Ablations isolate each ingredient. One drops the adversary, which is plain continued pre-training on soundscapes. One applies the adversary on all frames instead of background only; this is the confound test, and we predict it hurts species ABX. One swaps the adversary for recorder-conditioned decoding. One removes agreement filtering.

Why not just…?

Why not just normalize features per recorder? Subtracting each recorder's mean and dividing by its variance, as speech does with cepstral normalization, may remove much of the channel effect at zero cost. This is a real threat, so it is a baseline applied on top of AVES and WavLM features. If it matches the adversary on across-recorder ABX, the adversary is unnecessary. In that case the contribution shrinks to the evaluation protocol plus the finding that recorder variation is mostly a linear offset, and we would report it that way. The adversary can only win if part of the nuisance is nonlinear, such as site background sound and distance-dependent filtering, and the gap between the two arms measures exactly that.

Why not just use Perch 2.0 embeddings? For species within its 14,795 classes it will likely win. Perch 2.0's own analysis shows that transfer depends on species-level labels, and BIRB shows drops for unseen species. Our held-out species and non-bird BEANS taxa test exactly the regime where those labels do not exist. If Perch 2.0 still wins there, the case for unsupervised discovery in monitoring is weakened, and we will say so.

Why not just cluster existing self-supervised features, as ISPA and the canine study do? That is the main baseline. ISPA's drop from 0.77 to 0.69 on Watkins shows that discretization of those features loses information. Whether recorder nuisance explains part of that loss is precisely what the recorder-across-species score diagnoses.

Why not just augment with simulated channel effects? Random filtering and added noise can stand in for recorder variation. We include this as an arm with an equal compute budget. Simulated channels may not cover real sensor and site differences, but the held-out-recorder results, not argument, will decide.

Proposed Experiments

We pre-train on unlabeled training-region audio from BirdSet and BIRB, using recorder IDs where the metadata provides them and falling back to site level otherwise. We use the labeled test soundscapes only for ABX trials and retrieval queries. Generalization is tested in three ways: on held-out recorders, on held-out regions, and on BIRB's held-out species. Classification and detection on BEANS give a breadth check across taxa.

The first decisive check is whether recorder-across-species discrimination falls while species-across-recorder ABX error also falls relative to plain clustering. If both scores move together, recorder information is entangled with species information and the method has failed. The second check is the background-only versus all-frames adversary comparison under the shared-species restriction, which directly tests whether the confound handling works. The third check is annotation efficiency: how many exemplar clips are needed to reach the Conformer S passive-soundscape level of 0.75 cROC-AUC on held-out species, compared across our units, AVES/WavLM clusters, per-recorder normalization, and Perch 2.0. A curve that reaches 0.75 with markedly fewer exemplars is the result that would matter to monitoring practitioners.

One limitation is fixed in advance. The 16 kHz input caps analysis at 8 kHz, so we will report results separately for species whose calls lie mostly below 8 kHz.