Several recent results suggest that humans and other primates share acoustic building blocks for vocal communication. One striking example: a simple classifier separates the sounds that help 4-month-old infants learn object categories (English, German, lemur calls) from those that do not (Cantonese, zebra-finch song) with a reported area under the ROC curve (AUC) of 0.99 using rhythm features, against only 0.63 using pitch (Rhythm May Be Key to Linking Language and Cognition in Young Infants). Near-ceiling separation of five sound classes drawn from different sources should raise a question. A classifier can tell recordings apart by microphone, room, background hiss, file encoding or resampling path just as easily as by the vocalization itself. Bioacoustics already has a name for one version of this failure, the "background trap": pretrained audio embeddings group target calls with environmental noise (No Free Lunch from Audio Pretraining in Bioacoustics).
We propose a confound audit for cross-species acoustic claims. Every analysis runs in a pinned, containerized environment that logs each run. Each claim is then put through three tests. The first is a background-only control: train the same classifier on the non-vocal stretches of each recording. The second is a software-environment sweep that varies audio decoders, resamplers and feature-extraction library versions. The third is a leave-one-recording-source-out split. The output is a number per claim: how much of the reported discriminability survives once recording context and software environment are separated from the vocal signal. The reproducibility machinery is not the goal. It is the instrument that makes the second test possible and makes all three auditable.
Core Claim
For any reported cross-species acoustic result, part of the classifier's accuracy can be recovered from recording context alone, and part is sensitive to the software that decodes and transforms the audio. Neither part is currently measured. We define the background-recoverable fraction as
$$\rho = \frac{\mathrm{AUC}_{bg} - 0.5}{\mathrm{AUC}_{full} - 0.5},$$
where $\mathrm{AUC}_{full}$ is the classifier's performance on the original clips and $\mathrm{AUC}_{bg}$ is its performance when trained and tested only on vocalization-free segments from the same recordings. A value of $\rho$ near 0 means the result lives in the vocal signal. A value near 1 means the classifier could have reached the same verdict without hearing an animal.
We also report the environment spread: the range of $\mathrm{AUC}_{full}$ across a grid of container images that differ only in audio-handling software. The hypothesis is falsifiable in both directions. If the rhythm-over-pitch ordering holds with low $\rho$, small spread, and survival under source-held-out splits, the "primate-general template" reading gets much stronger evidence than it has now.
Where This Sits
On the auditory side, the claims under audit come from three places. The infant study above is the primary target, because its labels are tied to a cognitive outcome and its ceiling-level AUC makes it most exposed to recording confounds. The second is the MADUV mouse ultrasound challenge (MADUV), which reports a puzzling result: features from the audible band below 20 kHz outperform ultrasonic features (20–150 kHz) for separating autism-model mice from wild-type, reaching a subject-level unweighted average recall (UAR) of at most 0.625. Since mouse calls are ultrasonic, cage, handling or equipment noise in the audible band is a plausible explanation that nobody has tested. The third is the bioacoustic encoder literature. AVES adapts a self-supervised speech model to animal sounds, and AVEX compares 19 encoders across 26 datasets. Both operate at 16 kHz, so the resampling path is itself a hidden analysis choice.
Prior work motivates the gap in two ways. No Free Lunch shows that removing unlabeled background segments raises gibbon detection average precision (AP) for AVES-bio from 0.284 to 0.457. BIRB shows a drop from 0.87 to 0.75 cROC-AUC (the geometric mean of per-species ROC-AUC) when moving from focal recordings to passive soundscapes. Both papers treat background as a nuisance that lowers performance. We treat it as a possible source of false positive scientific conclusions, which neither paper tests.
The neuroimaging work on conserved sequence processing (Auditory sequence processing reveals evolutionarily conserved regions of frontal cortex) provides the broader motivation: claims of cross-species continuity are being built on acoustic comparisons. We do not audit the fMRI results themselves.
On the reproducibility side, we borrow methods without adopting their full apparatus. BIDS apps supplies the pattern of versioned container images consuming a standardized directory layout. It also states the limit we must respect: containers share the host kernel, so small numerical differences across hardware remain. Efficient and reproducible pipelines for spike sorting shows that containerized, orchestrated sweeps are cheap at scale. It reports a 20x speedup and about $66 for a six-probe session, which makes a large analysis grid affordable.
Audit-Grade Harness for Agent-driven Scientific Computation Workflows supplies the append-only run log. It also shows that outcome metrics miss silent execution deviations. We take the log from it but deliberately not its language-model verifier, which that paper itself flags as uncalibrated. Our checks are deterministic instead: content hashes of each image, input file and feature array, so that every reported number traces to exactly one logged run. A Qualitative Study on … Flaky Tests supports the premise that operating-system and library drift, not only language-level package versions, produces non-reproducible results.
Research Program
We build one container image per point in a small environment grid. The axes are the audio decoding backend (a libsndfile-based reader versus an ffmpeg-based reader), the resampling algorithm and its anti-aliasing filter, floating-point precision, and two versions of the feature-extraction libraries. The features are the envelope spectrum and temporal modulation spectrum (rhythm), MFCCs (a compact summary of the spectral envelope) and fundamental-frequency contours (pitch). Each run writes its image hash, input hashes, feature hashes and metrics to an append-only log.
Each dataset is then segmented into vocal and non-vocal stretches. For BEANS detection sets and the gibbon data we use the existing annotations. For MADUV we use the ultrasonic call segmentation. For the infant stimuli we use inter-call gaps; these may be short, so we also add leading and trailing silence where available.
Three analyses run on every claim:
- Background-only replication. The original pipeline is rerun on non-vocal segments, giving $\mathrm{AUC}_{bg}$ and hence $\rho$. The pipeline here is the 100-iteration Monte Carlo cross-validation with an ensemble of LASSO (sparse linear regression), a support vector machine and naive Bayes, as in the infant paper.
- Noise-bed remix. Every vocal segment is mixed onto a shared background drawn from neutral field recordings at a fixed signal-to-noise ratio. This removes source-specific context while keeping the call.
- Leave-one-source-out evaluation. Recordings are grouped by origin (archive, session or recording rig where metadata allows), and every test fold contains only sources unseen in training.
Baselines
The naive reference point is the published result rerun as-is in a single pinned image. This means AUC 0.9939 for rhythm, 0.9955 for MFCCs and 0.6287 for pitch on the infant classification (Rhythm May Be Key). It also means MADUV's audible-band subject UAR of 0.625 (MADUV) and AVES-bio gibbon AP of 0.284 before and 0.457 after background removal (No Free Lunch). A faithful reproduction within the environment spread is the first checkpoint. Failing to reproduce is itself a reportable finding.
The infant stimuli may not be publicly available. If they are not, we will request them from the authors. If the request fails, we will rebuild matched stimulus classes from public recordings and state clearly that this is a reconstruction rather than a replication.
The strongest existing method is a frozen-encoder linear probe on AVEX's sl-BEATs-bio, which reports 0.840 probe accuracy on BEANS classification, ahead of BirdNet at 0.796 and Perch at 0.768 (AVEX). We run it with the same splits, the same probe and the same hyperparameter budget as the hand-crafted features. It is the strongest competitor for two reasons. A reviewer will argue that a strong learned encoder makes hand-crafted rhythm features moot. And if any representation is vulnerable to the background trap, it is a high-capacity pretrained one. AVES-bio is included as a second learned baseline because the gibbon background result was measured on it.
"Beating" a baseline here does not mean higher accuracy. It means a lower $\rho$ at comparable $\mathrm{AUC}_{full}$, and a smaller drop under leave-one-source-out. A representation that loses less when context is removed is the better scientific instrument even if its in-distribution score is slightly lower.
Two ablations isolate the proposal's ingredients. In the environment ablation, the full analysis is rerun with only a pinned pip/conda manifest on two different host systems, compared against the container grid. This measures whether containerization changes any conclusion, rather than assuming it does. In the context ablation, the noise-bed remix and the source-held-out split are applied separately, to show which of them carries most of the effect.
Why not just…?
Why not just pin a conda environment and rerun the published code? Pinning fixes Python packages but not the system audio libraries underneath them. The flaky-test study (A Qualitative Study on … Flaky Tests) identifies exactly this layer as a source of drift. Still, we do not assume containers matter; the environment ablation measures it directly. If the conda reruns land inside the container grid's spread on every claim, we will report that software environment is a negligible factor for these features. The contribution then narrows to the confound audit, and that is an acceptable outcome.
Why not just run a background-only control in a notebook? This is the most serious threat to the proposal's novelty. For any single dataset, the control is cheap, and the infrastructure adds nothing to its logic. What the infrastructure adds is the ability to run the same control, remix and source split identically across four or five claims and several environments. Every cell is hash-traceable, so a reader can verify that a low $\rho$ was not obtained from a lucky configuration. If a reviewer sees only one claim, the notebook suffices. The value lies in the cross-claim, cross-environment table and in its auditability.
Why not just use a stronger pretrained encoder and trust its results? Encoder strength does not answer the question. No Free Lunch shows that pretrained embeddings are the representations most prone to absorbing background. The sl-BEATs-bio baseline is included precisely to measure its $\rho$. If it shows high accuracy together with high $\rho$, that is evidence against simply trusting the larger model.
Why not just re-record stimuli under matched conditions? Matched re-recording is the gold standard, and the noise-bed remix is only an approximation of it. We will say so explicitly. For archival lemur, finch and mouse recordings, re-recording is impossible. A remixed result that agrees with leave-one-source-out evaluation is the best evidence available without new data collection.
Decisive Checks
The central check runs on the infant stimuli. If $\mathrm{AUC}_{bg}$ reaches roughly 0.9, giving $\rho$ above about 0.8, the 0.99 rhythm result cannot be read as evidence about vocal rhythm. If $\rho$ stays near 0, and rhythm still beats pitch under both the noise-bed remix and the source-held-out split, the conserved-template claim survives a test it has not faced before.
For MADUV, the prediction is concrete. If the audible-band advantage is a recording artifact, non-vocal audible segments alone should approach the 0.625 subject UAR, and the advantage should shrink after remixing. If they stay near chance, the audible band carries real vocal information, which would be a surprising biological result worth following up.
For the gibbon data, we expect $\rho$ for AVES-bio to be large given the reported AP gain from background removal. Confirming this checks that our measure detects a known confound.
Across all claims, the environment spread tells us whether resampling to the 16 kHz input used by AVES and AVEX, and the choice of decoder, can flip a qualitative conclusion. Flipping means changing which feature family wins, or moving a result across chance. Results are reported per claim. This lets a reader see which cross-species acoustic conclusions are robust and which rest on the recording.
References
- Rhythm May Be Key to Linking Language and Cognition in Young Infants: Evidence From Machine Learning
- No Free Lunch from Audio Pretraining in Bioacoustics: A Benchmark Study of Embeddings
- MADUV: The 1st INTERSPEECH Mice Autism Detection via Ultrasound Vocalization Challenge
- AVES: Animal Vocalization Encoder Based on Self-Supervision
- AVEX: What Matters for Animal Vocalization Encoding
- BIRB: A Generalization Benchmark for Information Retrieval in Bioacoustics
- Auditory sequence processing reveals evolutionarily conserved regions of frontal cortex in macaques and humans
- BIDS apps: Improving ease of use, accessibility, and reproducibility of neuroimaging data analysis methods
- Efficient and reproducible pipelines for spike sorting large-scale electrophysiology data
- Audit-Grade Harness for Agent-driven Scientific Computation Workflows
- A Qualitative Study on the Sources, Impacts, and Mitigation Strategies of Flaky Tests