Distilling Only the Sounds That Matter: Concept-Filtered Students and Context-Swap Tests for Audio

Large audio models such as CLAP, which learns to match sound clips with their text descriptions, score well on standard sound-classification benchmarks. When they are compressed into small on-device models by knowledge distillation (training a small "student" network to imitate a large "teacher"), the student inherits whatever the teacher relied on. If the teacher labels a clip "rain" partly because of the room hum or traffic typical of the recordings it saw, the student learns that too. Standard benchmarks cannot reveal this. Their clips are static, the background usually co-occurs with the label, and some were scraped from the same Freesound pool the teacher was pre-trained on. A high score can therefore come from context and memorization rather than from recognizing the sound.

We propose to filter what is distilled at the level of named sound concepts, and to test the result with procedurally built audio mixtures that break the link between foreground and background. Each teacher embedding is decomposed into a sparse set of text-defined concepts. Concepts outside the target class's branch of the sound ontology are removed, and the student is trained to reproduce only the remaining foreground concepts. The test pastes fixed foreground events onto backgrounds from other classes at controlled signal-to-noise ratios. If filtering works, the student loses less accuracy when the context is swapped than a conventionally distilled student, at the same clean accuracy and the same size.

Core Claim

Concept-filtered distillation from CLAP into a small log-mel CNN will reduce the student's accuracy drop under context swaps compared with vanilla distillation, while matching clean accuracy. The effect should reverse when we invert the filter and distill only the removed context concepts. This inverse ablation is the causal test: it shows the filter transfers or blocks context reliance, not just sparsity. A second, smaller claim concerns contamination. A clean split of ESC-50 can be built by near-duplicate search against CLAP's pre-training pool. On that split, the gap between filtered and vanilla students should be at least as large as on the full set.

Where This Sits in Prior Work

The mechanism comes from Transformation of audio embeddings into interpretable, concept-based representations. That paper decomposes CLAP embeddings by Lasso regression (a linear fit that forces most weights to zero) into sparse non-negative combinations of text concept vectors built from FSD50K tags. Sparse codes matched or beat dense ones, raising TUT2017 accuracy from 0.538 to 0.556. Its class profiles also exposed context reliance: the "beach" scene is carried by "splashing" and "children."

That work stops at post-hoc analysis of a frozen teacher. It does not distill, and it does not act on the shortcuts it finds. We turn its decomposition into the distillation target and its shortcut diagnosis into a filter.

The idea that teachers should be filtered rather than copied is motivated by three papers:

We build an explicit foreground filter in audio and test the prediction behaviorally.

Two lines of audio work warn against substitutes:

The evaluation design borrows from DyVal. DyVal generates test items procedurally with tiered difficulty and showed that models strong on static benchmarks can collapse on fresh items. That work is confined to text reasoning; we carry its logic to acoustic mixtures, with SNR and distractor count as the difficulty tiers.

From the feature-engineering line, Attention Is Not Always the Answer found that simply adding MFCCs (classic hand-designed spectral features) to pretrained embeddings beat cross-attention for voice activity detection. This finding constrains the proposal. We must show that a cheap MFCC-based model is not equally robust to context swaps before attributing robustness to anything learned.

Research Program

The filter. For each training clip, the frozen CLAP audio embedding is decomposed by Lasso over the FSD50K concept vocabulary, as in Zhang et al. Because FSD50K tags follow the AudioSet ontology, each target class can be mapped to its ontology subtree. A concept is kept for a clip of that class if it lies inside the subtree, and suppressed otherwise. This default needs no extra data and no hand labels beyond the existing ontology mapping.

A second, learned variant drops a concept if it predicts the label across classes but is not in any class's subtree. We compare the two variants because the over-pruning risk noted in the file is real.

The student. The student is a small log-mel CNN, sized for edge deployment, that never sees CLAP at inference time. It is trained with two signals:

  • the usual cross-entropy on labels;
  • a regression loss pulling a projection head toward the filtered concept vector $c^{f}$.

The loss is

$$\mathcal{L} = \mathcal{L}_{CE}(y, \hat{y}) + \alpha \,\lVert P(z) - c^{f} \rVert_2^2,$$

where $z$ is the student's penultimate embedding, $P$ is a linear head onto the concept space, $c^{f}$ is the teacher's Lasso concept code with out-of-subtree concepts zeroed, and $\alpha$ is tuned on validation data.

The context-swap test. Foreground events are drawn from held-out ESC-50 and US8K test folds. They are mixed onto background beds taken from clips of other classes and from SONYC-UST urban recordings, and labeled with the foreground class. Difficulty is set in tiers, in the spirit of DyVal: SNR runs from 20 dB down to 0 dB, and one to three non-target distractor events are added. Matched control mixtures place each foreground on a background from its own class, so that mixing artifacts affect both conditions equally. The primary metric is the context-swap drop: control accuracy minus swapped accuracy, at each tier.

The contamination audit. We run near-duplicate search, using audio fingerprints plus CLAP-embedding cosine with a strict threshold that we will verify by listening, between the ESC-50 and US8K test folds and CLAP's listed training pools (FSD50K, Clotho, AudioCaps, MACS). Every metric is reported separately on flagged and clean test clips. We do not assume overlap exists; the audit measures it.

Baselines

Naive references. The first is the same student architecture trained on labels alone, with no teacher. The second is a feature-engineering baseline: logistic regression on MFCC statistics, plus a small model fusing MFCCs by simple addition, in the style of Tripathi et al.

Teacher references. We also report the teacher itself, CLAP zero-shot and with a linear probe. For calibration, CLAP's paper reports 82.6% zero-shot accuracy on ESC-50 and 73% on US8K (CLAP). AST reports 95.6% on ESC-50 as a large supervised ceiling.

Strongest competitor. This is vanilla distillation from CLAP to the identical student: soft-label matching plus regression of the unfiltered concept code. It is the strongest competitor because it uses exactly the same teacher signal and differs only in the filter. If filtering helps, it must help here. No reported result found on Lacuna for a CLAP-distilled compact student on ESC-50, or for any context-swap evaluation of audio models, so we rebuild this baseline ourselves.

Fairness. Every variant gets the same student architecture, training clips, epochs, augmentation, and number of hyperparameter trials, with $\alpha$ and the temperature tuned on the same validation fold.

Ablations. Three ablations isolate the key ingredient:

  • a random concept mask with the same number of kept concepts per clip;
  • unfiltered sparse codes, which separates sparsity from filtering;
  • the inverse filter, which keeps only out-of-subtree concepts.

What counts as beating vanilla distillation. Before running anything, we set the bar as a context-swap drop reduced by at least a third, averaged over the 0–10 dB tiers, with clean accuracy within one point. The inverse filter must show a larger drop than vanilla distillation.

Why not just…?

Why not just augment training with random background mixing? This is the most serious threat. If isolated foreground events are available, pasting them onto random backgrounds could break the shortcut without any teacher. The catch is that it requires clean isolated events, while most real training audio, such as TUT2017 scenes or field recordings, has foreground and background entangled.

We include this augmentation as a privileged baseline that gets extra isolated-event data, and we test both regimes. If augmentation matches our method even when the student trains only on natural recordings, the filter adds nothing and we will report that. We also test whether augmentation and filtering combine additively.

Why not just linear-probe the sparse CLAP concepts and skip the student? Zhang et al. already show sparse probes work well. But a probe keeps CLAP at inference time, which defeats deployment, and it inherits CLAP's context concepts unless they are filtered. We run the filtered probe as an upper reference to separate how much robustness comes from the filter and how much is lost in compression.

Why not just use hand-designed features, which the feature-engineering literature says are often enough? MFCC models have no learned scene priors, so they may be naturally robust to context swaps but weaker clean. The MFCC baseline shows directly whether robustness can be had for free. Our claim is a better point on the clean-versus-swapped trade-off, not dominance on either axis alone.

Why not just find shortcuts with saliency maps and mask them? Mishra et al. show that spectrogram masking yields unstable, method-dependent attributions. Concepts give a discrete, nameable unit to filter, and the inverse-filter ablation tests whether they carry the shortcut.

Proposed Experiments

Main experiment. We train all variants on ESC-50 and US8K with standard folds and evaluate on clean test folds and on the four context-swap tiers. Results are split by flagged versus clean clips from the contamination audit. The decisive checks are three:

  • the filtered student meets the pre-set criterion against vanilla distillation;
  • the inverse filter worsens the swap drop, while the random mask with matched sparsity sits near vanilla;
  • the advantage persists on the contamination-clean split, which rules out the possibility that it only reflects the teacher's memorized clips.

Over-pruning stress test. TUT2017 scene classification is a case where context is the label. Here the ontology filter should hurt, and comparing it with the learned filter tells us whether the method can tell when context is causal. We will report the Zhang et al. figures (0.538 dense, 0.556 sparse) as teacher-side reference points.

Interpretation of outcomes. If the filter helps only in the 0–10 dB tiers, or only on flagged clips, we will say so. The first result would limit the claim to low-SNR conditions; the second would make it a memorization effect rather than a context one.