Lung-sound classifiers work better when they look at the same recording through several different time-frequency "lenses". Examples are the log-Mel spectrogram, the cochlea-inspired Gammatone spectrogram, and wavelet scalograms, which trade time resolution for frequency resolution differently. The catch is cost. Every extra lens adds a signal transform at the front end and a separate network behind it, which is hard to fit on a stethoscope or wearable. There is a second, clinical problem: lung-sound models are known to pick up recording-device and patient artifacts instead of the underlying physiology. So the advantage an ensemble shows on a benchmark may not survive a new patient or a new device.
We propose to train a single small network that reads the raw waveform and computes no spectrograms at test time. During training it learns from an ensemble of teachers, each built on a different transform. The key design choice is how much the student trusts each teacher. Trust is set per class by how well that teacher generalizes to patients it never saw, not by how confident it looks. The goal is ensemble-level sensitivity to crackles and wheezes under patient- and device-independent evaluation, at the cost of one one-dimensional network. We also ask honestly whether the multi-transform teachers teach the student anything that a raw-waveform model trained alone could not learn.
Core Claim
A raw-waveform student can recover most of the patient-independent benefit of a multi-transform ensemble on the ICBHI 2017 respiratory benchmark if it is trained with two signals. The first matches the student's output to a mixture of teacher predictions. In that mixture, each teacher's weight for each class is its accuracy on held-out patients. The second uses small training-only "bridge" layers that make the student's intermediate feature maps predictable from, and predictive of, each teacher's intermediate features. Both signals are discarded at inference.
The claim is falsifiable in two directions. If a raw-waveform model with a learnable filterbank, trained without teachers, matches the distilled student, then the teachers add nothing and the transforms are only a convenience. If the teacher ensemble does not itself beat its best member under patient- and device-held-out splits, then there is nothing worth distilling. We will report either outcome as a result.
Why This Is Not Already Done
Three pieces of prior work motivate the proposal and supply parts of the method, but none combines them.
Multi-representation knowledge distillation for audio classification trains parallel branches on MFCC, Mel and constant-Q inputs. It averages their soft labels into a teacher and deploys one branch. It supplies our starting recipe. It differs from our proposal in three ways: it transfers only averaged output probabilities, its deployed branch still computes a spectrogram, and it was tested only on generic audio (FSDKaggle2018, where it reports 72.48% accuracy and 93.26% mAP@3). Its authors also report that ensembling the distilled branches still beats any single branch, which is the distillation deficit we target.
CNN-MoE Based Framework for Classification of Respiratory Anomalies and Lung Disease Detection distills a gated mixture of spectrogram experts into a student with about one-seventh the parameters. That student matches the teacher at an ICBHI score of 0.84 on Task 2-1. The same paper shows that Gammatone features are best for cycle anomalies and log-Mel is best for disease labels. That finding is the empirical basis for per-class rather than global teacher weighting. Their student still needs a spectrogram front end, and its distillation stays within the spectral domain.
Customizing Student Networks From Heterogeneous Teachers via Adaptive Knowledge Amalgamation supplies two components: 1×1-convolution bridges that align features between architecturally different networks, and per-sample teacher selection by prediction entropy. It worked on vision benchmarks. We keep the bridges but replace entropy weighting. The reason is specific to our domain. A teacher that has latched onto a device artifact can be confidently wrong, so low entropy is a poor sign of trustworthiness under patient shift.
Two further results constrain the design. Inception-Based Network and Multi-Spectrogram Ensemble Applied To Predict Respiratory Anomalies and Lung Diseases found that late-averaging scalogram and Gammatone models did not beat a single Inception-02 model on disease prediction (both 0.87/0.85 Average/Harmonic Score). That result warns that the ensemble benefit may be small or task-dependent. Multi-View Spectrogram Transformer for Respiratory Sound Classification reached a 66.55% Average Score with 51.10% sensitivity by running five Transformer encoders over differently shaped patches of one Mel spectrogram. It shows the multi-view benefit is real on cycle classification but comes at roughly five times the encoder cost.
Research Program
Teachers. We train four teachers on ICBHI cycles, one each on log-Mel, Gammatone, constant-Q and Morse-wavelet scalogram inputs. All use the asymmetric Inception backbone with 1×4 frequency-oriented kernels from the Inception paper, so that differences between teachers come from the transform rather than the architecture.
Reliability weights. We split the official training patients into $K$ patient-disjoint folds and run cross-fitting inside the training set. This gives a per-class reliability $r_{k,c}$ for each teacher $k$ and class $c$, defined as that teacher's recall on class $c$ for patients it did not train on. The distillation target for a training cycle $x$ is
$$ q(c \mid x) \propto \sum_k r_{k,c}\, p_k(c \mid x), $$
where $p_k$ is teacher $k$'s predicted probability for class $c$. The target is renormalized over the classes. Each training cycle's target comes from teachers that did not see that cycle's patient. This prevents teachers from passing on memorized patient noise.
Student. The student reads the raw waveform, resampled to a common rate. Its first layer is a learnable band-pass filterbank that produces a time-frequency-like map. A small asymmetric convolutional stack follows. Training minimizes three terms: cross-entropy on labels, KL divergence to $q$, and a feature term. The feature term uses a lightweight bridge per teacher that maps a mid-level student feature map onto that teacher's corresponding feature map. Each teacher's map is first resampled to the student's time grid, since the transforms have different frame rates. The bridges are deleted after training.
Cost accounting. We report compute for the whole pipeline, not just the network. That means FLOPs and wall-clock latency including the front-end transforms, measured on a single CPU core as a stand-in for embedded hardware. The wavelet front end in particular is where the teacher ensemble pays most, so a network-only count would understate the ensemble's cost.
Baselines
The naive reference points are two single models trained with plain cross-entropy. One is the best single-transform teacher. The other is the raw-waveform student architecture trained with no teachers. The second is the key ablation: it has exactly the student's architecture, data, augmentation, epochs and tuning budget, so any gap between it and the distilled student isolates what the teachers transfer. A distilled student counts as useful only if it beats this model by more than the seed-to-seed spread, estimated with five seeds and patient-level bootstrap intervals.
The upper reference is the late-fusion ensemble of all four teachers, which averages probabilities at test time. It shows how much benefit exists to recover and what it costs.
The strongest existing distillation baseline is the cyclic averaged-soft-label method from Multi-representation knowledge distillation for audio classification. We will reimplement it on ICBHI with the same four teachers, deploying its best branch. It is the most direct competitor because it asks the same question with output-only transfer and a spectrogram front end. We also run the CNN-MoE teacher-student scheme under the same splits, to compare against the reported 0.84 Task 2-1 ICBHI score.
The strongest reported number on cycle classification in the file is the Multi-View Spectrogram Transformer: 66.55% Average Score and 51.10% sensitivity on ICBHI. On disease prediction it is Inception-02 at 0.87/0.85 Average/Harmonic Score. Protocols and class groupings differ across these papers. We will therefore rerun the multi-view Transformer under our exact splits rather than compare against its published figure, and cite the published figure only as context.
Ablations isolate the proposal's two ingredients. First, the teacher weighting is swapped among uniform averaging, entropy-based selection and held-out-patient reliability. Second, the feature bridges are removed.
We count the proposal as succeeding if three things hold on the patient-independent split and on device-held-out splits. First, the student's Average Score and sensitivity fall within the ensemble's bootstrap interval. Second, the student beats both the raw-alone model and the reimplemented cyclic-distillation baseline. Third, it does this at no more than the compute of one teacher, front end included.
Why not just…?
Why not just run the late-fusion ensemble? It multiplies front-end and network cost by the number of transforms. The Inception study also shows it gave no gain over a single model on disease prediction (0.87/0.85 for both). The ensemble stays in our study as the upper reference. If it does not beat its best member under held-out patients or devices, we will report that as the answer and drop the distillation claim.
Why not just train a raw-waveform model with a learnable filterbank directly? This is the most serious threat. In principle, a learnable filterbank can discover Gammatone-like or wavelet-like filters on its own, and ICBHI's roughly 6,900 cycles might be enough. The file contains no ICBHI result for raw-waveform models, so the evidence is missing rather than negative. The raw-alone ablation settles the question directly. We also inspect whether the distilled student's learned filters move toward the frequency responses of the teacher transforms, which would show what the teachers transferred.
Why not just distill into a Mel-input student, as prior work does? That removes the multi-network cost but keeps a spectrogram front end, and it transfers only averaged outputs. The Mel student is run as a separate arm, with our weighting and bridges applied to it as well. That separates the benefit of the raw-waveform input from the benefit of the training signal. If a Mel student with our weighting matches the raw student at acceptable front-end cost, we will recommend it instead.
Why not just use a larger pretrained spectrogram Transformer? Per the multi-view Transformer paper, a pretrained Audio Spectrogram Transformer with patch-mix contrastive learning scored 4.18 points below the multi-view model, so scale alone did not close the gap. It is also the opposite of what an edge device can run. We include it as a reference, not a competitor.
Why not just average weights across the teachers? The teachers take differently shaped inputs and do not share parameter layouts, so there is no meaningful way to merge their weights.
Decisive Checks
The first check is whether any ensemble benefit survives a hard split. We build leave-one-device-out splits from ICBHI's recording-equipment metadata, alongside the official patient-independent 60/40 split. On these we measure whether the teacher ensemble beats its best member on crackle and wheeze sensitivity. The benefit is most likely to show there, given that Gammatone and Mel specialize by task.
The second check is whether held-out-patient weighting beats entropy weighting specifically on held-out devices. That is the setting where confident artifact-driven teachers should mislead an entropy rule.
The third check is whether the raw-waveform student closes at least most of the gap between the raw-alone model and the ensemble, while costing no more than one teacher end to end.
We report cycle-level Average and Harmonic Scores and per-class sensitivity for Task 1, and Average and Harmonic Scores for Task 2. Every comparison is paired across the same seeds and patient bootstrap resamples.
References
- Multi-Representation Knowledge Distillation For Audio Classification
- CNN-MoE Based Framework for Classification of Respiratory Anomalies and Lung Disease Detection
- Customizing Student Networks From Heterogeneous Teachers via Adaptive Knowledge Amalgamation
- Inception-Based Network and Multi-Spectrogram Ensemble Applied To Predict Respiratory Anomalies and Lung Diseases
- Multi-View Spectrogram Transformer for Respiratory Sound Classification