Medical audio classifiers fail mostly on new patients, new microphones and new operating rooms. The class itself is often not the hard part. A crackle recorded on one chest with one stethoscope sounds different from a crackle on another chest, and small datasets such as the ICBHI 2017 respiratory database (126 subjects) contain too few such combinations for a model to learn what stays constant. We propose to create the missing combinations by morphing. Two recordings of the same class from different patients are encoded into the continuous internal space of a pretrained audio generator. We walk the path between them, decode the intermediate points back into spectrograms, and train the classifier on these decoded sounds. Because both endpoints share a label, the morphs need no label guesswork. What varies along the path is the recording context: body habitus, sensor, background. That is the variation a cross-patient classifier has to ignore.
The bet rests on a finding from the sound-morphing literature. Interpolating inside a generator's learned space produces a single coherent intermediate sound, whereas mixing the raw signals produces two sounds playing at once. Mixing in the classifier's own feature space produces abstract vectors that never correspond to any sound. Our core claim is narrow and testable. On held-out patients or procedures, decoded within-class latent morphs improve classification more than spectrogram mixup or feature-space interpolation do at the same synthetic-data ratio. The gain comes from the interpolation itself, not merely from passing audio through a generator's encoder and decoder.
What exists and what is missing
The idea that interpolating inside a generator's space yields plausible hybrids comes from timbre work. A WaveNet autoencoder on NSynth produced cohesive hybrid instruments by linear latent interpolation (Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders). SoundMorpher adapts the AudioLDM2 latent diffusion model (a generator that learns to turn noise into audio by gradual denoising, working in a compressed space) and chooses interpolation steps so that perceptual change is uniform. Both papers supply our method. Neither uses the morphs as training data for a classifier.
Three lines of work come closest to joining interpolation and augmentation, and each stops short in a different place. MusicLDM interpolates VAE latents after beat alignment, but only to regularize its own text-to-music generator. Its beat tracking also assumes meter, which breathing sounds and surgical noise lack. Generalizable Audio Deepfake Detection via Latent Space Refinement and Augmentation interpolates and extrapolates spoof embeddings toward learned prototypes and reaches 5.54% equal error rate on In-The-Wild. Its augmentations, however, live in the detector's feature space and are never decoded, so nobody can check whether they correspond to real sound. RepAugment injects Gaussian noise into minority-class representations of respiratory sounds and reaches a 61.51% ICBHI score with an Audio Spectrogram Transformer. It also does not traverse between real samples or decode.
The generative augmentation work on clinical audio motivates the proposal but takes a different route. Conditional Wasserstein GANs on surgical spectrograms (Conditional Generative Data Augmentation for Clinical Audio Datasets; Improved Techniques for the Conditional Generative Augmentation of Clinical Audio Data) beat signal-processing augmentation. They sample from random noise conditioned on a label and never interpolate between specific recordings. GeoECG shows that straight-line mixing of heartbeats produces impossible doubled QRS complexes. It fixes this with optimal-transport paths between whole classes rather than with a generator's decoder.
Two constraints from medical imaging shape the design. Too much synthetic data hurts: lung-nodule detection improved at 1× augmentation and degraded at 3× (Synthesizing Diverse Lung Nodules Wherever Massively). Synthetic content that disagrees with its label also hurts (Optimizing Synthetic Data for Enhanced Pancreatic Tumor Segmentation). We therefore sweep the synthetic ratio and keep interpolation within a class by default.
Choosing within-class morphing over cross-class morphing is also a response to a conflict in the literature over how to label intermediate samples. Ma et al. favour soft labels that reflect disease severity. Revisiting Interpolation Augmentation for Speech-to-Text Generation finds that linear labels cause ambiguity and need self-distillation to fix. Within-class morphing avoids the question for the main experiments, and we test cross-class morphing only as an ablation.
One subtlety needs stating up front. On ICBHI, the "Both" class is literally the co-occurrence of crackles and wheezes. For that class, additive spectrogram mixup of a crackle cycle and a wheeze cycle is physically reasonable, because adventitious sounds really do superimpose. We do not claim latent morphing beats mixup everywhere. We expect it to win where superposition is wrong, namely when combining two instances of the same sound type, and we will report per-class results to check this.
Research Program
The generator is the AudioLDM2 variational autoencoder (VAE) and vocoder at 16 kHz, the same backbone SoundMorpher adapts. We fine-tune it only for reconstruction, without labels, on the training fold of each dataset. This narrows the gap that RepAugment documents between general-audio models and lung sounds.
Before any augmentation, we audit the reconstructions. Crackles are brief transients, and the known failure modes of VAE vocoding are smearing and phase artifacts. If a frozen classifier trained on real data loses more than a few points of crackle sensitivity on reconstructed held-out cycles, the decoder is the bottleneck. In that case we stop and report it.
For each class, we pair training recordings drawn from different patients (ICBHI) or different procedures (surgical audio). We encode each pair and decode intermediate points at several interpolation weights. The weights are spaced either uniformly or by SoundMorpher's perceptual-distance binary search. For sounds with mismatched temporal envelopes, we first time-align the two cycles by their onset envelopes. This is a lightweight stand-in for the envelope-morphing work of Learning Perceptually Relevant Temporal Envelope Morphing, which handles envelopes but not timbre.
Decoded spectrograms are appended to clean batches, following the appending finding of the speech-to-text interpolation study. They keep the shared hard label. We sweep the synthetic ratio from 0.5× to 3×.
Baselines
The naive reference is the same classifier with no augmentation. On ICBHI this means an Audio Spectrogram Transformer fully fine-tuned, which RepAugment and LungAdapter report at a 59.55% 4-class score; their AST with SpecAugment scores the same. On the hip-arthroplasty surgical dataset, the reference is ResNet-18 at 93.90% macro F1 under 5-fold cross-validation.
The strongest existing augmentation competitor on ICBHI is RepAugment. It is cheap, it targets minority classes, and it already beats SpecAugment, with a reported 61.51% 4-class score. We rerun it on the identical AST, split and seeds rather than copying the number.
On surgical audio, the strongest competitor is the SE-ResNet conditional Wasserstein GAN, reported at 96.74% macro F1. We retrain it on our spectrogram front end at the same synthetic ratio. Because our generator works from raw 16 kHz audio rather than the original 64×64 spectrograms, the comparison is fair only if both methods feed the same classifier input. We will resize our decoded output to that grid.
The parameter-efficient LungAdapter, reported at 62.40% 4-class and 69.53% 2-class, is the strongest ICBHI number in the file. Its approach is complementary to ours rather than a competitor, so we test our augmentation on top of it as well.
We also run feature-space interpolation in the style of the deepfake work. This means within-class linear interpolation of AST embeddings, without decoding. It is the cheapest version of our idea and the one most likely to match it.
Two ablations isolate the key ingredient. The first is reconstruction-only augmentation: encode and decode each real sample with an interpolation weight of zero. This tests whether gains come merely from the decoder acting as a learned perturbation. The second is cross-class morphing with soft labels, which tests whether the within-class restriction matters.
All methods get the same training data, epochs, classifier tuning budget (an equal number of hyperparameter trials) and synthetic ratio sweep. Beating a baseline means a gain on held-out patients that exceeds seed-to-seed spread across at least five seeds.
Why not just…?
Why not just use spectrogram mixup? Mixup helped ICBHI in Ma et al. However, morphing studies (SoundMorpher, MusicLDM) and GeoECG all document that additive blends produce overlaid or doubled events rather than intermediates. Two crackle cycles mixed additively give a cycle with twice the crackles at mismatched times. That is a different sound, not a different patient. Mixup is still a real threat for the "Both" class, as noted above. The per-class comparison decides this, and we expect mixup to remain competitive there.
Why not just interpolate in the classifier's feature space and skip the decoder? This is the most serious alternative. It costs almost nothing and worked for deepfake detection. Its weakness is that nothing keeps the interpolated vectors near real sounds, and the vectors cannot be listened to or audited. If feature-space interpolation matches decoded morphs on held-out patients, the generator is not earning its cost. We would report that as the main finding.
Why not just sample a conditional GAN? The surgical audio papers show conditional GANs work, but random samples carry no guarantee of covering the gaps between specific patients or procedures. The leave-one-procedure-out experiment tests this directly.
Why not just use parameter-efficient fine-tuning? LungAdapter is the best reported ICBHI result in the file. It changes how the model learns from data, not what data the model sees. We test our augmentation on top of it, and we would regard a gain there as the more convincing outcome.
Decisive Checks
The first check is the patient-wise official ICBHI split, which already holds out patients. We report Sensitivity, Specificity and ICBHI Score on the 4-class and 2-class tasks, plus per-class accuracy.
The second check is a leave-one-procedure-out protocol on the five-procedure surgical dataset. This is closer to deployment than the published 5-fold split. We found no reported result on Lacuna for leave-one-procedure-out on that dataset, so all methods will be rerun under it.
The proposal succeeds if decoded within-class morphs beat both feature-space interpolation and the reconstruction-only control. These comparisons are made at each method's best synthetic ratio, with the advantage concentrated on held-out patients or procedures rather than within-distribution folds. It fails informatively in two cases. If reconstruction-only augmentation matches the morphs, the decoder is simply acting as a learned noise source. If the reconstruction audit shows crackles are lost, current audio decoders are not yet faithful enough for this use.
References
- Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders
- SoundMorpher: Perceptually-Uniform Sound Morphing with Diffusion Model
- MusicLDM: Enhancing Novelty in text-to-music Generation Using Beat-Synchronous mixup Strategies
- Generalizable Audio Deepfake Detection via Latent Space Refinement and Augmentation
- RepAugment: Input-Agnostic Representation-Level Augmentation for Respiratory Sound Classification
- Conditional Generative Data Augmentation for Clinical Audio Datasets
- Improved Techniques for the Conditional Generative Augmentation of Clinical Audio Data
- GeoECG: Data Augmentation via Wasserstein Geodesic Perturbation for Robust Electrocardiogram Prediction
- Synthesizing Diverse Lung Nodules Wherever Massively: 3D Multi-Conditional GAN-Based CT Image Augmentation for Object Detection
- Optimizing Synthetic Data for Enhanced Pancreatic Tumor Segmentation
- LungRN+NL: An Improved Adventitious Lung Sound Classification Using Non-Local Block ResNet Neural Network with Mixup Data Augmentation
- Revisiting Interpolation Augmentation for Speech-to-Text Generation
- Learning Perceptually Relevant Temporal Envelope Morphing
- LungAdapter: Efficient Adapting Audio Spectrogram Transformer for Lung Sound Classification