TL;DR
S-SONDO is the first self-supervised knowledge distillation framework for general audio foundation models that operates entirely on output embeddings. By eliminating the need for class logits or complex layer-by-layer alignment, it compresses massive transformer-based models by up to 61× while retaining up to 96% of the teacher's performance, paving the way for efficient edge deployment.
What Problem Does This Paper Address?
Self-Supervised Learning (SSL) has revolutionized audio representation, yielding powerful foundation models capable of handling diverse tasks from environmental sound classification to music tagging. However, these state-of-the-art models are massive—often containing hundreds of millions of parameters—making them computationally expensive and impractical for resource-constrained edge devices like mobile phones or embedded systems.
While Knowledge Distillation (KD) is a standard approach to compress large "teacher" models into smaller "student" models, existing audio KD methods typically rely on supervised settings (matching class logits) or require strict architectural similarities to align intermediate layers. These assumptions fail for modern SSL foundation models that only output feature embeddings and lack explicit classification layers. S-SONDO addresses this gap by establishing an architecture-agnostic framework for embedding-only distillation in the general audio domain.
Methods
The S-SONDO framework adopts a response-based, self-supervised distillation approach that directly aligns a student model's output representations with those of a pre-trained teacher model.
The pipeline consists of three main components:
- Mapping Head: Because student and teacher embedding dimensions usually differ ($ds \neq dt$), the authors introduce a simple Multi-Layer Perceptron (MLP) mapping head with one hidden layer. This projects the student's representation into the teacher's well-organized, pre-trained latent space.
- Distillation Loss: The student is trained by minimizing a distance metric between its projected embedding ($Z'_s$) and the teacher's embedding ($Zt$). The authors evaluated several loss functions (including MSE, $\ell_1$, KL Divergence, and CLAP), finding that Cosine Similarity loss yielded the most robust performance.
- Balanced Data Sampling (BDS): To counteract data imbalance without relying on ground-truth labels, S-SONDO clusters the teacher's embeddings using $k$-means to generate unsupervised "pseudo-labels." It then samples data inversely proportional to cluster frequency, preventing the student from overfitting to overrepresented audio concepts.
Datasets
- Pre-training: The models were distilled using AudioSet, restricted to 10-second clips, totaling approximately 1.8 million audio samples.
- Downstream Evaluation: To test generalizability, the distilled students were evaluated across seven diverse audio benchmarks:
- Music Tasks: OpenMIC (instrument recognition), NSynth (musical note synthesis), GTZAN (genre classification), and MTT (music tagging).
- Environmental Tasks: FSD50K (sound events), ESC-50 (environmental sounds), and US8K (urban sounds).
Key Insights and Ideas
- Final Embeddings are Enough: A key "aha" moment is that aligning only the final output embeddings provides a sufficiently rich training signal. Complex, computationally heavy layer-wise alignment strategies are unnecessary for transferring general audio knowledge.
- Direction Over Coordinates: Cosine similarity significantly outperforms element-wise losses like MSE or $\ell_1$. This is because semantic information in self-supervised latent spaces is inherently encoded via relative directions (angles) rather than absolute spatial coordinates.
- The Smaller the Student, the More BDS Helps: Unsupervised Balanced Data Sampling proved critical for lower-capacity students (like ERes2Net), which failed to converge when distilling from large teachers without it. BDS acts as a regularizer that prevents representation collapse.
- Clustering Struggles with Multi-Label Audio: BDS provided substantial boosts for single-label datasets (like ESC-50 and GTZAN) but was less effective for multi-label tasks. This suggests that standard $k$-means clustering struggles to capture the overlapping semantic nuances present in complex, multi-label audio scenes.
Key Results
S-SONDO successfully distilled two massive 86M-parameter transformer teachers (MATPAC++ and M2D) into three lightweight student architectures: MobileNetV3 (2.9M), DyMN (8.7M), and ERes2Net (1.4M).
- Outperforming Supervised Baselines: In 4 out of 6 teacher-student configurations, the distilled self-supervised students outperformed counterparts trained directly via standard supervised learning on the same architecture.
- Massive Compression, Minimal Loss: The best-performing pair—MATPAC++ distilled into MobileNetV3—retained 96.4% of the teacher's performance while being roughly 30× smaller.
- Extreme Efficiency: When distilling into ERes2Net, the model size was slashed by 61× (from 86M down to 1.4M parameters) while still maintaining 93.5% of the teacher's baseline performance.
Limitations and Future Work
While S-SONDO sets a new baseline for self-supervised audio distillation, the authors note a few limitations:
- Contrastive Bottlenecks: Contrastive objectives like the CLAP loss underperformed relative to Cosine similarity due to batch size constraints (batch size of 64) and a lack of explicit negative pair control, which occasionally forced semantically similar samples apart.
- Simplistic Clustering: The use of standard $k$-means for BDS limits performance on multi-label audio datasets.
Future Directions: The authors plan to explore advanced clustering techniques that better accommodate multi-label semantic structures. They also aim to refine contrastive distillation objectives by implementing smarter positive and negative pair selection mechanisms to prevent representation collapse around semantically close negatives.
Summary updated