Variational Bayesian Adaptive Learning of Deep Latent Variables for Acoustic Knowledge Transfer

Hu Hu, Sabato Marco Siniscalchi, Chao-Han Huck Yang, Chin‐Hui Lee

Abstract

Deep neural networks (DNNs) often struggle with domain shift in acoustic tasks, where models trained on one device or environment fail in another. This paper proposes Variational Bayesian Knowledge Transfer (VBKT), a framework that adapts models by focusing on deep latent variables rather than the vast parameter space of the network. By treating these intermediate representations as probability distributions and employing variational inference, VBKT achieves state-of-the-art performance in acoustic scene classification and speech command recognition with minimal adaptation data.

Introduction: The Challenge of Domain Shift

Modern acoustic models are highly sensitive to changes in recording conditions. A model optimized for high-fidelity studio audio often experiences a catastrophic drop in accuracy when deployed on consumer devices or in noisy environments. This "domain shift" is a significant barrier to the real-world deployment of deep learning models.

As shown in the table above, standard architectures like ResNet see their accuracy plummet from ~79% to ~37% when moved from a known source device to unseen target devices. Traditional adaptation methods, such as fine-tuning, often require large amounts of labeled data and risk "catastrophic forgetting." While Bayesian methods offer a robust alternative for uncertainty estimation, applying them to millions of DNN parameters is computationally prohibitive. VBKT addresses this by performing Bayesian adaptation on a low-dimensional "bottleneck": the deep latent variables.

Methodology

The core innovation of VBKT is shifting the focus from model weights to deep latent variables ($Z$), the intermediate hidden embeddings produced by the network.

The VBKT Framework

The framework uses Variational Bayes to approximate the posterior distribution of latent variables. It encodes source-domain knowledge into a prior and combines it with target-domain likelihood from a small amount of new data. Two primary estimation strategies are employed:

  1. Gaussian Mean-Field Variational Inference (GMFVI): Utilized when parallel data (identical audio recorded by different devices) is available.
  2. Empirical Bayes (EB): Utilized for non-parallel data, estimating hyperparameters directly from the source data.

To further improve performance, the authors introduced VBKT-rela, which uses a relational distance term (Huber loss) to ensure the internal geometry of the latent space—how different classes relate to one another—is preserved during transfer.

Experimental Evaluation

The researchers evaluated VBKT on two distinct tasks:

  • Acoustic Scene Classification (ASC): Using the DCASE 2020 dataset, which includes 10 scenes recorded across 9 different physical and simulated devices.
  • Spoken Command Recognition (SCR): Using a 10-class subset of the Google Speech Command Dataset v2, with added background noise at various signal-to-noise ratios (SNRs).

Key Results and Insights

  • Superior Accuracy: In the ASC task, the VBKT-GMF-rela variant boosted accuracy from ~35% back up to ~70%, outperforming 13 other cutting-edge transfer methods, including standard Teacher-Student Learning (TSL).
  • Noise Robustness: In the SCR task, VBKT-EB-rela achieved 93.36% average accuracy, significantly higher than the "No-transfer" baseline of 53.91%.
  • Distributional Benefits: Unlike point-estimation methods, VBKT considers the entire distribution (mean and variance), capturing the inherent uncertainty in noisy acoustic data.
  • Structural Preservation: The success of the rela variants demonstrates that maintaining the relative positioning of classes in the latent space is crucial for preventing model confusion during adaptation.

Conclusion

VBKT provides a data-efficient and mathematically grounded framework for acoustic domain adaptation. By focusing on deep latent variables, it bypasses the curse of dimensionality associated with full-parameter Bayesian adaptation. Future work will explore learning target variance dynamically and extending the framework to more complex tasks like full-scale automated speech recognition (ASR).

Summary updated

Abstract

In this work, we propose a novel variational Bayesian adaptive learning approach for cross-domain knowledge transfer to address acoustic mismatches between training and testing conditions, such as recording devices and environmental noise. Different from the traditional Bayesian approaches that impose uncertainties on model parameters risking the curse of dimensionality due to the huge number of parameters, we focus on estimating a manageable number of latent variables in deep neural models. Knowledge learned from a source domain is thus encoded in prior distributions of deep latent variables and optimally combined, in a Bayesian sense, with a small set of adaptation data from a target domain to approximate the corresponding posterior distributions. Two different strategies are proposed and investigated to estimate the posterior distributions: Gaussian mean-field variational inference, and empirical Bayes. These strategies address the presence or absence of parallel data in the source and target domains. Furthermore, structural relationship modeling is investigated to enhance the approximation. We evaluated our proposed approaches on two acoustic adaptation tasks: 1) device adaptation for acoustic scene classification, and 2) noise adaptation for spoken command recognition. Experimental results show that the proposed variational Bayesian adaptive learning approach can obtain good improvements on target domain data, and consistently outperforms state-of-the-art knowledge transfer methods.
BibTeX
@article{hu2025variational,
  author = {Hu Hu and Sabato Marco Siniscalchi and Chao-Han Huck Yang and Chin‐Hui Lee},
  title = {{Variational Bayesian Adaptive Learning of Deep Latent Variables for Acoustic Knowledge Transfer}},
  journal = {IEEE Transactions on Audio Speech and Language Processing},
  year = {2025},
  doi = {10.1109/taslpro.2025.3530321},
}
Figure