Hermitian based Hidden Activation Functions for Adaptation of Hybrid HMM/ANN Models

Sabato Marco Siniscalchi, Jinyu Li, Chin-Hui Lee

Introduction

In Automatic Speech Recognition (ASR), models are typically trained on a wide variety of speakers to achieve "Speaker Independence" (SI). However, the unique vocal characteristics of individual users often lead to performance degradation. To mitigate this, researchers employ "Speaker Adaptation" to transform SI models into "Speaker Dependent" (SD) models.

Traditional adaptation techniques for Multi-Layer Perceptrons (MLPs) in hybrid HMM/ANN systems often involve adding auxiliary layers, such as a Linear Input Network (LIN) or Linear Hidden Network (LHN). While effective, these methods introduce architectural mismatches between SI and SD models and tie the number of adaptation parameters strictly to the dimensions of the input or output layers. This paper proposes a novel approach: adapting the shape of the hidden activation functions themselves using orthonormal Hermite polynomials.

The Hermitian-Based MLP (HMLP)

The core innovation of this research is the Hermitian-Based MLP (HMLP). Unlike standard neural networks that use fixed activation functions like Sigmoids or ReLUs, the HMLP utilizes a flexible function composed of a weighted sum of orthonormal Hermite polynomials.

The architecture of the HMLP, highlighting the hidden units $f_i$ where the Hermitian activation functions are applied.

The activation function $f_j(z)$ for a hidden unit $j$ is defined as: $$f_j(z) = \sum_{r=1}^{R} c_{jr} h_r(z)$$ where $h_r$ represents the Hermite functions and $c_{jr}$ are learnable coefficients. During the adaptation phase, the primary weights of the neural network remain frozen. Only the coefficients ($c_{jr}$) of the Hermite polynomials are updated via standard backpropagation. This strategy ensures that the network architecture remains identical between SI and SD versions while allowing for precise control over the number of adaptation parameters by adjusting the polynomial degree ($R$).

System Architecture and Feature Extraction

The HMLP is integrated into a standard hybrid HMM/ANN pipeline. The process begins with raw speech signals, which undergo feature extraction to produce Mel-bank outputs. These are then buffered, windowed, and transformed via Discrete Cosine Transforms (DCTs) before being fed into the MLP classifier.

Overview of the hybrid HMM/ANN speech recognition pipeline, from raw speech to MLP classification.

Experimental Setup

The proposed system was evaluated using the Wall Street Journal (WSJ0) corpus:

  • Training: The SI84 set, comprising approximately 15.3 hours of speech from 84 speakers.
  • Adaptation: The si_et_ad set, featuring 8 speakers with 40 utterances each.
  • Evaluation: The Nov92 evaluation set, containing 330 utterances from 8 speakers.

Key Results

The HMM/HMLP system demonstrated significant improvements over standard baselines. By decoupling the adaptation parameter budget from the network geometry, the authors utilized only 24,000 parameters—far fewer than the 85,000 to 300,000 required by traditional LIN methods.

System SI WER (Baseline) SD WER (Adapted) Relative Improvement
HMM/HMLP (Proposed) 6.4% 5.5% 14.1%
LOQ-1 (Standard) 8.4% 7.9% 5.9%
LOQ-2 (Standard) 6.5% 5.6% 13.8%

The Hermitian approach achieved the lowest overall Word Error Rate (WER) of 5.5%, representing a 14.1% relative improvement over its SI baseline. This suggests that the shape of the non-linearity is a powerful lever for domain adaptation.

Conclusion and Future Work

This research highlights the efficacy of using flexible, Hermitian-based activation functions for speaker adaptation. By maintaining architectural consistency and providing a tunable parameter space, the HMLP offers a robust alternative to traditional linear network adaptation. Future work will explore scaling this approach to Large Vocabulary Continuous Speech Recognition (LVCSR) tasks, implementing conservative training techniques to prevent catastrophic forgetting, and applying these principles to deeper neural network architectures.

Summary updated

Abstract

This work is concerned with speaker adaptation techniques for artificial neural network (ANN) implemented as feed forward multi-layer perceptrons (MLPs) in the context of large vocabulary continuous speech recognition (LVCSR). Most successful speaker adaptation techniques for MLPs consist of augmenting the neural architecture with a linear transformation network connected to either the input or the output layer. The weights of this additional linear layer are learned during the adaptation phase while all of the other weights are kept frozen in order to avoid over-fitting. In doing so, the structure of the speaker-dependent (SD) and speaker-independent (SI) architecture differs and the number of adaptation parameters depends upon the dimension of either the input or output layers. We propose a more flexible neural architecture for speaker-adaptation to overcome the limits of current approaches. This flexibility is achieved by adopting hidden activation functions that can be learned directly from the adaptation data. This adaptive capability of the hidden activation function is achieved through the use of orthonormal Hermite polynomials. Experimental evidence gathered on the Nov92 task demonstrates the viability of the proposed technique.
BibTeX
@inproceedings{siniscalchi2012hermitian,
  author = {Sabato Marco Siniscalchi and Jinyu Li and Chin-Hui Lee},
  title = {{Hermitian based Hidden Activation Functions for Adaptation of Hybrid HMM/ANN Models}},
  booktitle = {Interspeech},
  year = {2012},
  doi = {10.21437/interspeech.2012-13},
}
Figure