Sparse auto-associative neural networks: theory and application to speech recognition

Garimella S. V. S. Sivaram, Sriram Ganapathy, Hynek HermanskyInterspeech 2010 · DOI

Introduction

In the field of speech recognition, the representation of audio signals—known as feature extraction—is a critical component of system performance. Traditional methods for data compression and feature extraction, such as Principal Component Analysis (PCA) or standard Auto-associative Neural Networks (AANNs), often rely on a physical "bottleneck" layer to capture essential data structures. However, standard AANNs with a single bottleneck layer are mathematically constrained and cannot outperform linear Singular Value Decomposition (SVD).

This paper introduces the Sparse Auto-associative Neural Network (SAANN), which leverages the principle of sparsity to learn robust representations. By encouraging only a few features to be active at any given time, the SAANN provides a more effective unsupervised method for extracting speech features, particularly in challenging acoustic environments.

Methodology

The SAANN is designed as a three-layer feedforward network with an equal number of input and output nodes ($N$) and a hidden layer ($M$). Unlike traditional bottleneck architectures, the hidden layer $M$ does not need to be smaller than $N$.

The SAANN architecture featuring a sparse hidden layer α.

The Sparsity Constraint

The core innovation of the SAANN is its specialized cost function. Instead of minimizing only the reconstruction error, the authors added a logarithmic penalty term to enforce sparsity among the hidden neurons:

$$L = \text{Reconstruction Error} + \frac{\lambda}{2} \sum \log(1 + \alpha_j^2)$$

In this equation, $\alpha_j$ represents the activation of the hidden neurons. This penalty encourages most neurons to remain near zero, forcing the network to become highly selective.

Training and Feature Extraction

The network is trained in an unsupervised manner to map an input (consisting of 9 frames of PLP speech coefficients) back to itself. Once training is complete, the output layer is discarded, and the activations of the sparse hidden layer are utilized as the new features for a phoneme recognizer. This approach is computationally efficient; once the weights are learned, feature extraction is as fast as a simple matrix multiplication followed by a nonlinearity.

Experimental Setup

The researchers evaluated the SAANN using the TIMIT acoustic-phonetic corpus, a standard benchmark for phoneme recognition. The dataset included 3,000 training utterances and 1,344 test utterances. To test environmental resilience, the system was also evaluated against "babble noise" from the NOISEX-92 dataset at signal-to-noise ratios (SNR) of 10dB, 15dB, and 20dB.

Key Results

The SAANN features consistently outperformed the baseline Perceptual Linear Prediction (PLP) features across all tested conditions.

Phoneme recognition accuracy (%) for PLP and SAANN features under various noise conditions.

As shown in the results, SAANN achieved a 70.0% accuracy on clean speech compared to 68.2% for the PLP baseline. Notably, the performance gap widened in noisy conditions. At a 10dB SNR, SAANN maintained a 1% absolute lead over PLP, suggesting that sparse representations are significantly more resilient to environmental interference. In more advanced hierarchical recognition setups, the SAANN reached an accuracy of 72.1%, representing a 5.1% relative improvement in error rate over the baseline.

Discussion and Insights

A key finding of this research is that physical bottlenecks are not strictly necessary to prevent a network from learning a trivial identity mapping. By penalizing activity through sparsity, a large hidden layer can still be forced to learn meaningful, non-redundant features.

The authors also highlighted the importance of the sparsity weight ($\lambda$). When $\lambda$ was set to zero (reverting to a standard AANN), performance dropped significantly. Optimal results were achieved when $\lambda$ was approximately 0.5 to 1.0, confirming that the sparsity constraint is the primary driver of improved feature quality.

Limitations and Future Work

While the SAANN shows great promise, this study focused on a single hidden layer. Future research could explore "Deep SAANNs" to see if hierarchical sparsity yields even better results. Additionally, while the current model was trained on top of PLP features, future iterations might benefit from learning representations directly from raw filterbank outputs or waveforms.

Summary updated

BibTeX
@inproceedings{sivaram2010sparse,
  author = {Garimella S. V. S. Sivaram and Sriram Ganapathy and Hynek Hermansky},
  title = {{Sparse auto-associative neural networks: theory and application to speech recognition}},
  booktitle = {Interspeech},
  year = {2010},
  doi = {10.21437/interspeech.2010-623},
}
Figure