Hierarchical neural net architectures for feature extraction in ASR

František Grézl, Martin KarafiátInterspeech 2010

TL;DR

This paper explores hierarchical neural network architectures for speech recognition, specifically focusing on improving "bottle-neck" features. By stacking multiple layers of these networks, the researchers demonstrate consistent improvements in speech recognition accuracy, suggesting that a hierarchical approach can better capture discriminative speech information.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems rely heavily on effective feature extraction to convert raw audio signals into representations that can be understood by acoustic models. Traditional methods often struggle to capture the full discriminative power of speech. While "bottle-neck" (BN) features, derived from a compressed layer within a neural network, have shown promise by outperforming standard probabilistic features, their full potential within more complex architectures remains an area of exploration. This paper investigates how to leverage hierarchical neural network structures to further enhance the discriminative capabilities of these BN features for ASR, particularly for challenging tasks like meeting speech recognition.

Methods

The core of this work is the design and evaluation of hierarchical neural network architectures for feature extraction. The authors build upon the concept of Bottle-Neck (BN) features, which are linear outputs from a compressed layer in a neural network.

The proposed architecture consists of two levels:

  1. First Level Classifiers: These networks take Mel-scaled Critical Band Energies (CRBE) as input. Three different architectures are explored for this level:
  2. Five-layer ANN: A standard five-layer neural network with a bottle-neck layer.
  3. Split Context (SC) ANN: This architecture uses three ANNs. The input is split into left and right contexts, processed individually, and then merged.
  4. Universal Context (UC) ANN: This involves a tandem of two ANNs. A contextual ANN processes blocks of CRBE, and its outputs are then fed into a merger ANN. Variations with 3 or 5 splits of the input block were tested.
  5. Second Level Classifier: This is a five-layer ANN with a bottle-neck layer. It takes the BN features from the first level classifier, along with the original input features (CRBE processed with Hamming window and DCT), as its input.

A key aspect of the hierarchical approach is how information from the first level is presented to the second. Options include:

  • Using only the current frame's BN output.
  • Concatenating BN outputs from several consecutive frames (current frame ±2 frames).
  • Concatenating BN outputs with their delta parameters.

Datasets

  • Training Data: Approximately 180 hours of meeting speech data from the complete NIST, ISL, AMI, and ICSI corpora. For ANN training, 173 hours of speech were used.
  • Test Data:
    • NIST RT'05 evaluation set.
  • NIST RT'07 evaluation set.
  • The "independent head set microphone (IHM)" condition was used for both test sets.

Key Insights and Ideas

  • Hierarchy Boosts Discriminative Power: The core idea that hierarchical structures can improve feature discriminability is validated. Stacking layers of bottle-neck feature extractors consistently leads to better performance than using a single layer.
  • Context Matters, But How You Use It Is Crucial: More complex first-level classifiers (like Universal Context) that better capture temporal context generally achieve higher frame accuracy. However, simply adding more temporal information (like stacking many frames) to the second level doesn't always help, especially with already complex first-level structures.
  • Universal Context (UC) Architecture Shines: The UC architecture, particularly the UC5 variant (using 5 splits of the input block), demonstrated the best performance, even outperforming other hierarchical setups when not using additional temporal information. This suggests its method of splitting and processing context is highly effective.
  • Skipping the Merger Can Simplify and Improve: Omitting the merger ANN in the first level (e.g., directly using contextual BN outputs from SC or UC) led to improved performance and a simpler architecture, especially for SC and UC3. This indicates that the merger might sometimes over-compress information.
  • Delta Features are Key for Temporal Evolution: When temporal information is beneficial, delta parameters of the first-level BN outputs proved more effective than simply stacking multiple frames, particularly when the merger was skipped.

Key Results

  • Significant WER Reduction: The hierarchical architectures consistently outperformed the baseline HLDA-PLP features. For example, the best hierarchical setup (UC5 BN + 31 CRBE 16 DCT) achieved a Word Error Rate (WER) of 22.7% on RT'05 and 29.5% on RT'07. This is a substantial improvement over the baseline's 27.6% (RT'05) and 36.0% (RT'07).
  • Contextual BN Features Outperform Baseline: Even the first-level BN features (e.g., ANN BN, SC BN, UC3 BN, UC5 BN) concatenated with original features showed significant improvements over the baseline HLDA-PLP features.
  • UC5 Architecture Dominates: The Universal Context (UC5) architecture, when used as the first level classifier, consistently yielded the best results across various configurations.
  • Overall Improvement: The paper claims an absolute improvement of 1% on RT'05 and 1.5% on RT'07 by using the hierarchical architecture compared to just a single classifier.

Limitations and Future Work

  • Non-Optimal LM Scale Factor/Word Insertion Penalty: In one instance, the hierarchical structure did not improve performance, which the authors attribute to non-optimal language model scaling and word insertion penalty tuning.
  • UC5 Temporal Information Issue: For the UC5 structure, adding temporal evolution information (stacked frames or deltas) did not yield further improvements and sometimes hurt performance. The authors suggest this is due to the UC5's inherent complexity and how its overlapping blocks are processed, making temporal shifts less informative after merger.
  • Future Directions:
    • Investigating how to better represent temporal evolution information, especially for complex architectures like UC5.
  • Exploring more sophisticated ways to split input blocks and process them in the hierarchy.
  • The authors suggest that the UC structure's success implies that input blocks should be split into more, largely overlapping blocks, processed to get compressed representations, and then appended to original features for the second level classifier.
  • Careful consideration is needed when choosing the means of representing temporal information to avoid redundancy.

Summary updated

BibTeX
@inproceedings{grezl2010hierarchical,
  author = {František Grézl and Martin Karafiát},
  title = {{Hierarchical neural net architectures for feature extraction in ASR}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure