Low Resource Automatic Intonation Classification Using Gated Recurrent Unit (GRU) Networks Pre-Trained with Synthesized Pitch Patterns

Atreyee Saha, Chiranjeevi Yarra, Prasanta Kumar GhoshInterspeech 2019

TL;DR

This paper tackles the challenge of classifying intonation patterns in British English for second language learners, a task typically requiring extensive expert-annotated data. The authors propose a novel approach using Gated Recurrent Units (GRUs) pre-trained on synthesized pitch patterns derived from domain knowledge, significantly improving classification accuracy in low-resource scenarios.

What Problem Does This Paper Address?

Learning correct intonation is crucial for second language (L2) learners to avoid miscommunication. British English (BE) has four primary intonation patterns: Glide-up, Glide-down, Dive, and Take-off. Automatically classifying these patterns is challenging due to the subtle temporal dependencies in pitch and the scarcity of expertly labeled data. Traditional methods like Hidden Markov Models (HMMs) struggle with these complexities, and even standard deep learning models require large datasets, which are difficult and expensive to obtain for intonation annotation. This paper aims to bridge this gap by developing a robust intonation classification system that performs well with limited data.

Methods

The proposed approach involves three main stages:

  1. Feature Computation: A 3-dimensional (3D) feature sequence is extracted from speech. This sequence includes:
  2. Estimated pitch contour in Mel scale, normalized for mean and range.
  3. The derivative of the pitch contour.
  4. A confidence score for pitch estimation, hypothesized to help the model learn to suppress pitch estimation errors.
  5. Pre-training: This is the core innovation for low-resource scenarios.
  6. Tone Sequence Fetching: Instead of relying on potentially error-prone automatic tone sequence extraction, the authors leverage domain knowledge to define representative tone sequences for each of the four intonation classes.
  7. Artificial Feature Synthesis: Using these knowledge-based tone sequences, a 2D artificial feature sequence is synthesized. This synthesis mimics the characteristics of real pitch contours, including variations based on tone types (Rise, Fall, Low, Mid, High) and durations. White Gaussian noise is added to enhance robustness.
  8. GRU Network Initialization: A GRU network is trained on these synthesized features to obtain initial parameters. This pre-trained model serves as a better starting point for the final classification task.
  9. Classification:
    • A Time-Distributed Layer (TDL) is used to transform the 3D input feature sequence into a 2D sequence. This layer is designed to suppress unwanted variations in the pitch contour, guided by the confidence scores.
  10. A Gated Recurrent Unit (GRU) network, initialized with parameters from the pre-training stage, processes the 2D sequence from the TDL. GRUs are chosen over LSTMs due to their fewer parameters and comparable or better performance in this task.
  11. The output of the GRU is fed into a softmax layer to predict the posterior probability of each of the four intonation classes.

Datasets

The experiments were conducted on speech data collected from a spoken English training material used for teaching British English intonation. This dataset contains utterances annotated with intonation class labels and tone sequences. Specifically, the dataset comprises 233 utterances, distributed as follows: 50 for Glide-up, 68 for Glide-down, 82 for Dive, and 33 for Take-off. The data was spoken by one male and one female native BE speaker. The authors note that this is a relatively small dataset, highlighting the need for low-resource methods.

Key Insights and Ideas

  • Leveraging Domain Knowledge for Synthesis: The paper demonstrates that synthesizing realistic pitch patterns from abstract tone sequences, guided by expert knowledge, can effectively create training data for pre-training deep learning models. This circumvents the need for large amounts of manually transcribed real speech.
  • Pre-training for Low-Resource Scenarios: Pre-training a GRU network with synthesized data significantly boosts performance, proving the value of this technique when labeled data is scarce.
  • TDL for Error Suppression: The Time-Distributed Layer acts as a crucial component for filtering out noise and estimation errors in the pitch contour, leading to more reliable feature representations for the GRU.
  • GRU's Efficiency: GRUs prove to be a suitable choice for this task, offering a good balance between performance and computational efficiency compared to LSTMs, especially in low-resource settings.

Key Results

The proposed approach, combining a TDL and GRU pre-trained with synthesized pitch patterns, achieved significant improvements in Unweighted Average Recall (UAR) compared to a baseline method (Yarra et al., 2018) and the proposed method without pre-training.

  • Absolute Improvement: The proposed scheme with pre-training achieved an absolute improvement of 6.01% over the baseline on the test set.
  • Pre-training Benefit: The highest average UAR was 4.14% higher than the UAR obtained without pre-training, underscoring the benefit of the proposed pre-training strategy.
  • TDL Benefit: The TDL also contributed positively, with all combinations of the proposed approach showing higher UARs when the TDL was included.
  • Best Performance: The highest UAR was achieved with the proposed approach using both pre-training and the TDL.

Limitations and Future Work

  • Glide-up vs. Dive Confusion: The authors note that while their method improves overall accuracy, the Glide-up and Dive classes are sometimes confused. They suggest that combining complementary properties of their proposed method with HMM-based schemes could improve discrimination between these two classes.
  • Future Work: Future research directions include:
  • Investigating methods to combine the strengths of the proposed approach with HMMs.
  • Exploring the incorporation of linguistic features to further enhance intonation classification accuracy.

Summary updated

BibTeX
@inproceedings{saha2019low,
  author = {Atreyee Saha and Chiranjeevi Yarra and Prasanta Kumar Ghosh},
  title = {{Low Resource Automatic Intonation Classification Using Gated Recurrent Unit (GRU) Networks Pre-Trained with Synthesized Pitch Patterns}},
  booktitle = {Interspeech},
  year = {2019},
}

Loading related content...

Figure