A Comparison of Input Types to a Deep Neural Network-based Forced Aligner

Matthew C. Kelley, Benjamin V. TuckerInterspeech 2018

TL;DR

This paper investigates how different audio representations (raw audio vs. MFCCs) impact the performance of a deep neural network (DNN) for forced alignment. They found that using raw audio as input significantly outperforms MFCCs and a traditional forced aligner, suggesting that DNNs can learn richer temporal features directly from the raw signal.

What Problem Does This Paper Address?

Forced alignment, the process of precisely timing phonetic segments within an audio recording, is a crucial tool in speech analysis. While traditional methods often rely on Hidden Markov Models (HMMs), recent advancements in deep learning have shown promise in speech recognition. However, there's been limited exploration into how different input representations for DNNs affect forced alignment accuracy. Specifically, it's unclear whether raw audio signals or more processed features like Mel-frequency cepstral coefficients (MFCCs) are more beneficial for this task. This paper aims to fill that gap by comparing these input types within a consistent DNN architecture.

Methods

The authors developed a DNN architecture consisting of convolutional layers followed by bidirectional Long Short-Term Memory (LSTM) layers. The convolutional layers act as feature extractors, with their weights frozen after initial training. The subsequent LSTM layers then learn temporal relationships from these extracted features. This architecture was chosen for its efficiency and ability to capture both local features and long-term dependencies.

Two main experimental setups were used:

  1. Raw Audio Input: Each audio file was pre-processed into short frames (25ms) with a 1ms stride. The label for each frame was determined by the majority phone class within the central 1ms of the frame, with careful handling of ambiguous cases.
  2. MFCC Input: Audio files were processed into frames containing 13 MFCCs, 13 delta coefficients, and 13 delta-delta coefficients.

The output of the network, which provides frame-level phone probabilities, was then decoded using a dynamic programming approach (similar to Viterbi decoding) to produce the final phone sequence with temporal boundaries.

Datasets

  • TIMIT Speech Corpus: Used for both training and testing the DNN-based forced aligners. TIMIT is a standard benchmark dataset known for its detailed phonetic transcriptions.

Key Insights and Ideas

  • Raw Audio Dominance: The most striking finding is that raw audio input significantly outperforms MFCCs for forced alignment. This suggests that DNNs, particularly with convolutional layers, can effectively learn relevant acoustic features directly from the raw signal, potentially capturing nuances that MFCCs might smooth over.
  • Temporal Dynamics Matter: The use of bidirectional LSTMs is crucial for capturing the temporal dependencies in speech, which is essential for accurate phone boundary detection.
  • Human Cognition Parallel: The authors draw an interesting parallel between the DNN's learning process and human speech perception, noting that both seem to benefit from dynamic processing of temporal information. This hints at deep neural networks as potentially powerful tools for modeling human cognitive processes in speech.
  • MFCC Limitations: The poor performance of the MFCC-based aligner suggests that either the MFCC representation is less suitable for this specific task, or that the network architecture and hyperparameters used were not optimal for MFCCs, requiring further tuning.

Key Results

  • Path Accuracy (accpath):
    • Raw Audio Network: 74.7%
    • MFCC Network: 22.0%
    • Montreal Forced Aligner (MFA): 72.1%
  • Median Absolute Error (MAE) in seconds:
    • Raw Audio Network: 0.008 s (8 ms)
    • MFCC Network: 1.55 s
    • Montreal Forced Aligner (MFA): 0.1 s (100 ms)

The raw audio network achieved the highest path accuracy and the lowest median absolute error, indicating superior performance in both frame-level identification and boundary precision compared to the MFCC network and the Montreal Forced Aligner.

Limitations and Future Work

  • MFCC Architecture: The authors acknowledge that the MFCC network's architecture might not be ideal and plan further comparisons with different configurations.
  • Montreal Forced Aligner (MFA) Training Data: The MFA was not trained on the TIMIT corpus, which could have affected its performance relative to the DNNs trained on TIMIT. Future work aims to address this by training MFA on TIMIT for a more direct comparison.
  • Generalizability: The current results are based solely on the TIMIT corpus. Future work will involve testing on other speech datasets to assess generalizability.
  • Computational Efficiency: While the current architecture shows promise, further improvements in computational efficiency are desired.
  • Open-Source Release: The authors intend to release their developed system as an open-source command-line utility.

Summary updated

BibTeX
@inproceedings{kelley2018comparison,
  author = {Matthew C. Kelley and Benjamin V. Tucker},
  title = {{A Comparison of Input Types to a Deep Neural Network-based Forced Aligner}},
  booktitle = {Interspeech},
  year = {2018},
}
Figure