Multi-Task CTC Training with Auxiliary Feature Reconstruction for End-to-End Speech Recognition

Gakuto Kurata, Kartik AudhkhasiInterspeech 2019

TL;DR

This paper introduces a novel multi-task learning approach for end-to-end speech recognition by adding an auxiliary task: reconstructing the input audio features. By intentionally distorting these features during reconstruction training, the model learns more robust representations, leading to improved accuracy, especially on unseen data.

What Problem Does This Paper Address?

End-to-end (E2E) automatic speech recognition (ASR) systems, particularly those using Connectionist Temporal Classification (CTC), have simplified ASR pipelines by directly mapping acoustic features to symbol sequences. However, like many deep learning models, they are prone to overfitting, especially when dealing with data that differs from the training set. While multi-task learning (MTL) has been used to mitigate overfitting by sharing layers across related tasks, this paper explores a new auxiliary task: reconstructing the input audio features. The challenge is to design this auxiliary task effectively to improve the primary ASR task without introducing new complexities or requiring significant architectural changes.

Methods

The core innovation is a multi-task learning framework where the main task is standard E2E CTC ASR (either phone-based or word-based). The auxiliary task involves a decoder network that attempts to reconstruct the input audio features from the encoder's latent representations. This reconstruction is trained using Mean Squared Error (MSE) loss.

Crucially, the authors introduce a novel way to train this auxiliary task: by distorting the input features. These distortions, applied only to the auxiliary task's reconstruction target, include:

  • Swap: Splitting the input feature sequence into two parts and swapping them.
  • Strip: Removing either the beginning or end portion of the input feature sequence.

These distortions are designed to break long-span temporal dependencies, forcing the model to rely on more localized information and thus preventing overfitting to specific long-range patterns in the training data. The training procedure involves randomly selecting mini-batches to perform the auxiliary reconstruction task alongside the main CTC task, while other mini-batches only train the main task.

Datasets

  • Training Data: 262 hours of segmented speech from the Switchboard-1 English conversational telephone speech dataset.
  • Evaluation Data:
    • Matched: Switchboard (SWB) and Call-Home (CH) subsets of the NIST Hub5 2000 evaluation data.
  • Mismatched: Down-sampled DARPA EARS dev04f broadcast news data (BN). The paper highlights that CH is considered mismatched due to acoustic and linguistic differences from SWB.

Key Insights and Ideas

  • Feature Reconstruction as a Regularizer: Using feature reconstruction as an auxiliary task, especially with distorted inputs, acts as a powerful regularizer for the E2E ASR encoder.
  • Distortion is Key: Simply reconstructing features isn't enough; intentionally distorting them (swap, strip) is what significantly improves robustness and reduces overfitting. This is because it forces the model to learn more localized temporal patterns.
  • Mismatched Data Benefits Most: The improvements are most pronounced on mismatched test sets (like Broadcast News), indicating that the auxiliary task helps the model generalize better to unseen acoustic conditions and styles.
  • Simplicity and Effectiveness: The proposed method is relatively simple, leveraging existing CTC architectures and adding a straightforward reconstruction decoder, yet yields significant gains.

Key Results

  • Phone CTC: Achieved consistent Word Error Rate (WER) improvements across all test sets (SWB, CH, BN) compared to the baseline without MTL. The static-only reconstruction target (80 dimensions) performed comparably to the full target (240 dimensions) and was chosen for subsequent experiments.
  • Word CTC: Similar improvements were observed, particularly on the mismatched CH and BN test sets. The "swap" distortion method showed the best average WER reduction.
  • Generalization: The auxiliary task significantly improved performance on mismatched test sets, demonstrating its effectiveness in combating overfitting.

Limitations and Future Work

  • Word CTC Degradation: For word CTC models, a minor degradation was observed on some SWB test cases where training and test data were perfectly matched. The authors hypothesize this might be due to the auxiliary task not considering linguistic information, potentially hurting the implicit language model within the word CTC model. They suggest rescoring with an external language model could mitigate this.
  • Future Work: The authors suggest that this feature reconstruction approach could be combined with other E2E modeling techniques beyond CTC, such as attention-based encoder-decoder models.

Summary updated

BibTeX
@inproceedings{kurata2019multi,
  author = {Gakuto Kurata and Kartik Audhkhasi},
  title = {{Multi-Task CTC Training with Auxiliary Feature Reconstruction for End-to-End Speech Recognition}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure