Text-Only Domain Adaptation for End-to-End Speech Recognition through Down-Sampling Acoustic Representation

Jiaxu Zhu, Weinan Tong, Yaoxun Xu, Changhe Song, Zhiyong Wu, Zhao You, Dan Su, Dong Yu, Helen Meng

TL;DR

This paper tackles the challenge of adapting end-to-end Automatic Speech Recognition (ASR) systems to new domains using only text data. They propose a novel method to align speech and text representations by down-sampling acoustic features, creating a more consistent length for better cross-modal learning. This approach effectively leverages readily available text-only data for domain adaptation, showing significant performance improvements.

What Problem Does This Paper Address?

End-to-end ASR models, while powerful, often suffer a performance drop when deployed in new domains with different vocabulary or speaking styles. Acquiring large amounts of paired audio-text data for every new domain is expensive and time-consuming. Traditional methods for adapting ASR models using text-only data are limited, especially for end-to-end systems. A key challenge is the inherent mismatch in length between acoustic representations (sequences of audio frames) and text representations (sequences of tokens). Previous attempts to bridge this gap by up-sampling text representations to match speech duration might not accurately reflect the actual temporal alignment. This paper aims to solve this by proposing a method to align these modalities by down-sampling the acoustic side, making it more amenable to text-only adaptation.

Methods

The core of the proposed method lies in a "Match Module" designed to create aligned representations of speech and text. This module works in conjunction with a standard end-to-end ASR architecture (specifically, a Conformer-based encoder-decoder with CTC and Attention losses).

The key innovations are:

  • Down-sampling Acoustic Representation: Instead of up-sampling text, the authors introduce a Continuous Integrate-and-Fire (CIF) module. This module processes the acoustic features from the shared encoder and generates a shorter acoustic sequence whose length is consistent with the text representation. This is achieved by accumulating acoustic features until a threshold is met, signaling a "boundary" and generating a new integrated embedding. This directly addresses the length mismatch problem.
  • Syllable-based Representation: To further improve alignment and robustness, the paper proposes using syllables as the fundamental unit for text representation, rather than characters. Syllables are considered more pronunciation-related and less susceptible to the issues of rare or long-tail words compared to individual characters. A transformer-based Syllable Encoder is used to generate these representations.
  • Modality-Matched Training: The model is trained using a Mean Absolute Error (MAE) objective to align the down-sampled acoustic representations with the syllable-based text representations. This joint training allows the model to learn a shared space for both modalities.

For text-only domain adaptation, the CIF module is bypassed, and the Syllable Encoder's output is used to train the Attention Decoder while keeping other parameters fixed. This allows the model to leverage unpaired text data effectively.

Datasets

  • Training: The primary training dataset used is Aishell-1, a public Mandarin ASR corpus containing 178 hours of labeled speech across five domains (Finance, Science and Technology, Sports, Entertainment, and News).
  • Evaluation:
    • Aishell-2: Used for evaluating domain adaptation capabilities. It's a larger corpus (1000 hours) with a similar acoustic environment to Aishell-1 but covers different text domains (voice commands, digital sequences, places of interest, etc.).
  • WenetSpeech: A multi-domain Mandarin corpus with a more complex acoustic environment, used for further testing domain adaptation on more challenging data.

Key Insights and Ideas

  • Down-sampling is Key: The paper convincingly argues that down-sampling acoustic representations is a more effective strategy for aligning modalities than up-sampling text, as it better respects the natural temporal structure of speech.
  • Syllables for Robustness: Using syllables as the text unit offers a more pronunciation-aligned and robust representation compared to characters, especially for domain adaptation where vocabulary might differ significantly.
  • Efficient Text-Only Adaptation: The proposed method provides a streamlined way to adapt E2E ASR models using text-only data, avoiding the need for complex TTS systems or external language model fusion. The text-only training process is shown to be effective and relatively quick to converge.

Key Results

  • State-of-the-Art Performance: On the Aishell-1 dataset, the proposed model achieves comparable performance to existing state-of-the-art approaches, with a character error rate (CER) of 4.1% on the dev set and 4.5% on the test set.
  • Significant Domain Adaptation Gains: When tested on the Aishell-2 dataset, the text-only adaptation significantly improves performance. The proposed model with text-only adaptation achieved a CER of 11.2% on the dev set and 11.0% on the test set, outperforming the baseline model with an LM (11.6%/11.5%).
  • Rapid Convergence: Figure 3 shows that the text-only performance improves significantly within approximately 40 training epochs, demonstrating the efficiency of the adaptation process.
  • Syllable Advantage: Using syllables as the model unit for text-only adaptation on Aishell-2 resulted in better performance (11.2%/11.0% dev/test CER) compared to using characters (12.0%/11.9% dev/test CER).

Limitations and Future Work

  • Complex Acoustic Environments: The paper notes that while the proposed method shows improvements, its performance degrades in more complex acoustic environments, as seen with the WenetSpeech dataset. This suggests that the acoustic modeling component of the base model might need further enhancement for such challenging scenarios.
  • Future Directions: The authors plan to:
  • Further explore the performance of different model units.
  • Investigate the method's effectiveness on large-scale datasets.
  • Validate the approach on English datasets.

Summary updated

Abstract

Mapping two modalities, speech and text, into a shared representation space, is a research topic of using text-only data to improve end-to-end automatic speech recognition (ASR) performance in new domains. However, the length of speech representation and text representation is inconsistent. Although the previous method up-samples the text representation to align with acoustic modality, it may not match the expected actual duration. In this paper, we proposed a novel representations match strategy through down-sampling acoustic representation to align with text modality. By introducing a continuous integrate-and-fire (CIF) module generating acoustic representations consistent with token length, our ASR model can learn unified representations from both modalities better, allowing for domain adaptation using text-only data of the target domain. Experiment results of new domain data demonstrate the effectiveness of the proposed method.
BibTeX
@inproceedings{zhu2023text,
  author = {Jiaxu Zhu and Weinan Tong and Yaoxun Xu and Changhe Song and Zhiyong Wu and Zhao You and Dan Su and Dong Yu and Helen Meng},
  title = {{Text-Only Domain Adaptation for End-to-End Speech Recognition through Down-Sampling Acoustic Representation}},
  booktitle = {Interspeech},
  year = {2023},
  doi = {10.21437/interspeech.2023-1378},
}
Figure