Using ASR-Generated Text for Spoken Language Modeling

Nicolas Hervé, Valentin Pelloin, Benoit Favre, Franck Dary, Antoine Laurent, Sylvain Meignier, Laurent BesacierACL 2022

TL;DR

This paper introduces "FlauBERT-Oral," a new family of language models trained on a massive dataset of automatically transcribed French TV shows. These models demonstrate that noisy ASR-generated text can significantly improve spoken language modeling, outperforming standard models on downstream tasks like TV show classification and syntactic parsing.

What Problem Does This Paper Address?

Modeling spoken language is challenging because "oral-style" transcripts are difficult to obtain at scale. Existing large language models (LLMs) are typically trained on vast amounts of written text, which doesn't capture the nuances of spontaneous speech, such as hesitations, disfluencies, and informal grammar. This gap limits the effectiveness of LLMs in applications dealing with spoken language. The authors aim to bridge this gap by leveraging readily available, large-scale ASR transcriptions of broadcast content to create more robust spoken language models.

Methods

The core of this work involves training and evaluating new language models based on the existing French FlauBERT model. The authors explore three main strategies:

  • Fine-tuning (FT): Taking the pre-trained FlauBERT model and continuing its training for a few epochs on the ASR-generated transcripts.
  • Mixed Retraining (MIX): Retraining a full model using a combination of ASR text and clean written text (Wikipedia, news articles). This approach also involves retraining the BPE tokenizer.
  • ASR-only Retraining (ORAL/ORAL_NB): Retraining a full model from scratch using only the ASR-generated text. Two variants are presented: one using the original FlauBERT tokenizer and another with a newly trained BPE tokenizer on the ASR data.

These models are then evaluated on word prediction and two downstream tasks: TV show classification and syntactic parsing of spoken conversations.

Datasets

  • Training Data:
    • INA Collection: Approximately 350,000 hours of French TV shows and radio news programs (2013-2020), automatically transcribed using an ASR system. This resulted in 51 million speech segments and 3.5 billion words (19GB of text).
  • Written Text: French Wikipedia dump and online press articles (AFP, OTMedia).
  • Evaluation Data:
    • Word Prediction:
      • AFP dispatches from 2021 (written text).
  • French National Assembly sessions (parl_13 and parl_15 mandates) - manual transcriptions of spoken proceedings.
  • INA web studio data: manual transcriptions of educational videos/interviews (studio_manual) and ASR transcriptions of the same content (studio_asr).
  • TV Show Classification: 47,867 short TV shows (news items) from TF1, France 2, France 3, and M6 (2017-2019).
  • Syntactic Parsing: Annotated subset of the Orfeo project corpus (Benzitoun et al., 2016; Nasr et al., 2020) containing spoken French conversations from various sources.

Key Insights and Ideas

  • ASR Text is Valuable: Despite its inherent noise and lack of punctuation, ASR-generated text can be a valuable resource for improving spoken language models.
  • Oral vs. Written Style: The models trained on ASR data show improved performance on spoken text datasets, highlighting their ability to capture oral-style language characteristics.
  • Tokenizer Matters: The choice of BPE tokenizer significantly impacts performance. Using a tokenizer trained on clean data (even for ASR-only models) seems to be beneficial.
  • Punctuation's Role: The absence of punctuation in ASR transcripts can affect downstream tasks like syntactic parsing. Adding virtual punctuation can help compensate for this.
  • Domain Specificity: Models trained on ASR data (especially news content) perform well on related written texts (like AFP dispatches) because those texts are also news-oriented.

Key Results

  • Word Prediction: The FlauBERT-Oral models, particularly the ASR-only retrained ones, generally outperform the original FlauBERT model on spoken text datasets (parliamentary sessions, INA studio data).
  • TV Show Classification: The ORAL model (ASR-only retraining with original tokenizer) achieved the best performance in a standard learning setting, outperforming the baseline SVM and other FlauBERT variants. In few-shot learning scenarios, the performance gap between FlauBERT-Oral models and SVMs became more pronounced as training data decreased.
  • Syntactic Parsing: Pretraining on ASR data (MIX and ORAL) led to substantial improvements in Labeled Attachment Score (LAS) compared to the text-only FlauBERT model (FBU). The ORAL model showed the best global performance, though it was more affected by out-of-vocabulary (OOV) words compared to the FBU baseline.

Limitations and Future Work

  • Uncased Text: The ASR system produced uncased transcripts, and the models were trained accordingly. The authors suggest that re-capitalization and punctuation restoration could further improve spoken language models.
  • ASR Specificities: Further analysis is needed to understand the specific oral features present in the ASR-generated text (e.g., word repetitions, interjections).
  • Tokenizer Impact: More investigation into the impact of BPE units for spoken language modeling is warranted.
  • Downstream Task Expansion: The authors plan to evaluate the FlauBERT-Oral models on text downstream tasks and more speech tasks like spoken language understanding (SLU) or ASR re-scoring.

Summary updated

BibTeX
@inproceedings{herve2022using,
  author = {Nicolas Hervé and Valentin Pelloin and Benoit Favre and Franck Dary and Antoine Laurent and Sylvain Meignier and Laurent Besacier},
  title = {{Using ASR-Generated Text for Spoken Language Modeling}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure