Punctuation Restoration in Spanish Customer Support Transcripts using Transfer Learning

Xiliang Zhu, Shayna Gardiner, David Rossouw, Tere Roldán, Simon Corston-OliverNAACL 2022

TL;DR

This paper tackles the challenge of accurately punctuating Spanish customer support transcripts, a task made difficult by noisy ASR output and domain-specific language. They propose novel transfer learning strategies, combining out-of-domain Spanish data and cross-lingual transfer from English, to significantly improve punctuation restoration accuracy without modifying the underlying pre-trained models.

What Problem Does This Paper Address?

Automatic Speech Recognition (ASR) systems are ubiquitous, but their output often lacks punctuation, making transcripts hard to read and comprehend. Restoring punctuation is crucial for improving readability. This paper focuses on the specific, challenging domain of Spanish customer support conversations. Real-world spoken dialogue is inherently messy, featuring filler words and false starts. Furthermore, domain-specific language in customer support differs from general Spanish text, making it hard to leverage existing public datasets. Acquiring sufficient in-domain Spanish data for punctuation restoration is a significant hurdle, especially for low-resource scenarios.

Methods

The core of this work lies in leveraging transfer learning to overcome the data sparsity issue. The authors explore two main strategies:

  • Domain Adaptation: This involves using out-of-domain Spanish text data to improve the model's performance on the target customer support domain. They employ two techniques:
  • Data Selection: Using a 4-gram language model trained on their in-domain data, they select the most relevant 100,000 utterances from the massive Spanish OpenSubtitle corpus.
  • Data Augmentation: To better match the distribution of terminating punctuation marks in their in-domain data, they augment out-of-domain datasets by concatenating sentences.
  • Cross-lingual Transfer Learning: This strategy utilizes English in-domain data to enhance the Spanish punctuation restoration system. Recognizing that punctuation conventions differ (e.g., Spanish uses inverted question/exclamation marks), they propose a method to convert English punctuation to mimic Spanish conventions during training. This allows the model to learn Spanish punctuation patterns even when processing English text. They investigate three fine-tuning strategies: "ES->EN" (Spanish then English), "EN->ES" (English then Spanish), and "Joint EN, ES" (joint training).

The system is built upon pre-trained transformer models (BETO, mBERT, XLM-R) and formulates punctuation restoration as a sequence labeling task. A token classification layer is added on top, and simple post-processing heuristics are used to correct unmatched paired punctuation marks.

Datasets

The paper utilizes several datasets:

  • In-domain Spanish Transcripts: Approximately 5,000 transcribed Spanish utterances from real customer support calls. This is the primary target domain data.
  • LDC Fisher Spanish Speech and Fisher Spanish Transcripts: Corpora containing about 163 hours of telephone conversations from native Spanish speakers. While acoustically relevant, the transcripts are mostly social calls, not customer support specific.
  • Spanish OpenSubtitle Corpus: A very large corpus (179 million sentences) from movie subtitles, providing exposure to a wider range of punctuation, including exclamation marks. However, the domain is distant from customer support.
  • English In-domain Transcripts: Around 200,000 manually transcribed English utterances from their call center product, used for cross-lingual transfer.

Key Insights and Ideas

  • Transfer Learning is Key for Low-Resource Domains: The paper strongly demonstrates that transfer learning is essential for achieving good performance in specialized, low-resource domains like Spanish customer support punctuation.
  • Bridging the Domain Gap: Both domain adaptation (using out-of-domain Spanish data) and cross-lingual transfer (using English data) are effective in mitigating the mismatch between general language models and the specific customer support domain.
  • Cross-lingual Transfer Requires Careful Handling of Conventions: Simply applying cross-lingual transfer isn't enough. The authors highlight the importance of adapting punctuation conventions (like inverted marks) to make the transfer effective.
  • Joint Training Offers Significant Gains: Combining English and Spanish data for fine-tuning ("Joint EN, ES") proved to be a highly effective strategy for multilingual models.
  • Multilingual Models Shine: XLM-R, a multilingual model, consistently outperformed the monolingual BETO model, especially after domain adaptation and cross-lingual transfer.

Key Results

  • Domain Adaptation Improves Performance: Both data selection and data augmentation in domain adaptation led to improved F1 scores across all three pre-trained models (BETO, mBERT, XLM-R). For example, augmented LDC and OpenSubtitle data improved performance, with XLM-R achieving a 54.7% F1 score.
  • Cross-lingual Transfer Boosts Multilingual Models: The "EN->ES" and "Joint EN, ES" strategies significantly improved the performance of mBERT and XLM-R. The "Joint EN, ES" strategy yielded the best results, with XLM-R achieving a 64.4% F1 score.
  • XLM-R Outperforms: XLM-R consistently achieved the highest F1 scores, outperforming the monolingual BETO model by 1.5% F1 score after joint training.
  • "ES->EN" is Less Effective: The "ES->EN" fine-tuning strategy actually resulted in lower accuracy compared to models trained without cross-lingual transfer, suggesting that transferring from a higher-resource language (English) to a lower-resource one (Spanish) is more beneficial in this context.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that the post-processing step for correcting paired punctuation marks, while improving overall accuracy, might not always produce the correct result.
  • A significant prediction error identified is the confusion between CLOSE_QUESTION and PERIOD classes, which is common across all models. This is attributed to the linguistic properties of Spanish, where intonation plays a crucial role in distinguishing questions from declarative sentences, and structural cues are less explicit.
  • Future Work:
    • The authors suggest incorporating acoustic information into punctuation restoration tasks to better handle the ambiguity that arises from intonation in Spanish. This could involve leveraging prosodic features of speech to aid in distinguishing questions from statements.

Summary updated

BibTeX
@inproceedings{zhu2022punctuation,
  author = {Xiliang Zhu and Shayna Gardiner and David Rossouw and Tere Roldán and Simon Corston-Oliver},
  title = {{Punctuation Restoration in Spanish Customer Support Transcripts using Transfer Learning}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure