Aligning and recognizing spoken books in different varieties of Portuguese

Isabel Trancoso, António Serralheiro, Céu Viana, Diamantino CaseiroInterspeech 2005

TL;DR

This paper explores the challenges of adapting speech recognition and alignment systems from European Portuguese (EP) to Brazilian Portuguese (BP) using digital spoken books. They demonstrate that existing EP models perform poorly on BP, but also show that even limited adaptation techniques can yield improvements, highlighting the potential of spoken books as a diagnostic tool for cross-dialectal speech technology.

What Problem Does This Paper Address?

Developing robust speech recognition and alignment systems for diverse languages and dialects is a significant challenge. While systems are often trained on one variety of a language, their performance can degrade substantially when applied to another due to subtle but impactful differences in pronunciation, phonology, and even orthography. This paper specifically investigates this problem for Portuguese, comparing European Portuguese (EP) and Brazilian Portuguese (BP). The authors aim to understand how differences between these two major varieties affect existing speech technology components and to explore methods for adapting these technologies, particularly using digital spoken books as a novel diagnostic tool. They also address the inadequacy of broadcast news (BN) models for the more literary content found in spoken books.

Methods

The core of this work relies on a Weighted Finite-State Transducer (WFST) based decoder for speech alignment and recognition. For alignment, the language model is simply the word sequence of the transcription. For recognition, they employ hybrid acoustic models combining Hidden Markov Models (HMMs) with Multi-Layer Perceptrons (MLPs) trained on various feature extraction methods (PLP, Log-RASTA, MSG).

Key methodological contributions include:

  • Spoken Book Alignment: Modifying a BN recognizer's decoder to achieve successful alignment of spoken books, even for long recordings, in a single step.
  • Grapheme-to-Phone (GtoP) Adaptation: Porting an EP GtoP system to BP by considering phonetic and phonological differences. This involved developing a BP pronunciation lexicon and adapting rules.
  • Pronunciation Lexicon Adaptation: Investigating the impact of adding pronunciation variants derived from the BP GtoP system to the EP lexicon (BP1) and further refining these rules to better match the BP speaker's accent (BP2).
  • Diagnostic Tool: Utilizing digital spoken books as a means to systematically evaluate and diagnose alignment and recognition problems across different language varieties.

Datasets

The primary datasets used are:

  • EP Spoken Books Corpus: An existing repository of aligned spoken books in European Portuguese, including fiction, didactic texts, poetry, and children's stories.
  • BP Spoken Book Corpus: A newly created, limited parallel corpus consisting of a single short story, "O Monge Desastrado," read in both EP (Lisbon accent) and BP (Rio de Janeiro accent). This is the main benchmark for cross-dialectal evaluation.
  • Broadcast News (BN) Database: Used for training the initial EP recognizer (45 hours of transcribed speech).

Key Insights and Ideas

  • Spoken Books as Diagnostic Tools: The paper strongly advocates for digital spoken books as an effective and insightful tool for identifying and understanding cross-dialectal speech technology challenges, offering a more naturalistic domain than broadcast news.
  • Significant EP-BP Differences: The study confirms substantial phonetic and phonological differences between EP and BP, particularly in vowel reduction, which directly impact speech recognition performance.
  • EP Models Degrade on BP: As expected, an EP-trained aligner and recognizer perform significantly worse on BP spoken books, highlighting the need for adaptation.
  • Limited Adaptation Yields Gains: Even with a small BP corpus and rule-based GtoP adaptation, improvements in recognition scores were observed, suggesting that targeted adaptation can mitigate some of the performance degradation.
  • BN Models are Inadequate for Literary Content: The BN recognizer, trained on a different domain, shows significant performance degradation on spoken books due to factors like higher Out-Of-Vocabulary (OOV) rates and different linguistic patterns.

Key Results

  • Alignment: The WFST-based alignment system achieved successful alignment of EP spoken books, with informal evaluations suggesting similar promising results for the BP spoken book. Speaker adaptation further improved alignment accuracy.
  • Recognition (EP vs. BP):
    • Using an EP recognizer (EPO lexicon) on an EP spoken book yielded a Word Error Rate (WER) of 10.9%.
  • Using the same EP recognizer (EPO lexicon) on the BP spoken book resulted in a significantly higher WER of 36.5%.
  • Recognition with Adapted Lexicons:
    • Adapting the lexicon to BP (BP1) improved BP recognition, reducing the WER to 34.6%.
  • Further refinement of the BP lexicon (BP2) yielded a slightly better WER of 32.9% for the BP spoken book.
  • Pronunciation Adaptation: Adding pronunciation variants derived from a rule-based GtoP system (BP1, BP2) resulted in a very small improvement in recognition scores for the BP spoken book, despite a 4.3% word-level transcription error rate in the GtoP system itself.

Limitations and Future Work

  • Limited BP Corpus: The authors acknowledge that the BP repository is "very limited," consisting of only one short story. This restricts the conclusiveness of their findings and necessitates future work with larger, more diverse BP spoken book corpora.
  • Pronunciation Adaptation Impact: While adaptation showed promise, the improvement in recognition scores was "very small." Future work should explore more sophisticated pronunciation modeling techniques.
  • BN Model Inadequacy: The paper confirms the inadequacy of BN models for literary content and suggests that increasing lexicon size and decreasing OOV rates through automatic TV captioning efforts will be crucial.
  • Variety Identification: A key future direction is the development of a variety identification module that can dynamically switch between EP and BP models to avoid performance degradation when encountering segments from different speakers.
  • Data-Driven Prosodic Modeling: Spoken books are also highlighted as valuable for text-to-speech synthesis, particularly for data-driven prosodic modeling and unit selection.

Summary updated

BibTeX
@inproceedings{trancoso2005aligning,
  author = {Isabel Trancoso and António Serralheiro and Céu Viana and Diamantino Caseiro},
  title = {{Aligning and recognizing spoken books in different varieties of Portuguese}},
  booktitle = {Interspeech},
  year = {2005},
}
Figure