MUST&P-SRL: Multi-lingual and Unified Syllabification in Text and Phonetic Domains for Speech Representation Learning

Noé TitsEMNLP 2023

TL;DR

This paper introduces MUST&P-SRL, a novel methodology for extracting rich linguistic features, including phonetic transcriptions, stress marks, and unified syllabification across text and phonetic domains. This approach is designed to be compatible with forced-alignment tools and aims to provide fine-grained expert annotations for speech representation learning, disentangling speech factors, and advancing speech technologies.

What Problem Does This Paper Address?

Modern speech technologies, particularly end-to-end deep learning models, often function as "black boxes," lacking explainability. This has led to a growing interest in Explainable AI (XAI) and representation learning, which seeks to discover robust and meaningful representations of speech data. A key challenge in this area is the lack of fine-grained, expert-level annotations that can be reliably aligned with speech signals. Specifically, automatically identifying syllable boundaries and stress patterns across multiple languages, in both written text and spoken forms, is complex due to linguistic variations and ambiguities. Existing tools often focus on either text or phonetics, or lack multilingual support, creating a gap for comprehensive linguistic feature extraction needed for advanced speech analysis and representation learning.

Methods

The MUST&P-SRL methodology comprises several key steps:

  • Text Normalization: Standardizes input text, handling punctuation and acronyms.
  • Grapheme-to-Phoneme (G2P) Conversion: Converts written text into phonetic transcriptions. The system uses a two-step approach: first, it queries a pronunciation dictionary, and if the word is not found, it employs a machine learning model to estimate pronunciation. This leverages existing high-quality transcriptions while handling out-of-vocabulary words.
  • Phonetic Syllabification: Employs the Sonority Sequencing Principle (SSP) to determine syllable boundaries based on the relative sonority of speech sounds. The implementation is based on the SyllabiPy library. Special rules are included to handle hiatuses and sibilant-stop consonant clusters. Stress marks are extracted using eSpeak.
  • Text Syllabification: Aims to match the number of syllables in the text domain with the phonetic domain. This involves looking up words in manually syllabified corpora and applying systematic corrections. For words not found in corpora, it uses SSP on letters, with a fallback to a DTW-based approach for better alignment between phonetic and spelling domains.
  • Consistency Analysis: A consensus mechanism is used to identify inconsistencies between text and phonetic syllabification, facilitating manual correction of exceptional cases.

The system is designed to be multilingual and compatible with tools like the Montreal Forced Aligner (MFA).

Datasets

  • CMU ARCTIC sentences: Used for evaluation, providing a multi-speaker corpus of phonetically balanced English utterances.
  • MFA pronunciation dictionary (en_US): Used for evaluation and as a source for phonetic transcriptions.
  • Gutenberg Project (English) and Lexique3838 (French): Manually syllabified word datasets used for text syllabification.

The paper also mentions the generation of valuable annotations for the CMU ARCTIC dataset, which are made available online.

Key Insights and Ideas

  • Unified Syllabification is Crucial: The paper highlights the importance of aligning syllabification across both text and phonetic domains for robust speech analysis. Discrepancies can hinder accurate representation learning.
  • SSP as a Strong Baseline: The Sonority Sequencing Principle proves to be a reliable method for phonetic syllabification, forming the core of the system.
  • DTW for Bridging Domains: Dynamic Time Warping (DTW) is effectively used to align sonority sequences between the phonetic and spelling domains, helping to overcome limitations of text-based SSP and map phonetic syllable breaks to their textual counterparts.
  • Corpus Lookup Enhances Accuracy: Integrating lookups in manually syllabified word datasets significantly improves word accuracy, especially for languages like English and French.
  • Multilingual Compatibility is Achievable: The methodology demonstrates a successful approach to multilingual linguistic feature extraction, compatible with existing tools like MFA.

Key Results

The paper reports word accuracies for different methods and languages. For example, in an ablation study on English, French, and Spanish:

  • The lkp-SSP-DTW version (combining corpus lookup and DTW) achieved the highest accuracy for English (e.g., ~95.5% for en_US).
  • The SSP-DTW methodology generally performed better than the naive SSP on text across all tested languages.
  • For the CMU ARCTIC dataset, the word accuracy obtained on all words was > 99.8% after applying the phonetization and unified syllabification.

Limitations and Future Work

  • Multilingual Scope: While aiming for multilingualism, the current implementation and evaluation were primarily focused on English, French, and Spanish. Extending to languages with significantly different phonetic structures is a future challenge.
  • Reliance on Dictionaries: The G2P conversion heavily relies on the availability and quality of pronunciation dictionaries. Handling out-of-vocabulary words and context-dependent pronunciation remains a challenge, often requiring manual correction.
  • Consensus Mechanism: The consensus mechanism, while effective, may still inherit inaccuracies from the underlying resources.
  • Future Work:
    • Extending the methodology to more languages.
  • Investigating the impact of these linguistic features on specific speech technology applications (e.g., speech synthesis, analysis).
  • Refining the methodology by incorporating language-specific rules.
  • Progressively incorporating edge cases into the training dataset to further enhance performance.
  • Improving the scalability of the system by addressing dictionary limitations.

Summary updated

BibTeX
@inproceedings{tits2023must,
  author = {Noé Tits},
  title = {{MUST\&P-SRL: Multi-lingual and Unified Syllabification in Text and Phonetic Domains for Speech Representation Learning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
}
Figure