NaijaTTS: A pitch-controllable TTS model for Nigerian Pidgin

Emmett Strickland, Dana Aubakirova, Dorin Doncenco, Diego Torres, Marc EvrardSsw 2023

TL;DR

This paper introduces NaijaTTS, the first pitch-controllable Text-to-Speech (TTS) model for Nigerian Pidgin, a widely spoken but under-resourced West African language. By adapting the FastSpeech 2 architecture and carefully curating a new corpus, the authors demonstrate a promising approach to building high-quality TTS for low-resource languages, with potential applications for linguistic research.

What Problem Does This Paper Address?

Nigerian Pidgin (Naijá) is a major lingua franca in Nigeria, spoken by an estimated 75 million people. Despite its widespread use, it remains significantly understudied and underserved by the Natural Language Processing (NLP) community, particularly in the domain of speech synthesis. Existing TTS models often struggle with low-resource languages due to a lack of sufficient training data and specialized linguistic resources. This paper tackles the challenge of creating a high-quality, pitch-controllable TTS system for Nigerian Pidgin, aiming to not only provide a functional speech synthesis tool but also to facilitate linguistic research into the language's prosody.

Methods

The core of NaijaTTS is built upon the FastSpeech 2 (FS2) architecture, a non-autoregressive end-to-end TTS system. FS2 is known for its ability to extract duration, pitch, and energy information directly from ground truth target inputs during training. The authors adapted this architecture by integrating a Nigerian Pidgin pronunciation dictionary. A key feature of FS2 that NaijaTTS leverages is its variance adaptor, which allows for separate control over predicted duration, pitch, and energy during inference. This design choice is crucial for the project's goal of pitch controllability, as it allows for modification to accept user-provided pitch vectors for desired prosodic contours.

Datasets

The primary dataset used is the NaijaSynCor corpus, a 30-hour treebank of transcribed Nigerian Pidgin. For training NaijaTTS, a meticulously curated subset of approximately 7.5 hours of speech from 80 speakers was used. This subset was selected based on careful transcription, alignment, and annotation, with a focus on excluding files recorded in noisy environments or containing significant dysfluencies.

Key Insights and Ideas

  • Low-resource TTS is achievable with careful data curation: Even with a relatively small dataset (3.7 hours after filtering), excluding dysfluent speech and poor-quality recordings significantly improved the naturalness and intelligibility of the synthesized speech.
  • Speaker anonymization can be achieved through data filtering: By treating speakers as binary categories (male/female) and filtering data, the authors achieved anonymized speech that was still natural-sounding. This approach also highlighted how data filtering can impact vocal characteristics.
  • Prosody control is key for linguistic research: The explicit goal of pitch controllability in NaijaTTS is to enable linguists to conduct perceptual experiments, shedding light on the role of tone and melody in semantic and syntactic interpretation.
  • The challenge of spontaneous speech: Spontaneous vernacular speech, common in corpora like NaijaSynCor, inherently contains dysfluencies and can be difficult to understand even for humans, posing a significant challenge for TTS model training.

Key Results

The paper reports the successful creation of a functional TTS model from a small multi-speaker corpus. While specific quantitative metrics like Mean Opinion Scores (MOS) are not detailed, the authors claim that NaijaTTS compares favorably against Meta's Massively Multilingual Speech (MMS) model in terms of naturalness and intelligibility, despite the MMS model likely being trained on a much larger dataset. The model successfully generates speech that is anonymized, with efforts made to ensure consistency across utterances.

Limitations and Future Work

  • Comprehensibility: The authors acknowledge that while naturalness and comprehensibility have improved, generated utterances are not always easily understandable due to artifacts, unusual phonemes, high speech rates, and unnatural prosody. They suggest that future work might involve limiting training data to more prepared speech (e.g., radio broadcasts) to improve comprehensibility.
  • Alignment and Phonetic Representation: The current alignments and phonetizations are described as phonetic rather than phonological, leading to potential inconsistencies. Future work includes regenerating alignments without human intervention and rephonetizing the corpus for greater consistency.
  • Speaker Anonymization Consistency: While the model protects speaker identities, certain phoneme sequences can cause vocal characteristics to approach those of specific speaker groups. The authors aim to resolve this for more consistent model output and to fully guarantee anonymization.
  • Generalized Speaker Embeddings: Inspired by other work, they plan to experiment with leveraging learned speaker representations to perform anonymization using a single, generalized speaker embedding, which could lead to more consistent and versatile voices.

Summary updated

BibTeX
@inproceedings{strickland2023naijatts,
  author = {Emmett Strickland and Dana Aubakirova and Dorin Doncenco and Diego Torres and Marc Evrard},
  title = {{NaijaTTS: A pitch-controllable TTS model for Nigerian Pidgin}},
  booktitle = {Ssw},
  year = {2023},
}
Figure