Towards Language-Agnostic STIPA: Universal Phonetic Transcription to Support Language Documentation at Scale

Jacob Lee Suchardt, Hana El-Shazli, Pierluigi CassottiEMNLP 2025

TL;DR

This paper introduces Speech-to-IPA (STIPA) models, leveraging the powerful Whisper ASR architecture, to create universal phonetic transcriptions for under-resourced languages. They demonstrate that fine-tuning Whisper can achieve state-of-the-art results, significantly aiding language documentation efforts by overcoming the limitations of traditional orthographic transcription.

What Problem Does This Paper Address?

The vast majority of Automatic Speech Recognition (ASR) systems are designed for high-resource languages and produce orthographic (textual) outputs. This leaves a significant gap for documenting and analyzing languages that lack standardized writing systems or are severely under-resourced. Manual phonetic transcription, while crucial for linguistic documentation, is incredibly time-consuming. This paper addresses the need for a scalable, language-agnostic method to generate accurate phonetic transcriptions, enabling broader language documentation and analysis.

Methods

The core of this work involves fine-tuning the state-of-the-art Whisper ASR model for the Speech-to-IPA (STIPA) task. Instead of predicting orthographic text, the model is trained to output phonetic transcriptions using the International Phonetic Alphabet (IPA). The authors explore both full fine-tuning and parameter-efficient fine-tuning techniques like LoRA (LoWhIPA). They also introduce a custom Buckwalter-to-IPA transliteration module to create a new STIPA dataset for South Levantine Arabic. The evaluation uses both string-level (PER) and phonologically informed (PFER) metrics, with modifications to improve reliability.

Datasets

  • South Levantine Arabic (ASC): A new STIPA dataset constructed by mapping Buckwalter notations to IPA.
  • Common Voice (CV): Used for training and evaluation, specifically focusing on languages with orthographically transparent scripts.
  • THCHS-30: A Mandarin speech corpus used for training and evaluation.
  • VoxAngeles: A corpus with phonetic diversity across 21 language families, used for evaluation.
  • Sanna: An extremely low-resource, endangered language used as a case study for zero-shot transfer.

Key Insights and Ideas

  • Whisper's Latent Representations are Phonetically Rich: The paper demonstrates that Whisper's pre-trained encoder contains rich phonetic information that can be effectively leveraged for STIPA tasks, even without explicit phonetic pre-training.
  • Language-Agnostic STIPA is Feasible: By fine-tuning Whisper with a universal IPA vocabulary, the models can generalize across diverse languages, including those with no standardized orthography.
  • Parameter-Efficient Fine-Tuning (PEFT) is Effective: LoRA-based fine-tuning (LoWhIPA) achieves competitive or even superior results to full fine-tuning, while significantly reducing computational costs and training time.
  • Curated, Diverse Datasets Matter: The study suggests that the quality and diversity of training data are more critical than sheer volume, especially for low-resource settings.
  • Typological Similarity Aids Transfer: Fine-tuning on typologically similar languages (e.g., using Levantine Arabic data for Sanna) significantly improves zero-shot performance on related languages.

Key Results

  • Fine-tuned Whisper models achieve state-of-the-art (SOTA) results on STIPA tasks, significantly outperforming previous methods on various datasets.
  • LoWhIPA Large achieved the best results on CommonVoice seen languages (14.18% PER, 4.95% PFER).
  • For the endangered language Sanna, a LoRA-tuned Whisper model (LoWhIPA Large-SR) fine-tuned on related languages achieved the best performance (60.23% PER, 19.33% PFER).
  • The models demonstrate robust generalization, performing well even on unseen languages, though performance naturally degrades compared to seen languages.
  • Even minimal multilingual input improved performance over monolingual baselines, supporting few-shot learning for STIPA.

Limitations and Future Work

  • Tokenizer Limitations: The authors acknowledge that the Whisper tokenizer was not modified, and explicitly restricting the output to IPA tokens could potentially improve performance and reduce interference from pre-training.
  • Hyperparameter Search: A thorough hyperparameter search was not conducted, suggesting potential for further optimization.
  • Evaluation Metrics: While PFER was refined, the authors note that standardized metrics and benchmarks for STIPA are still lacking, hindering direct comparability across studies.
  • Tonal and Suprasegmental Phenomena: The current models may not fully capture complex tonal and suprasegmental phenomena, which are critical for many languages.
  • Future Work: The authors suggest exploring joint fine-tuning of the encoder-decoder, investigating different PEFT methods, and conducting more extensive hyperparameter searches. They also highlight the need for unified evaluation benchmarks and metrics.

Summary updated

BibTeX
@inproceedings{suchardt2025towards,
  author = {Jacob Lee Suchardt and Hana El-Shazli and Pierluigi Cassotti},
  title = {{Towards Language-Agnostic STIPA: Universal Phonetic Transcription to Support Language Documentation at Scale}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure