Read to Hear: A Zero-Shot Pronunciation Assessment Using Textual Descriptions and LLMs

Yu-Wen Chen, Melody Ma, Julia HirschbergEMNLP 2025

TL;DR

This paper introduces TextPA, a novel zero-shot pronunciation assessment system that leverages Large Language Models (LLMs) and textual descriptions of speech signals. Instead of relying on traditional audio-score pairs, TextPA uses LLMs to interpret transcripts, phoneme sequences, and pause durations, providing interpretable feedback beyond just numerical scores. This approach is cost-effective and shows competitive performance, even enhancing existing models.

What Problem Does This Paper Address?

Traditional automatic pronunciation assessment systems typically rely on acoustic models trained on vast amounts of audio data paired with human-assigned scores. While effective, these systems often provide only numerical scores, leaving learners with little insight into why they received a particular score. Collecting detailed, descriptive feedback from human raters is time-consuming and expensive. The paper addresses this by exploring how Large Language Models (LLMs), which have demonstrated strong capabilities in language understanding and generation, can be utilized for pronunciation assessment, offering explainable feedback without requiring extensive audio-score training data.

Methods

The core of TextPA is its zero-shot approach, which bypasses the need for supervised training on audio-score data. Instead, it extracts several textual acoustic descriptors from speech signals:

  • Transcript: Obtained from an Automatic Speech Recognition (ASR) model. This helps identify issues with word recognition and fluency markers like repeated words or filler sounds.
  • Phoneme Sequences (IPA and CMU): These provide a more detailed representation of spoken sounds. The International Phonetic Alphabet (IPA) offers fine-grained phonetic detail, while the CMU format is common in speech processing. These sequences help pinpoint articulation errors.
  • Pause Durations: Extracted from phonetic alignment, these are crucial for assessing fluency and identifying unnatural pauses.

These textual descriptors are then fed into an LLM, along with a carefully crafted prompt, to assess pronunciation accuracy and fluency. The LLM is asked to provide scores and detailed reasoning. To further refine accuracy assessment, IPA match scoring is introduced. This method compares the recognized IPA sequence with the canonical IPA sequence derived from the transcript using the Smith-Waterman algorithm for sequence alignment.

Datasets

The paper evaluates TextPA on two datasets:

  • Speechocean762: A dataset of 5,000 utterances from 250 native Mandarin speakers learning English. It consists of relatively short, scripted utterances from read-aloud tasks, providing ground-truth transcripts.
  • MultiPA: A dataset of 50 audio clips capturing free-form, conversational speech from approximately 20 anonymous users interacting with a chatbot. This dataset allows for a more authentic assessment of learners' speaking abilities in open-ended scenarios.

Key Insights and Ideas

  • LLMs as Pronunciation Assessors: The study demonstrates that LLMs, when provided with appropriate textual representations of speech, can effectively assess pronunciation accuracy and fluency without direct audio-score training.
  • Textual Descriptors are Key: The combination of transcripts, phoneme sequences (IPA and CMU), and pause durations provides a rich set of cues that LLMs can leverage to understand pronunciation characteristics.
  • Interpretable Feedback: TextPA moves beyond numerical scores to provide detailed, human-readable reasoning, which is crucial for language learners to understand and improve their pronunciation.
  • IPA Match Scoring Enhances Accuracy: Using IPA sequences for similarity matching with the canonical transcript proves to be a highly effective method for refining accuracy assessment, outperforming CMU sequences due to IPA's greater phonetic detail.
  • Complementary Strengths: TextPA's textual approach complements traditional audio-score trained models. When combined, they can achieve improved performance, especially on out-of-domain data, by leveraging different sources of information.
  • Cost-Effectiveness: TextPA significantly reduces API costs compared to large audio-language models (ALMs) that process audio tokens directly.

Key Results

  • Competitive Performance: TextPA achieves competitive or superior performance compared to existing models, particularly in zero-shot settings.
  • MultiPA Dataset: TextPA (using GPT-4o-mini) achieved an accuracy of 0.728 and fluency of 0.650 on the MultiPA dataset, outperforming GPT-40-mini-audio and the in-domain MultiPA model in some aspects.
  • Speechocean Dataset: TextPA (using Gemini-2.0-flash) achieved an accuracy of 0.532 and fluency of 0.557 on the Speechocean dataset. Performance on this dataset was lower than on MultiPA, attributed to the scripted nature and limited variation in Speechocean.
  • LLM Reasoning Quality: Gemini-2.0-flash generally provided more constructive reasoning than GPT-40-mini, with a higher proportion of its feedback being correct or constructive.
  • Prosody Assessment Challenges: While TextPA can assess accuracy and fluency, evaluating prosody using textual descriptions alone proved challenging for LLMs, and including it as an assessment criterion sometimes led to a decline in accuracy and fluency scores.

Limitations and Future Work

  • Prosody Assessment: The paper acknowledges that accurately assessing prosody (rhythm, intonation) from textual descriptions alone is difficult for LLMs. Future work could explore more advanced methods for prosody representation and assessment.
  • LLM Variability: LLMs and ASR systems can introduce variability across runs, leading to inconsistent results.
  • Budget Constraints: The study was limited by budget constraints, preventing the evaluation of the most advanced LLMs or large ALMs across all settings.
  • Accent Variation: The study primarily focused on General American English and did not account for accent variation, which could lead to an overemphasis on a single accent and potentially marginalize diverse English accents.
  • Hallucinations and Irrelevant Reasoning: LLMs occasionally produce hallucinations or irrelevant reasoning, requiring further investigation into conditions that lead to errors and strategies to mitigate them.
  • Future Directions: The authors suggest future research could focus on more effectively integrating TextPA with audio-trained models, improving prosody assessment, expanding datasets, and developing automatic reasoning evaluation metrics. They also highlight the importance of balancing intelligibility and nativeness in pronunciation assessment to avoid marginalizing learners' linguistic identities.

Summary updated

BibTeX
@inproceedings{chen2025read,
  author = {Yu-Wen Chen and Melody Ma and Julia Hirschberg},
  title = {{Read to Hear: A Zero-Shot Pronunciation Assessment Using Textual Descriptions and LLMs}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}

Loading related content...

Figure