PSET: a Phonetics-Semantics Evaluation Testbed

Gianluca Sperduti, Dong NguyenEMNLP 2025

TL;DR

This paper introduces PSET, a novel evaluation testbed designed to assess how well phonetic embeddings prioritize sound over meaning. By testing PSET on existing phonetic and semantic models, as well as large language models (LLMs), the authors reveal that while some phonetic models perform well, newer LLMs show promise, with Gemini 2.5 Flash leading the pack.

What Problem Does This Paper Address?

Modern Natural Language Processing (NLP) heavily relies on embeddings, which are vector representations of text. Most existing embeddings are "semantic," focusing on capturing the meaning of words. However, for tasks involving sound, like identifying rhymes or improving speech recognition, "phonetic" embeddings are crucial. These embeddings should represent words based on their pronunciation. The challenge is that there's a lack of robust evaluation methods specifically for phonetic embeddings, especially for assessing whether they truly prioritize phonetic information over semantic similarity. Existing benchmarks often have limitations, such as being biased towards certain embedding types or not rigorously testing the dominance of phonetic features.

Methods

The core contribution is the Phonetics-Semantics Evaluation Testbed (PSET). PSET is built on the principle that phonetic embeddings should prioritize sound. It uses word quintets, each containing:

  • An anchor word.
  • A homophone (sounds the same, different spelling/meaning).
  • A synonym (different sound, similar meaning).
  • A phonetic-based distractor (IPA transcribed, edit-distance 1 from anchor).
  • A grapheme-based distractor (spelling-based, edit-distance 1 from anchor).

The testbed's design aims to differentiate between models that truly capture phonetic relationships (favoring homophones) and those that are swayed by semantic similarity (favoring synonyms) or superficial spelling similarities.

PSET can be used with any method that assesses word similarity, including static embedding models and prompting LLMs. The authors also detail their process for selecting homophones and synonyms, ensuring semantic precision and minimizing ambiguity. They also describe the creation of phonetic and grapheme-based distractors using IPA transcriptions and edit-distance.

Datasets

The paper introduces the Phonetics-Semantics Evaluation Testbed (PSET), a newly curated dataset of 250 word quintets. This dataset is the primary evaluation benchmark used in the paper.

For training and evaluating existing models, the authors mention:

  • Articulatory Embeddings: Based on IPA symbols.
  • Phoneme2Vec: Trained using ARPAbet and the CMU Pronunciation Dictionary.
  • XPhoneBERT: Trained on IPA transcriptions.
  • BERT and Word2Vec: Standard pre-trained models.
  • LLMs: GPT-4o, Gemini 2.5 Flash, Llama 3.1-8B, OLMO-7B, and OLMO 2-7B.

Key Insights and Ideas

  • Phonetic embeddings should prioritize sound: The fundamental idea behind PSET is that a good phonetic embedding model should identify homophones as more similar to an anchor word than synonyms or grapheme-based distractors.
  • Semantic models are easily fooled: As expected, semantic models like Word2Vec and BERT consistently favored synonyms over homophones, highlighting their focus on meaning.
  • XPhoneBERT's surprising underperformance: Despite being a more recent phonetic model, XPhoneBERT performed the worst among the phonetic models tested, possibly due to its original design for Text-To-Speech tasks rather than general phonetic representation.
  • LLMs can be prompted for phonetic tasks: The study demonstrates that LLMs, when prompted appropriately, can perform phonetic similarity tasks, suggesting they have acquired some phonetic understanding.
  • Technical prompts yield better results for LLMs: LLMs generally performed better when asked phonetic similarity questions using more technical language compared to layman's terms.

Key Results

  • Phonetic Models: Phoneme2Vec performed best among the phonetic models tested on PSET, followed by Articulatory Embeddings. XPhoneBERT was the weakest.
  • LLMs: Gemini 2.5 Flash emerged as the top-performing LLM on PSET, correctly identifying homophones in a high percentage of cases (0.87 average for both technical and layman prompts). GPT-4o followed closely. OLMO 2-7B showed a tendency to pick synonyms, while OLMO-7B struggled with prompt adherence.
  • Frequency Matters: Word frequency impacts model performance, with higher frequency distractors sometimes leading to incorrect predictions.

Limitations and Future Work

  • Dataset Size: The PSET dataset is relatively small due to manual curation. Future work could expand it using automatic methods.
  • Language Specificity: PSET is currently limited to English. Extending it to other languages is a goal, though challenging due to varying rates of homophony.
  • Word Frequency: The inclusion of low-frequency words might affect semantic models. An analysis showed frequency impacts performance, with higher frequency distractors often causing errors.
  • Synonym Ambiguity: Some synonyms might not perfectly match the anchor's most frequent meaning, potentially simplifying the task for semantic models.
  • Confounding Factors: Potential influences like Part-of-Speech, syntactic dependencies, and polysemy were not explicitly controlled for.
  • Model Exposure: The authors acknowledge the possibility that LLMs might have been exposed to the test lists during their training, though their imperfect performance suggests this isn't the sole factor.

Future work could involve expanding the dataset, adapting it to other languages, and further investigating the impact of word frequency and other linguistic factors.

Summary updated

BibTeX
@inproceedings{sperduti2025pset,
  author = {Gianluca Sperduti and Dong Nguyen},
  title = {{PSET: a Phonetics-Semantics Evaluation Testbed}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure