Speech Sense Disambiguation: Tackling Homophone Ambiguity in End-to-End Speech Translation

Tengfei Yu, Xuebo Liu, Liang Ding, Kehai Chen, Dacheng Tao, Min Zhang

Introduction

End-to-end speech translation (ST) has made significant strides, yet it faces a unique challenge: homophones. Words like "plane" and "plain" are acoustically identical, leading to "speech sense" errors where models fail to disambiguate meaning based on audio alone. While traditional cascade systems can address this during transcription or text-based translation, direct end-to-end models often lack explicit mechanisms for phonetic disambiguation. This paper introduces Speech Sense Disambiguation (SSD) and the AmbigST framework to address this gap.

Methods

The AmbigST framework integrates homophone awareness into the training process through a multi-stage pipeline involving data annotation, specialized masking, and contrastive learning.

Homophone Dictionary and Annotation

The authors constructed comprehensive homophone dictionaries for English, French, German, and Spanish. Using the Montreal Forced Aligner (MFA), they identified precise timestamps for these ambiguous words within speech audio, resulting in an annotated dataset of 1 million instances.

Construction of the Homophone Dictionary from ST Triple Datasets.

The AmbigST Framework

AmbigST employs a novel training strategy that forces the model to rely on context rather than just acoustic signals.

  1. Homophone-aware Masking: Unlike random masking, AmbigST specifically masks the acoustic segments of homophones. This compels the model to utilize the surrounding "auditory context" to infer the correct word sense.
  2. Multi-level Contrastive Learning (CL):
    • Token-level: Aligns representations of masked homophones with their unmasked counterparts.
  3. Sentence-level: Enhances global semantic understanding using dropout and masking across the entire utterance.
  4. Model-level: Implements self-knowledge distillation to ensure consistent translation output regardless of input masking.
The AmbigST framework overview, including word-level alignment, masking, and multi-level contrastive learning.

Datasets and Evaluation

The framework was evaluated on the MuST-C (En to De, Es, Fr) and CoVoST-2 (De, Es, Fr to En) benchmarks. The researchers also released their homophone dictionaries and the 1M-instance alignment dataset to support further research in the field.

Results and Insights

AmbigST achieved state-of-the-art (SOTA) results on MuST-C, with an average BLEU score of 35.5. Key findings include:

  • Semantic Accuracy: A +0.9 average increase in BLEURT scores indicates improved semantic fidelity.
  • Contextual Robustness: The model showed significant gains in short sentences and for low-frequency words, where contextual clues are typically harder to leverage.
  • Efficiency: Since the homophone-aware mechanisms are applied only during training, the model maintains zero additional computational cost during inference.

Limitations

While effective, AmbigST currently focuses on homophones (different spelling) rather than general polysemy (same spelling). Its benefits are also less pronounced in very long sentences where natural context is abundant. Additionally, the reliance on standard pronunciation models may introduce errors for regional dialects or low-resource languages lacking high-quality alignments.

Summary updated

Abstract

End-to-end speech translation (ST) presents notable disambiguation challenges as it necessitates simultaneous cross-modal and cross-lingual transformations. While word sense disambiguation is an extensively investigated topic in textual machine translation, the exploration of disambiguation strategies for ST models remains limited. Addressing this gap, this paper introduces the concept of speech sense disambiguation (SSD), specifically emphasizing homophones - words pronounced identically but with different meanings. To facilitate this, we first create a comprehensive homophone dictionary and an annotated dataset rich with homophone information established based on speech-text alignment. Building on this unique dictionary, we introduce AmbigST, an innovative homophone-aware contrastive learning approach that integrates a homophone-aware masking strategy. Our experiments on different MuST-C and CoVoST ST benchmarks demonstrate that AmbigST sets new performance standards. Specifically, it achieves SOTA results on BLEU scores for English to German, Spanish, and French ST tasks, underlining its effectiveness in reducing speech sense ambiguity. Data, code and scripts are freely available at https://github.com/ytf-philp/AmbigST.
BibTeX
@inproceedings{yu2024speech,
  author = {Tengfei Yu and Xuebo Liu and Liang Ding and Kehai Chen and Dacheng Tao and Min Zhang},
  title = {{Speech Sense Disambiguation: Tackling Homophone Ambiguity in End-to-End Speech Translation}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.acl-long.435},
}
Figure