Introduction
End-to-end speech translation (ST) has made significant strides, yet it faces a unique challenge: homophones. Words like "plane" and "plain" are acoustically identical, leading to "speech sense" errors where models fail to disambiguate meaning based on audio alone. While traditional cascade systems can address this during transcription or text-based translation, direct end-to-end models often lack explicit mechanisms for phonetic disambiguation. This paper introduces Speech Sense Disambiguation (SSD) and the AmbigST framework to address this gap.
Methods
The AmbigST framework integrates homophone awareness into the training process through a multi-stage pipeline involving data annotation, specialized masking, and contrastive learning.
Homophone Dictionary and Annotation
The authors constructed comprehensive homophone dictionaries for English, French, German, and Spanish. Using the Montreal Forced Aligner (MFA), they identified precise timestamps for these ambiguous words within speech audio, resulting in an annotated dataset of 1 million instances.
The AmbigST Framework
AmbigST employs a novel training strategy that forces the model to rely on context rather than just acoustic signals.
- Homophone-aware Masking: Unlike random masking, AmbigST specifically masks the acoustic segments of homophones. This compels the model to utilize the surrounding "auditory context" to infer the correct word sense.
- Multi-level Contrastive Learning (CL):
- Token-level: Aligns representations of masked homophones with their unmasked counterparts.
- Sentence-level: Enhances global semantic understanding using dropout and masking across the entire utterance.
- Model-level: Implements self-knowledge distillation to ensure consistent translation output regardless of input masking.
Datasets and Evaluation
The framework was evaluated on the MuST-C (En to De, Es, Fr) and CoVoST-2 (De, Es, Fr to En) benchmarks. The researchers also released their homophone dictionaries and the 1M-instance alignment dataset to support further research in the field.
Results and Insights
AmbigST achieved state-of-the-art (SOTA) results on MuST-C, with an average BLEU score of 35.5. Key findings include:
- Semantic Accuracy: A +0.9 average increase in BLEURT scores indicates improved semantic fidelity.
- Contextual Robustness: The model showed significant gains in short sentences and for low-frequency words, where contextual clues are typically harder to leverage.
- Efficiency: Since the homophone-aware mechanisms are applied only during training, the model maintains zero additional computational cost during inference.
Limitations
While effective, AmbigST currently focuses on homophones (different spelling) rather than general polysemy (same spelling). Its benefits are also less pronounced in very long sentences where natural context is abundant. Additionally, the reliance on standard pronunciation models may introduce errors for regional dialects or low-resource languages lacking high-quality alignments.
Summary updated