Hearing Through the Spelling: Phonetic Views for Noise-Robust Multilingual Sentence Embeddings

Multilingual sentence embedding models map sentences from many languages into one vector space, so that a sentence and its translation land close together. They power cross-lingual search, parallel-data mining and fact-check retrieval. These models are trained on clean, standard spelling. A large share of real multilingual text is not clean. It includes speech-recognition transcripts that confuse "knight" and "night", Bengali or Hindi typed in improvised Roman spelling, and the same word written in Devanagari by one community and in Perso-Arabic script by another. In each case the words are pronounced consistently but spelled inconsistently. The subword tokenizer then breaks them into unfamiliar pieces, and the sentence vector drifts away from its meaning.

We propose giving the sentence encoder a second view of every input: a sequence of articulatory features, meaning sound descriptors such as "voiced", "nasal" and "plosive". This sequence is derived either from the text through grapheme-to-phoneme conversion or directly from audio through a multilingual phone recognizer. A learned per-sentence gate decides how much to trust the spelling view versus the sound view. The student encoder is trained to reproduce the embedding that a frozen, clean-text teacher assigns to the clean version of the sentence. The bet is narrow and testable. The sound view should add information that noise augmentation on spelling alone cannot supply, specifically on noise types never seen in training, and the gate should learn to turn the sound view down when it would merge homophones.

Core Claim

Our hypothesis is that pronunciation acts as a partly language-independent, spelling-independent code. It collapses many surface variants of a sentence to one representation. Spelling, by contrast, carries the distinctions that pronunciation erases, such as homophones and tone-free romanizations of tonal words. Each view therefore contains information the other lacks, and the right mix varies from sentence to sentence. We make three claims.

First, a dual-view student whose views are combined by a gate will beat an equally trained spelling-only student on cross-lingual retrieval from noisy inputs. Second, the gap will be largest on noise distributions that are absent from training, because augmentation can only teach the noise it is shown. Third, a sample-level measurement of how much information each view uniquely contributes will track noise severity, and the learned gate will track that measurement.

If the second claim fails, meaning spelling-only augmentation matches the dual-view model on held-out noise, then the phonetic view is not earning its place. We will report that outcome as the main finding.

Where This Sits

Three strands of prior work motivate the proposal.

The first strand, in phonetic alignment, shows that comparing sounds rather than characters recovers correspondences that spelling hides. Identifying Cognates by Phonetic and Semantic Similarity aligns phonetic feature sequences to find cognates across sound change. Phonetic Vector Representations for Sound Sequence Alignment learns segment vectors that beat one-hot and hand-built feature encodings on alignment benchmarks. Normalization of Transliterated Words in Code-Mixed Data zeroes the edit cost between phonetically equivalent characters, such as a/o and b/v, to normalize romanized Bengali. All three operate on single words and produce no sentence semantics.

The second strand is the closest existing work, and both papers in it supply method. Multilingual Jointly Trained Acoustic and Written Word Embeddings decomposes phones into articulatory distinctive features so that sound sharing transfers across 12 languages, which removes the alphabet barrier. It stops at isolated-word discrimination. Machine Translation by Projecting Text into the Same Phonetic-Orthographic Space Using a Common Encoding maps Indic scripts into the shared WX notation and gains about 10 BLEU on Nepali–Hindi. It relies on hand-written transliteration tables, evaluates translation generation rather than embeddings, and weakens across language families. That last result constrains us: a phonetic projection is no universal bridge, and the gate exists partly because of it.

The third strand, multilingual embeddings, supplies the training recipe and the evaluation logic. Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation trains a student to imitate a teacher's embeddings, and we reuse that teacher–student setup with the clean sentence as teacher input. Margin-based Parallel Corpus Mining with Multilingual Sentence Embeddings introduced ratio-margin scoring against hubs, which are vectors that become everyone's nearest neighbour. It also notes that margin scoring penalizes dense neighbourhoods. A phonetic view that merges homophones could create exactly such neighbourhoods, so we measure hubness directly. Language Concept Erasure for Language-invariant Dense Retrieval erases language identity from embeddings. We do not erase the phonetic view's language-specific detail, because it is what aligns cognates, and we test whether erasure and phonetic input conflict.

Efficient Quantification of Multimodal Interaction at Sample Level supplies our analysis tool. Its estimator splits, per example, the information two inputs carry about a target into three parts: redundant (shared by both), unique (present in only one), and synergistic (available only from the combination). That paper also reports that out-of-distribution robustness leans on non-redundant information, and that interaction-aware fusion beats uniform or weighted averaging. This is the prediction we test in a new setting: two views of a single sentence rather than two sensors.

The literature contains an apparent contradiction. Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spaces argues that speech–text alignment works only because its speech embeddings ignore how words sound. The acoustic-written word embedding work argues that sound is the bridge. Our design treats both as partly right. Sound is a supplementary view, never a replacement, and the gate is where the disagreement gets settled empirically.

Research Program

The student has two encoders over one sentence. The spelling encoder $f_o$ is initialized from a multilingual text encoder and reads the possibly noisy surface text $\tilde{x}$. The sound encoder $f_p$ is a small Transformer over articulatory feature vectors $\phi(\tilde{x})$. When only text is available, $\phi$ comes from a universal grapheme-to-phoneme converter applied after language-agnostic romanization. When audio exists, $\phi$ comes from a multilingual phone recognizer. A gate $g \in [0,1]$ is computed from both views' pooled states. Training minimizes

$$ \mathcal{L} = \big\lVert\, g\, f_o(\tilde{x}) + (1-g)\, W f_p(\phi(\tilde{x})) - T(x) \,\big\rVert^2 + \lambda\, \mathcal{L}_{\text{par}}, $$

where $T(x)$ is the frozen teacher's embedding of the clean sentence $x$, $W$ projects sound-encoder outputs into the teacher's space, and $\mathcal{L}_{\text{par}}$ is the same squared distance applied to the clean translation of $x$ in another language. Weighted by $\lambda$, this second term keeps the space cross-lingual. Noisy inputs $\tilde{x}$ come from three noise families: speech-recognition transcripts, romanization of native-script text under several conventions, and phonetically motivated character substitution. Training uses only a subset of these families, so that the others serve as held-out tests.

The analysis component estimates, per test sentence, the redundant and unique information that the spelling and sound views carry about retrieval identity. Retrieval identity is defined as the index of the correct parallel sentence within a fixed candidate pool, which gives the estimator a discrete target. This is an adaptation of the bimodal estimator, and we will validate it on a synthetic case where the answer is known: inputs whose spelling is pure noise but whose pronunciation is intact. If the estimates are unstable, the analysis falls back to simpler ablation-based attribution, and the method claims stand on retrieval numbers alone.

Baselines

The naive reference point is the frozen teacher itself, a standard multilingual sentence encoder applied directly to the noisy text. On clean Tatoeba sentence retrieval, XLM-R reaches 57.3% accuracy and mBERT 38.7% (XTREME). We will rerun the teacher on our noisy variants of the same sentences, because no reported result was found on Lacuna for multilingual sentence retrieval from ASR-corrupted or romanized queries. The teacher's drop from clean to noisy input defines the gap any method must close.

The strongest competitor is a spelling-only student. It has the same architecture as $f_o$ and the same teacher, data, noise families, number of training steps and hyperparameter search budget, but no sound view and no gate. This is the noise-augmented distillation a skeptical reviewer would propose first. Comparing against it isolates exactly what pronunciation adds. Beating it means a statistically significant retrieval gain, assessed with paired bootstrap over queries, on held-out noise families, while staying within one point of it on clean input.

The pipeline competitor normalizes before embedding. On the Bengali–English data it applies the two-stage normalizer of Normalization of Transliterated Words in Code-Mixed Data, which reports 90.27% word-level normalization accuracy and raises downstream sentiment accuracy from 80.97% to 82.47%. On Indic cross-script pairs it applies WX projection as in Machine Translation by Projecting Text into the Same Phonetic-Orthographic Space, and then embeds with the teacher. For speech-derived inputs, the competitor averages teacher embeddings over the recognizer's n-best hypotheses. This borrows the observation from FastCorrect 2 that alternative hypotheses share pronunciation and that combining them recovers errors.

Three ablations isolate the gate: concatenation, a fixed 50/50 average, and sound-only. The fixed average is the "uniform averaging" that sample-level interaction-aware fusion beat in Efficient Quantification of Multimodal Interaction at Sample Level. A further ablation replaces articulatory features with phoneme IDs, mirroring the feature-versus-ID comparison of the acoustic-written word embedding paper. A last variant adds language-concept erasure to the dual-view student to test whether it conflicts with phonetic input.

Why not just…?

Why not just augment the text encoder with spelling noise? This is the real threat. With enough synthetic misspellings, a subword encoder may learn the same invariances on its own. Augmentation can only cover noise it was shown, whereas pronunciation collapses unseen spelling conventions to the same code by construction. That is the reason for the held-out-noise-family protocol. We include augmentation in our own model, so the comparison asks only whether sound adds something on top. If the spelling-only student matches us on held-out families, we will conclude that the phonetic view is unnecessary.

Why not just normalize or transliterate first, then embed? Normalization needs a parallel transliteration lexicon. The Bengali normalizer was trained on 6,000 hand-made pairs, and WX needs expert tables per script, so neither covers most languages. Pipelines also commit to one hard choice per word, and that choice propagates errors. The pipeline baseline tests this directly. Where lexicons exist it may win on in-domain words, and we expect our advantage to appear on out-of-lexicon tokens and on languages without converters.

Why not just run a better speech recognizer and use its transcript? Recognizer errors on code-mixed speech are severe. Even on transcripts, CoMuMDR reports only 0.5547 F1 for its best model on combined link and relation parsing. The n-best-averaging baseline gives the cascade its fairest form. If n-best averaging matches the audio-derived phonetic view, the gain from audio is merely recognizer diversity, and the pronunciation signal itself adds nothing.

Proposed Experiments and Decisive Checks

The main retrieval testbed is built from Fleurs within XTREME-S, whose n-way parallel speech lets us pair real recognizer transcripts and audio-derived phones in one language with clean translations in others. XTREME-S lists speech retrieval as optional with results still being collected, so no reported number exists for this setting. Text-only robustness uses Tatoeba and BUCC sentences corrupted with held-out noise families. Scoring uses accuracy and F1 respectively, under both cosine and ratio-margin scoring. Cross-script robustness uses Hindi–Urdu and Hindi–Nepali pairs from CVIT-PIB, retrieving romanized user-style input against native-script targets. We also test retrieval with noisy queries on MIRACL, with nDCG@10 as the metric. For reference, the published hybrid BM25+mDPR system reaches 0.578 average nDCG@10 on clean queries. Downstream utility is measured on CoMuMDR discourse parsing, substituting our embeddings for paraphrase-xlm-r-multilingual-v1. The reported structure-aware model reaches 0.9530 link-prediction F1 with that encoder, and 0.5547 F1 on link plus relation.

Four checks decide the outcome. The first is the held-out-noise gap between the dual-view and spelling-only students; this is the result that justifies or kills the idea. The second is the homophone check. We build minimal pairs of sentences that differ only in a homophone or a tone-dropped romanization, and measure whether the dual-view model confuses them more than the teacher does. We also count hub occurrences, meaning how often a target appears among other queries' top-k neighbours. Both should stay flat if the gate is doing its job.

The third check is the interaction analysis. The estimated unique contribution of the sound view should rise with noise severity and be near zero on clean text. The correlation between the gate value $1-g$ and that estimate should be clearly positive. The fourth is the cross-family check, Hindi–Tamil against Hindi–Nepali. Given the WX results, we expect smaller gains across families, and the gate should shift weight back toward spelling there. If it does not, the model is overusing pronunciation.