When a Mandarin or Vietnamese speaker says a person's or place's name to a multilingual assistant, the system must find the right entry in a knowledge graph (a database of entities and the relations between them), often an English one. The usual pipeline transcribes the audio, turns the text into a flat phoneme string, and fuzzy-matches that string against entity names. In a tonal language that last step throws away the pitch contour. Pitch is what separates otherwise identical syllables: Mandarin ma can mean "mother," "hemp," "horse," or "scold." For rare names that a speech recognizer's language model has never seen, the tone in the audio may be the only evidence that tells two candidate entities apart. This note proposes linking spoken tonal mentions to knowledge-graph entities by matching them on joint syllable-plus-tone units, called tonemes, which are scored directly from the audio.
The practical obstacle is that toneme inventories are large and each unit gets little training data. The natural fix is to let the linker label unlabeled speech for itself: an utterance confidently linked to an entity supplies a toneme transcription, because the entity's name fixes the tones. That creates a feedback loop, and loops like this can amplify their own mistakes. We borrow one idea from control of nonlinear systems. We measure the loop's error-transfer curve, meaning how the fraction of wrong pseudo-labels in one round maps to the fraction in the next. We then admit pseudo-labels only where that curve's slope stays below one, which is the region where errors shrink rather than compound.
Core Claim
We claim two things. First, for spoken entity names in tonal languages, matching audio to candidate names on tonemes beats both tone-blind phonetic matching and text-based matching of speech-recognizer output. The gain should be concentrated on names whose toneless pronunciation collides with other entities in the graph. Second, knowledge-graph-confirmed self-training can reduce the data sparsity of toneme models, but only when pseudo-labels are admitted under a measured contraction condition. An uncontrolled loop reaches a tipping point beyond which error grows. That tipping point can be located empirically and avoided.
We do not claim formal Lyapunov certificates. The control literature supplies a lens and a design pattern here, not a proof. We say this plainly because the file's control results hold only for linear plants, and this loop is nothing like one.
Relation to Prior Work
Three lines of work motivate the gap. The first shows that tone must be modeled jointly with segments. Hwang et al.'s tone-related MLP posteriors and Li et al.'s spectro-temporal evidence with tonal acoustic units found that collapsing to tone-agnostic phones erases lexical evidence before any downstream model can use it. Dey et al.'s Mizo recognizer found that explicit toneme units beat merely appending pitch features: DNN-HMM word error rate was 11.96% with toneme modeling, against 12.27% with appended pitch features and 12.63% with neither. Under 5 dB white noise the tone-modeled system reached 31.57%, against 54.38% for spectral features alone. Cui et al.'s low-resource work added a pitch measure that stays robust in noise, called fundamental frequency variation, together with a tonal lexicon for Vietnamese. We reuse this toneme-plus-robust-pitch recipe as the acoustic side of our system. None of these papers touch entities or knowledge graphs.
The second line matches entities using pronunciation, and it is tone-blind. XLEnt's LSP-Align mines 164 million cross-lingual entity pairs by combining lexical, semantic, and transliteration-based phonetic signals, and it reports an average Fuzzy-F1 of 0.86. Tao et al. pair phonetic similarity with temporal co-occurrence in news. Both work over text strings. The closest system to ours, Phonetic Embedding for ASR Robustness in Entity Resolution, trains a Siamese encoder over grapheme-to-phoneme output of English speech-recognizer text and reports a 49.9% relative error reduction over lexical search on its video domain. It is monolingual, and it builds its phonetic representation from recognizer text rather than from audio. Our system differs in three ways: it scores tonemes from the acoustic signal, it targets tonal source languages, and it resolves into an English graph through a cross-lingual alignment.
The third line aligns knowledge graphs across languages and supplies the mapping from the Chinese entity to its English counterpart. It also supplies the cautionary tale about self-training. Relevant systems include JAPE, GCN-Align, OTEA, and ARNM-DAE2A, which reports 83.91% Hits@1 on DBP15K Chinese–English. BootEA gains 13–18% Hits@1 from iterative pseudo-labeling. It needs a heuristic global editing step to undo errors that accumulate, and it gives no account of when accumulation takes over.
We reuse BootEA's bipartite one-to-one matching. We do not reuse its heuristic editing. We replace it with an admission rule based on measured error transfer. That rule is inspired by the constraint tightening of indirect-feedback stochastic MPC, where noisy measurements enter the cost but never the nominal plan. It also takes a cue from the observation in data-driven MPC that removing terminal constraints makes the loop diverge.
Research Program
The linker works in three stages. First, an acoustic model with a toneme output layer produces frame-level toneme posteriors from the utterance. It is pretrained on Mandarin and Vietnamese speech; for Vietnamese it follows the Babel limited-pack recipe of Cui et al. Second, each candidate Chinese entity label in the knowledge graph is expanded into its toneme sequence, which is deterministic from the characters. The utterance is then scored against each candidate by forced alignment, giving a log-likelihood. Candidates come from a tone-blind phonetic shortlist, so every query does not have to be aligned against the whole graph. Third, the top Chinese entity is mapped to English through a fixed alignment model trained on DBP15K. Because the alignment model is fixed, the gains we measure come from the spoken-mention side.
Self-training runs on a pool of unlabeled spoken mentions. In each round $t$, the linker proposes (utterance, entity) pairs. The admitted set $A_t$ has an unknown fraction $e_t$ of wrong links. The toneme model is retrained on the toneme transcriptions implied by $A_t$, and relinking produces round $t+1$ with error $e_{t+1}$. We call the curve
$$e_{t+1} = f_\tau(e_t)$$
the error-transfer curve, where $\tau$ is the confidence threshold for admission. We estimate $f_\tau$ on a labeled calibration split by injecting controlled fractions of wrong labels and measuring the resulting error after one retraining round.
The loop contracts toward a low-error fixed point when the slope satisfies
$$\frac{d f_\tau}{d e}(e) < 1$$
over the range of errors actually encountered. This is the ordinary stability condition for a one-dimensional discrete-time map. We expect a nonlinear shape. The slope should be small near zero, because a few wrong tone labels are outvoted by correct ones. Beyond a critical error level it should rise steeply, because the model starts learning the linker's confusions, such as systematically swapping two contour tones.
The controller picks the loosest threshold $\tau_t$ that keeps the estimated slope below $1-\delta$ at the current estimated error, where $\delta$ is a safety margin. As a variant, we route low-confidence links into training only as soft loss weights and never as hard labels. This is the analogue of Hewing et al.'s indirect feedback, and it separates what the controller contributes from what soft labeling alone contributes.
Baselines
The naive reference is lexical search. A strong off-the-shelf Mandarin or Vietnamese recognizer produces text, and that text is matched to Chinese entity labels by exact and edit-distance lookup, then mapped to English with the same alignment model. This shows how much any phonetic modeling buys at all.
The strongest existing competitor is the Siamese phonetic-embedding resolver of Phonetic Embedding for ASR Robustness in Entity Resolution. It is the only published system in the file built for spoken mentions corrupted by recognition errors. We retrain it on our data with a Mandarin grapheme-to-phoneme front end. We give it the same N-best hard-negative augmentation and the same 10% recognizer-data mixture its authors found best. It gets the same training utterances and the same hyperparameter search budget as our model, and the same acoustic pretraining for its recognizer. It reports a 49.9% relative error reduction over lexical search on video and 35.3% on books. For spoken cross-lingual entity linking in tonal languages, no reported result found on Lacuna, so the bar is our own reproduction. Beating it means a statistically significant Hits@1 gain on the English entity overall, and a larger gain on the homophone-collision subset.
Two ablations isolate the key ingredients. The first gives the same resolver tone-marked phonemes produced from recognizer text, so tones are present but only symbolically. This separates "tone as a symbol from recognizer text" from "tone as acoustic evidence." The second replaces toneme units with spectral features plus appended pitch features, following the weaker condition in Dey et al. For the self-training loop, the baselines are a fixed high threshold, BootEA-style bipartite matching with global editing, and no self-training at all. All of them get the same unlabeled pool and the same number of rounds. The controller wins if it reaches a lower final error than every fixed threshold, without the collapse runs that uncontrolled loops show.
Why not just…?
Why not just run a strong recognizer and match its characters, since Chinese characters already encode tone? This is the most serious threat. For common names with a good language model, it may match our system. It fails where the recognizer's language model has no prior, on rare and long-tail names. XLEnt found that frequency-based methods break down on entities seen three or fewer times, and the spoken-resolution paper found lexical search breaks badly on phonetic variants. We stratify results by entity frequency and by collision count. If the recognizer-plus-match pipeline equals our system on the long tail as well, the first claim is false, and we will report it that way.
Why not just append pitch features to the phonetic embedding? Dey et al. report that appending pitch features beats nothing but falls short of explicit toneme modeling, and the gap widens sharply in noise (31.57% versus 54.38% at 5 dB for tone modeling against spectral features alone). Our appended-pitch ablation tests whether that holds when the task is entity linking rather than transcription.
Why not just use a conservative fixed threshold or BootEA's editing for self-training? A fixed high threshold is safe, but it admits few examples, and the toneme sparsity problem persists. A low threshold admits many examples but risks crossing the tipping point. BootEA's editing works in practice but gives no signal for when it will fail. If a single well-tuned fixed threshold matches the controller at every noise level, the control contribution reduces to "tune the threshold," and we will say so. The controller's claimed advantage is that it adapts as the estimated error changes across rounds and across noise conditions, where no single threshold is best.
Proposed Experiments
We build the evaluation set from the DBP15K Chinese–English test split. We sample person and place entities and stratify them by how many other Chinese labels in the graph share their toneless pinyin. Native speakers record several readings of each name, and each name is also read inside a short carrier sentence. We add white and pink noise at 5 dB, as Dey et al. did, to test robustness. A larger text-to-speech set gives scale for self-training. We treat synthesized speech only as unlabeled pool data and never as test data, because it realizes tones perfectly. A smaller Vietnamese extension uses Vietnamese–English entity pairs from XLEnt, if its language coverage includes Vietnamese, which the file does not confirm. The metrics are Hits@1 and MRR on the English entity, reported overall, on the collision subset, and by entity frequency. We also report toneme error rate as a diagnostic.
Three checks are decisive. The first is whether the tone-aware linker's advantage over the phonetic-embedding resolver grows monotonically with collision count and with rarity. If it does not, tone is not doing the disambiguating work we claim. The second is whether the measured error-transfer curves really are nonlinear, with a critical error beyond which the slope exceeds one. It also asks whether uncontrolled loops started above that point degrade while loops started below it improve. This is a falsifiable prediction about the curve's shape, not a tuning result. The third is whether the controlled loop improves toneme accuracy on rare tone–syllable combinations, which are exactly the units that dictionary expansion leaves starved of data, and whether it does so without the collapse seen in uncontrolled runs.
References
- Incorporating tone-related MLP posteriors in the feature representation for Mandarin ASR
- Improved Tonal Language Speech Recognition by Integrating Spectro-Temporal Evidence and Pitch Information with Properly Chosen Tonal Acoustic Units
- Robust Mizo Continuous Speech Recognition
- Recent improvements in neural network acoustic modeling for LVCSR in low resource languages
- XLEnt: Mining a Large Cross-lingual Entity Dataset with Lexical-Semantic-Phonetic Word Alignment
- Unsupervised Named Entity Transliteration Using Temporal and Phonetic Correlation
- Phonetic Embedding for ASR Robustness in Entity Resolution
- Cross-Lingual Entity Alignment via Joint Attribute-Preserving Embedding
- Entity Alignment for Cross-lingual Knowledge Graph with Graph Convolutional Networks
- Improving Cross-lingual Entity Alignment via Optimal Transport
- An attribute-enhanced relationship-aware neighborhood matching model with dual attention
- Bootstrapping Entity Alignment with Knowledge Graph Embedding
- Recursively Feasible Stochastic Model Predictive Control using Indirect Feedback
- Data-Driven Model Predictive Control with Stability and Robustness Guarantees