Modern Natural Language Processing often relies on character-level or subword-level generative models to handle tasks like speech recognition, machine translation, and transliteration. While these models are highly flexible, they suffer from a fundamental flaw: they lack an inherent understanding of "existence." A neural network can easily generate a character sequence that is phonologically plausible but lexically non-existent—a phenomenon often called hallucination.
To solve this, researchers use lexical constraints, which act as a symbolic "sanity check" on probabilistic outputs. By anchoring generative models to external dictionaries or large-scale corpora, systems can ensure that every generated string corresponds to a valid word in the target language.
The Gap Between Probability and Validity
Character-level models, such as those used for reconstructing simplified scripts or generating abbreviations, often produce outputs that are semantically or lexically invalid because they lack access to a formal dictionary. As Chenchen Ding, Masao Utiyama, and Eiichiro Sumita observe in Simplified Abugidas, without these constraints, a model might predict a sequence of characters that simply does not form a recognized word.
This problem is particularly acute in end-to-end (E2E) speech recognition. Systems using subword units can "invent" words during decoding. Dongwei Jiang and colleagues found that implementing dictionary-based constraints during the beam search process significantly improves accuracy by filtering out candidates that fail to form valid lexical entries.
Dictionaries as Anchors
The most straightforward way to enforce validity is a post-processing filter. In the normalization of historical text, Tanja Säily and Jörg Tiedemann demonstrated that filtering Neural Machine Translation (NMT) candidates through the Oxford English Dictionary prevents the model from generating plausible-looking but non-existent "modernized" spellings.
Similarly, in the field of cognate production, Lisa Beinborn and Iryna Gurevych used target language dictionaries to prune phonologically plausible strings generated by statistical translation. This ensures that the system doesn't just learn how words sound across languages, but which words actually exist.
Snapping to the Nearest Valid Word
When a model produces an invalid string, the system doesn't have to simply reject it. Instead, it can "snap" the prediction to the nearest valid entry in a lexicon. This is typically done using a distance function, such as edit distance.
- Morphosyntactic Tagging: Tiberiu Boros and Dan Tufiş used this to map raw neural predictions to the nearest valid label in a predefined lexicon, preventing the generation of impossible attribute combinations.
- Ancient Languages: For Ancient Greek inscriptions, Evelien de Graaf and Silvia Stopponi applied edit distance to correct invalid lemmas, ensuring the output of their neural lemmatizer matched a reference dictionary.
- Speech Annotation: In recent work on Japanese speech annotation, Xiaolong Lin and Jiawang Liu used dictionary-enhanced decoding to map phonetic hallucinations back to valid pronunciations, effectively "snapping" model guesses to linguistic reality.
Empirical Validation via Corpora
In many cases, a static dictionary is insufficient, especially for low-resource languages or evolving dialects. Here, researchers turn to large-scale text corpora or the web to provide empirical evidence of a word's existence.
Kostadin Cholakov and Gertjan van Noord pioneered "error mining" techniques that validate generated morphological paradigms by checking their frequency in web corpora. If a generated word form never appears in natural usage, it is likely a hallucination of the grammar rules. Similarly, Garrett Nicolai and Grzegorz Kondrak used corpus frequency to re-rank the outputs of generative lemmatizers, ensuring that the predicted base forms are valid entries in the language's actual vocabulary.
Preventing Error Accumulation
Lexical constraints are not just for final output; they are critical during the training process itself. In unsupervised or self-training scenarios, a model uses its own predictions to improve. Without a "grounding" mechanism, the model can drift into a feedback loop of nonsense.
Shyam Upadhyay and Dan Roth addressed this in Bootstrapping Transliteration with Constrained Discovery. By cross-referencing generated outputs against a dictionary of valid tokens, they ensured that only linguistically plausible candidates were used to augment the training data, preventing the accumulation of errors that typically plagues self-supervised systems.
By combining the creative power of neural networks with the structural rigor of lexicons and corpora, these researchers have created a hybrid architecture: the model proposes what is likely, while the lexical constraint determines what is possible.