For decades, Natural Language Processing (NLP) relied on a "pipeline" approach. To understand a sentence, a system first needed a dictionary, then a morphological analyzer to handle word endings, and finally a set of hand-crafted rules to identify parts of speech. This required immense human effort and language-specific expertise.
A significant shift in research has demonstrated that neural networks can bypass these rigid pipelines. By processing text at the level of individual characters rather than whole words, these models autonomously discover the underlying rules of language—from the way sounds harmonize to the way complex medical concepts are structured.
Moving Beyond the Dictionary
Traditional models often struggle with "out-of-vocabulary" words—new terms, typos, or complex inflections that aren't in their pre-defined lists. Matthieu Labeau, Alexandre Allauzen, and Kevin Löser addressed this in Non-lexical neural architecture for fine-grained POS Tagging by building word representations directly from characters.
Instead of looking up a word in a table, their model uses a Convolutional Neural Network (CNN) to "look" at the sequence of letters. This allows the model to recognize that "running" and "jumping" share a suffix, even if it has never seen those specific words before. This "non-lexical" approach proved that end-to-end architectures could surpass traditional systems without any manual feature engineering.
The Discovery of Internal Logic
If a model isn't told about grammar, does it simply memorize patterns, or does it learn the actual rules? Fréderic Godin and colleagues explored this "black box" in Explaining Character-Aware Neural Networks for Word-Level Prediction: Do They Discover Linguistic Rules?. They found that character-level models independently discover established morphological rules, such as gender and plurality suffixes, without explicit supervision.
This internal discovery extends even to the phonetic and sub-character level. In Korean, characters are composed of smaller components (graphemes). While traditional methods involve splitting these manually, Andrew Matteson and Heuiseok Lim showed in Rich Character-Level Information for Korean Morphological Analysis and Part-of-Speech Tagging that Bi-LSTM models can implicitly capture these sub-character features. Similarly, David Guriel and Omer Goldman demonstrated in Morphological Inflection with Phonological Features that models learn phonological rules like vowel harmony directly from character distributions, making explicit linguistic features redundant.
End to End Sequence Labeling
The power of this approach is most evident when character-level features are combined with global context. Xuezhe Ma and Eduard H. Hovy pioneered a "truly end-to-end" architecture in End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF.
Their model uses a three-tier approach:
- CNNs extract morphological information from characters (e.g., identifying prefixes/suffixes).
- Bi-LSTMs process the sequence of words to understand context.
- CRFs ensure the final labels (like "Person" or "Location") make sense in a sequence.
By automating feature discovery, this architecture matches the performance of systems that previously required decades of human-designed rules.
Identifying Patterns in Specialized Domains
Because these models learn from raw text, they are highly adaptable to specialized fields where human experts are scarce. In Neural Networks for Joint Sentence Classification in Medical Paper Abstracts, Franck Dernoncourt and Peter Szolovits used hybrid word-character embeddings to automatically categorize sentences in medical abstracts, reducing the need for domain-specific dictionaries.
Perhaps most strikingly, this autonomous discovery can be used for clinical diagnosis. Sweta Karlekar and Mohit Bansal showed in Detecting Linguistic Characteristics of Alzheimer's Dementia by Interpreting Neural Models that neural models can identify pathological linguistic patterns in speech transcripts. By "peeking inside" these models, the researchers rediscovered known clinical markers of Alzheimer's and uncovered new, task-specific grammar patterns that traditional analysis had overlooked.
The recurring evidence across these papers suggests that linguistic structure is not something that must be given to a model; it is something the model can find if it is allowed to look closely enough at the characters themselves.