Supervised machine learning has driven significant progress in Natural Language Processing (NLP), but it relies on a fragile foundation: the availability of massive, human-annotated datasets. As noted by Ng, this dependency creates a significant barrier for "resource-scarce" languages and specialized domains where manual labeling is prohibitively expensive or time-consuming. To scale language systems globally, researchers have developed methods to bypass this "annotation bottleneck" by extracting structure directly from unlabeled text.
Iterative Refinement through Expectation Maximization
One of the most intuitive ways to learn without labels is to treat the missing information as a hidden variable. The Expectation-Maximization (EM) algorithm provides a framework for this: start with a simple guess, use that guess to label raw data, and then use that newly "labeled" data to improve the model.
For example, Cherry et al. applied this to pronoun resolution. By observing patterns in large volumes of unlabeled text, their system learned to associate pronouns with antecedents based on the statistical likelihood of specific word pairings. Similarly, Chen et al. demonstrated that an unsupervised probabilistic model for Chinese zero pronouns—gaps in sentences where a pronoun is implied—could rival supervised systems by training on overt pronouns using EM. Ng further showed that treating coreference as a document-level clustering problem solved via EM allows models to incorporate global linguistic constraints without needing a single human-provided "gold" label.
Leveraging Cross-Lingual Consensus
If labeling data in a new language is too hard, why not use a language we already understand? Cross-lingual and bilingual methods exploit the fact that while languages differ in syntax, they often share the same underlying semantic entities.
Che et al. demonstrated that unannotated bilingual text can improve Named Entity Recognition (NER) by forcing models to reach a consensus across languages. If a phrase is identified as a person in an English sentence, its Chinese translation is likely a person too. Ni et al. and Kundu et al. extended this logic to "zero-shot" scenarios, where a model trained on English data can be projected onto target languages like Spanish or Chinese without any native-language training data. This effectively allows the "wealth" of English annotations to subsidize the development of tools for lower-resource languages.
Discovering Latent Linguistic Structures
Another approach is to treat language as a graph or a set of latent categories that can be discovered through clustering. Biemann showed that Part-of-Speech (POS) tags can be induced entirely from raw text by using graph clustering; words that appear in similar contexts are grouped together, automatically forming syntactic categories.
For more complex tasks like event coreference—determining if two mentions refer to the same real-world event—Bejan et al. and Klein utilized nonparametric Bayesian models. These models are powerful because they don't require the researcher to pre-define the number of categories. Instead, they discover the "hidden" structure of events or grammar rules by finding the most parsimonious explanation for the patterns seen in the raw data. Qin et al. took this a step further by modeling mentions as vertices in a hypergraph, using shared properties like gender or semantic type to group them into entities without supervised guidance.
Bootstrapping and Self-Improvement
Bootstrapping starts with a "seed"—a tiny amount of knowledge—and uses it to harvest more information from a massive unlabeled corpus. Federico et al. built a high-performing Italian NER system using only a small list of names and a large pile of newspaper text. By identifying the contexts in which those known names appeared, the model learned to recognize new, unknown names in similar contexts.
In the era of Large Language Models (LLMs), this concept has evolved into "self-improvement." Xie et al. developed a framework where an LLM boosts its own NER performance by generating its own high-quality demonstrations from an unlabeled corpus. By checking for consistency across its own outputs, the model filters out noise and refines its understanding in a strict zero-shot environment. As Luo et al. demonstrated, even the latent semantic structures within pre-trained word embeddings contain enough information to perform NER without ever seeing a human-provided label.
Key Foundational Work
An Expectation Maximization Approach to Pronoun Resolution (2005) — contributed 1 concept elements to this direction.
Bootstrapping Named Entity Recognition for Italian Broadcast News (2002) — contributed 1 concept elements to this direction.