The Density Problem in Sequence Supervision
In standard text classification, a single label describes an entire document or sentence. A model learns from a relatively sparse signal: hundreds of words map to a single binary or multi-class decision. In sequence labeling tasks—such as Named Entity Recognition (NER), Part-of-Speech (POS) tagging, and semantic parsing—supervision must be dense. Every single token requires a corresponding label.
As noted by Steven Bethard et al., neural sequence models demand significantly larger quantities of human-annotated data than traditional rule-based systems to reach optimal accuracy. This creates a severe annotation bottleneck. When moving to specialized domains (such as medical text, explored by Franck Dernoncourt et al.) or low-resource languages, manual token-level annotation becomes prohibitively expensive. To bypass this bottleneck, researchers have developed techniques to extract dense structural signals from unlabeled text, noisy supervision, cross-task transfer, and semantic reformulations.
Extracting Structure from Unlabeled Text
When human labels are scarce, models must rely on the intrinsic statistical and syntactic regularities of raw text. One intuitive approach is to exploit empirical language constraints. For instance, Yingchuan Wong leverages the observation that most named entities belong to only one semantic class across a corpus. By using an initial weak classifier to "machine-tag" a massive unlabeled corpus, the author extracts "majority tags" for entities and feeds them back into the final model as features. Similarly, Behrang Mohit et al. combine constituency and dependency parses to bootstrap classifiers, while Wei Li et al. use a syntactic topic model (HMM-LDA) to discover context-aware word clusters from unlabeled data, serving as continuous features for Conditional Random Fields (CRFs).
A more tightly integrated approach couples discriminative sequence labeling with generative sequence reconstruction. Traditional discriminative models only learn the mapping from words to labels, ignoring the underlying distribution of the words themselves. To capture this distribution, Xiao Zhang introduced the Neural CRF Autoencoder (NCRF-AE).
Discriminative Path: Input Sentence ───► [ Encoder ] ───► Predicted Labels (CRF)
│
Generative Path: ▼
Input Sentence ◄─── [ Decoder ]
The NCRF-AE treats the labeling process as an encoding step and adds a generative decoding step to reconstruct the original sentence. By forcing the hidden representations to retain enough information to rebuild the input text, the model learns the language's syntax and vocabulary from readily available unlabeled corpora, drastically reducing the amount of labeled data needed to constrain the CRF. Taking this probabilistic framing further, Mingda Chen et al. developed Variational Sequential Labelers (VSLs), bringing neural variational inference to sequence labeling to model latent sequence structures without explicit supervision.
Learning from Incomplete and Noisy Supervision
Often, researchers do not have a perfectly labeled corpus, but they might have access to partially labeled data or noisy automated annotations. Traditional sequence models like CRFs break down if even a single token is missing a label because the transition lattice cannot be fully computed.
To resolve this, Hisashi Kashima proposed a parameter estimation method that allows CRFs to learn from incomplete annotations. By marginalizing over the unknown labels during training, the model sums over all valid paths in the sequence lattice, allowing it to learn from certain tokens without being penalized by uncertain or missing ones. In a similar spirit of breaking rigid sequence dependencies, Shinsuke Mori et al. developed a "pointwise" approach for text processing, decomposing the sequence into independent classification steps so the system can learn from partially labeled data and specialized dictionaries.
When labels are generated automatically via distant supervision (e.g., matching text against a database), they introduce systematic errors. Lange et al. addressed this by clustering words based on their features and calculating unique, feature-dependent confusion matrices for each cluster. This allows the model to explicitly learn how the automated labeling system makes mistakes, effectively filtering out noise. To select the most reliable tokens during self-training, Chengyu Wang et al. introduced SeqUST, which uses Bayesian uncertainty estimation to filter out unreliable pseudo-labels in low-resource settings.
Cross-Task and Cross-Lingual Knowledge Transfer
If token-level labels are missing entirely for a target task or language, models can transfer structural knowledge from related tasks or high-resource source languages.
Cross-Task Transfer
Marek Rei et al. demonstrated that cheap sentence-level labels can mitigate token-level scarcity. By jointly training a model to predict both sentence-level and token-level classifications, the token-level predictions act as direct supervision for a self-attention mechanism, guiding the model to focus on the correct words.
Similarly, Zhen Wu et al. developed the Latent Opinions Transfer Network (LOTN), which extracts target opinion words by transforming global attention weights from a pre-trained sentiment classifier into target-specific sequence signals. Even non-textual auxiliary tasks can help: Sigrid Klerke et al. improved sentence compression by training a model to predict human eye-tracking gaze data, helping it differentiate core semantic elements from non-essential information.
Cross-Lingual Transfer
For languages completely lacking annotated data, researchers use teacher-student frameworks to project labels across language barriers. The authors developed a framework that uses models trained on high-resource source languages as "teachers" to project soft labels onto unlabeled target-language text.
Chunming Hu et al. expanded this into an unsupervised Multiple-Task and Multiple-Teacher (MTMT) framework, combining an entity-recognition teacher and a word-similarity teacher to guide a student model without requiring parallel corpora. For low-resource part-of-speech tagging, Guillaume Wisniewski et al. utilize cross-lingual ambiguous learning to navigate the lack of exact target annotations.
Reformulating Sequences as Semantic Matches
Instead of treating sequence labeling as a traditional token-classification problem (where a model maps a token to an arbitrary integer ID), recent work leverages the rich semantic spaces of Pre-trained Language Models (PLMs).
Traditional NER: [ "London" ] ───► Classify ───► ID: 3 (Location)
Semantic Matching: [ "London" ] ───► Vector Space ◄─── [ "location" ]
▲
(Close Match)
Jie Ma et al. introduced a two-tower BERT architecture that treats entity labels (like "person" or "organization") as meaningful words rather than arbitrary IDs. By projecting label names into the same semantic space as the document text, the model performs few-shot NER by matching token representations directly to label representations, exploiting the linguistic knowledge already baked into the PLM.
Taking this conceptual shift further, Baotian Hu et al. developed Prompt-based Text Entailment (PTE), which reformulates entity detection as a "true or false" logic puzzle. By asking the model whether a text span entails a specific entity type, they tap into the deep linguistic reasoning present in PLMs, drastically reducing the need for manual training examples. For multimodal contexts, Chenran Cai et al. leverage the in-context learning capabilities of Large Language Models by converting images to captions and retrieving relevant labeled examples directly into the prompt.
Synthetic Generation and Pre-training
Finally, the annotation bottleneck can be bypassed entirely by generating synthetic data or using unsupervised pre-training tailored for sequences.
The authors eliminated data scarcity in citation parsing by using structured BibTeX files to automatically compile a massive dataset of 41 million labeled citation strings, achieving a 24.48% relative error reduction over human-labeled baselines. To synthesize data for complex sequence tasks like nested or discontinuous NER, Xuming Hu et al. introduced ENTDA, a framework that takes a list of target entities and uses a language model to generate semantically coherent sentences around them, flipping the traditional text-to-entity extraction paradigm.
On the pre-training front, Asli Celikyilmaz et al. used clustered word embeddings as "latent labels" to pre-train spoken language understanding models on unlabeled text, achieving high performance with a fraction of the standard training data. Crucially, Prajit Ramachandran et al. demonstrated that initializing the weights of sequence-to-sequence encoders and decoders with pretrained language models drastically improves downstream generalization, fundamentally altering how models handle data-scarce sequence mappings.
Key Foundational Work
One Class per Named Entity: Exploiting Unlabeled Text for Named Entity Recognition (2007) — contributed 1 concept elements to this direction.
Syntax-based Semi-Supervised Named Entity Tagging (2005) — contributed 1 concept elements to this direction.
Semi-Supervised Sequence Modeling with Syntactic Topic Models (2005) — contributed 1 concept elements to this direction.