Text Classification with Minimal Supervision

137 concepts130 papers1 proposal

Supervised machine learning models are notoriously data-hungry. To build a text classifier that distinguishes between "Sports" and "Finance," traditional pipelines require annotators to read and label thousands of documents. This creates a massive annotation bottleneck, particularly in specialized domains like medicine or law where expert time is prohibitively expensive.

But why do we need thousands of examples? Standard supervised models start completely blind; they must learn what "Sports" means purely by observing statistical correlations across a vast dataset. Humans, however, can classify documents instantly given only the category names. This realization has driven a rich line of research: text classification with minimal supervision. Instead of forcing humans to act as high-volume label engines, how can we inject minimal, high-leverage human knowledge to bootstrap powerful text classifiers?

Dataless Classification via Shared Semantic Spaces

The most radical approach to bypassing the annotation bottleneck is to eliminate document labeling entirely. This paradigm, known as dataless classification, relies on a simple intuition: both documents and category labels can be mapped into a shared semantic space where their similarity can be directly computed.

In pioneering work, Ming-Wei Chang et al. (Importance of Semantic Representation: Dataless Classification) demonstrated that Wikipedia could serve as a proxy for "world knowledge." By representing both the incoming text and the target category labels as vectors of Wikipedia concepts, a model can determine a document's class without a single training example. Dan Roth (On Dataless Hierarchical Text Classification) extended this to complex hierarchical taxonomies, proving that semantic understanding alone can rival supervised baselines.

To make this mapping precise, researchers have long leveraged structured lexical resources. For instance, Libby Barak et al. (Text Categorization from Category Name via Lexical Reference) combined WordNet and Wikipedia to identify explicit "lexical references"—words that unambiguously point to a category—to ground zero-shot classification. Similarly, Sejeong Kwon (Near-Zero-Shot Suggestion Mining with a Little Help from WordNet) utilized WordNet hyponyms alongside Natural Language Inference (NLI) to identify specific user intentions with minimal human intervention.

Shifting the Annotation Unit from Documents to Features

If some human intervention is available, where should it be spent? Traditional supervised learning asks humans to label instances (documents). However, labeling features (words) provides vastly higher leverage. If an expert labels the word "chemotherapy" as highly indicative of the "Oncology" class, that single action effectively influences thousands of documents containing that word.

This concept of feature-level feedback was formalized by Hema Raghavan et al. in their tandem learning framework (An interactive algorithm for asking and incorporating feature feedback into support vector machines), which interactively prompts users to label both documents and words. Bing Liu et al. (Text Classification by Labeling Words) showed that clustering and feature selection could allow users to build accurate classifiers simply by labeling a small set of representative keywords.

This "seed-guided" approach scales exceptionally well when paired with probabilistic models. Arjun Mukherjee et al. (Aspect Extraction through Semi-Supervised Modeling) used seed words to guide fine-grained aspect extraction in sentiment analysis, while Chenliang Li et al. (Seed-Guided Topic Model for Document Filtering and Classification) developed a topic model that filters and categorizes massive document collections using only a few keywords per category, explicitly modeling and discarding irrelevant background noise.

Bootstrapping and Iterative Self-Training Pipelines

While seed words and label names provide an excellent starting point, they lack the contextual nuance required to handle complex, real-world text. A keyword-matching system might easily misclassify a document about a "computer window" into an architecture category instead of software. To solve this, modern frameworks use minimal supervision to bootstrap a model, which then refines itself using a vast pool of unlabeled data.

A prominent example of this is WeSTClass, developed by Yu Meng et al. (Weakly-Supervised Neural Text Classification). WeSTClass generates synthetic "pseudo-documents" based on user-provided keywords to pre-train a neural network, which is subsequently refined via iterative self-training on real, unlabeled text.

The authors pushed this further with LOTClass (Text Classification Using Label Names Only: A Language Model Self-Training Approach), demonstrating that pre-trained language models like BERT can substitute for explicit keyword lists. By looking at the contextual words BERT predicts around a label name, LOTClass automatically extracts category-related words from unlabeled text and trains a classifier using only the names of the categories.

To handle the inherent noise of these automatically generated "pseudo-labels," recent architectures introduce structural and ensemble constraints:

The Role of Large Language Models and Synthetic Data

The advent of Large Language Models (LLMs) has fundamentally shifted the baseline for zero-shot and few-shot text classification. Instead of building complex semantic mapping pipelines, researchers can now use natural language prompts or codebooks to instruct an LLM to perform classification directly. For example, Ziang Xiao et al. (Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding) demonstrated that framing a qualitative research codebook as a prompt allows GPT-3 to achieve high agreement with human experts, bypassing the need for any training data.

However, deploying massive LLMs for high-volume production classification is often computationally and financially prohibitive. This has led to a "synthetic-data-first" paradigm, where LLMs are used as high-quality data generators to train smaller, efficient student models. Yet, this approach has its own boundaries. Zhuoyan Li et al. (Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations) systematically evaluated this pipeline, warning that while synthetic data can effectively train classifiers, it often suffers from distribution shifts and fails to capture the full complexity of real-world data.

To understand where these zero-shot methods stand today, Ilias Aarab introduced BTZSC (BTZSC: A Benchmark for Zero-Shot Text Classification Across Cross-Encoders, Embedding Models, and Rerankers), a rigorous benchmark evaluating genuine zero-shot text classification. The benchmark reveals that modern rerankers and cross-encoders have overtaken traditional NLI-based methods, providing a clear map for practitioners looking to break the annotation bottleneck without sacrificing accuracy or latency.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 132 papers
  2. Version 1 · 2026-10-04 · 114 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_annotation_bottleneck_labeled_data_scarc_03a2ca7e,
  title = {annotation bottleneck (labeled data scarcity)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/annotation-bottleneck-labeled-data-scarcity/rdr_03a2ca7e47974c2ab30309b62fbc1ef4},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 2, frozen text):

@misc{lacuna_annotation_bottleneck_labeled_data_scarc_03a2ca7e_v2,
  title = {annotation bottleneck (labeled data scarcity)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/annotation-bottleneck-labeled-data-scarcity/rdr_03a2ca7e47974c2ab30309b62fbc1ef4/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake