In traditional supervised learning, labels are treated as abstract, numerical indices—integers like 0, 1, or 2. This approach assumes that the relationship between a document and its category must be learned entirely from scratch through thousands of examples. However, in natural language tasks, labels often have inherent meaning. A label like "Toxic" or "Mortality" carries a semantic footprint that can guide a model's focus before it even sees a single training instance.
A growing body of research suggests that by mapping labels and words into a shared semantic space, we can treat classification not as a mapping of text to integers, but as a semantic alignment problem.
The Index Problem vs. Semantic Anchoring
Standard neural classifiers typically pass a document representation through a linear layer to produce a probability distribution over $N$ classes. In this setup, the model has no "intuition" about what the classes represent. If you swapped the names of the classes "Joy" and "Sadness," the model's architecture wouldn't notice the difference.
Researchers like Guoyin Wang and Chunyuan Li proposed the Label-Embedding Attentive Model (LEAM) to fix this. Instead of treating labels as indices, Wang et al. (2018) embed both words and labels into the same high-dimensional vector space. This allows the label embeddings to act as "anchor points." By calculating the compatibility between a word embedding and a label embedding, the model can determine which parts of a sentence are most relevant to a specific category.
Labels as Attention Queries
The core mechanism of this research direction is label-aware attention. In a standard attention mechanism, the model learns which words are important based on the internal patterns of the text. In label-aware attention, the labels themselves serve as the "queries."
As Michael Abaho and colleagues demonstrate in their LCAM framework, treating labels as semantic concepts allows the model to prioritize tokens that have strong semantic relationships with target categories (Abaho et al., 2021). For example, in clinical text, the embedding for the label "Mortality" would naturally "search" for and weight words like "deceased" or "expired" more heavily.
This "semantic search" capability is pushed even further by Malte Feucht and Volker Tresp in their Description-based Label Attention Classifier (DLAC). Instead of just using the label name, they use full textual descriptions of medical codes to query long clinical notes, grounding the model's focus in official medical definitions (Feucht et al., 2021).
Solving Label Shift and Data Scarcity
One of the most powerful applications of this idea is in low-resource and few-shot learning. When training data is scarce, a model cannot easily learn the nuances of a category from examples alone. By injecting label semantics, we provide the model with a "sense of purpose."
Rui Wang and Guoyin Wang showed that baking label information directly into the self-attention mechanism of models like BERT acts as a semantic prior, helping the model ignore noise and focus on task-relevant keywords (Wang et al., 2020). Similarly, Huy Nghiem and Fred Morstatter addressed the "label shift" problem in offensive speech detection—where different datasets define "toxic" differently—by providing the model with the actual dictionary definitions used by human annotators (Nghiem et al., 2024).
Hierarchical and Multi-Task Context
This approach isn't limited to simple classification. In Named Entity Recognition (NER), Ying Luo and Hai Zhao use label-embedding attention to capture context at both the sentence and document levels. By weighting words based on their similarity to specific entity types (like "Person" or "Organization"), the model creates "entity-aware" representations that are more precise than standard contextual embeddings (Luo et al., 2019).
By treating labels as rich semantic entities rather than abstract integers, these models bridge the gap between how humans understand categories—through definitions and concepts—and how machines process them. The result is a more interpretable, efficient, and robust approach to understanding text.