TL;DR
Most text classification models treat labels as arbitrary numbers, ignoring the rich semantic information they carry. This paper introduces DELE, a framework that enriches label embeddings with multi-aspect definitions from WordNet and uses a mutual interaction module with contrastive learning to align text with the most relevant label descriptions.
What Problem Does This Paper Address?
In standard text classification, labels are typically treated as one-hot vectors—essentially meaningless indices. This approach assumes that labels are independent and carry no internal meaning. However, a label like "Contradiction" or "Business" has deep semantic roots that could guide a model's understanding of the input text.
Existing "label embedding" methods often use only the label name or a single, coarse description. The authors identify two main challenges:
- Semantic Sparsity: A single word (the label) isn't enough to capture the complex patterns it represents.
- Noise: Dictionary definitions (like those in WordNet) often contain multiple meanings. If a model uses the wrong definition for a specific context, it introduces noise that can degrade performance.
Methods
The authors propose two nested architectures: $R^2$-Net and its evolution, DELE.
1. $R^2$-Net (Relation of Relation Learning Network)
This preliminary model focuses on better label utilization through self-supervision. It introduces a Relation of Relation ($R^2$) classification task, where the model must predict if two different input sentences share the same label. It also employs a triplet loss to pull samples of the same class together and push different classes apart in the embedding space.
2. DELE (Description-Enhanced Label Embedding)
DELE extends $R^2$-Net by incorporating external knowledge:
- Label Encoder: For every label, the model retrieves up to three fine-grained noun descriptions from WordNet.
- Mutual Interaction Module: This is the "secret sauce." It uses a bidirectional attention mechanism:
- S2L (Sentence-to-Label) Attention: Uses the input text to select the most relevant parts of the label descriptions, effectively "denoising" the WordNet definitions.
- L2S (Label-to-Sentence) Attention: Uses label semantics to highlight the most important keywords in the input text.
- Contrastive Learning (CL): The model uses a projection head and a contrastive loss (NT-Xent) to maximize the agreement between the text representation and its corresponding "text-supervised" label representation.
Datasets
The models were evaluated across several diverse text classification tasks:
- Natural Language Inference (NLI): SNLI (Full and Hard sets), SICK, and SciTail.
- Paraphrase Identification (PI): Quora Question Pair (Quora) and MSRP.
- QA Topic Classification: Yahoo! Answers (10-class classification).
- Sentiment Analysis: SST-5 (5-class fine-grained sentiment).
Key Insights and Ideas
- Labels as Guidance, Not Just Targets: By treating labels as semantically rich objects, the model uses them to "probe" the input text for relevant features, rather than just trying to map text to a number at the very last layer.
- The Power of Denoising: The authors found that simply adding WordNet descriptions isn't enough; you need the Mutual Interaction Module to filter out irrelevant dictionary definitions. For example, the word "Business" can mean a "commercial enterprise" or an "immediate objective"—the model uses the input sentence to decide which one fits.
- Self-Supervision via $R^2$: The task of identifying whether two samples share a label forces the model to learn much more robust inter-class boundaries than standard cross-entropy alone.
Key Results
DELE consistently outperformed strong backbones like BERT and RoBERTa, as well as previous state-of-the-art label embedding methods:
- SNLI Hard Test: DELE-RoBERTa achieved 83.2% accuracy, a significant improvement over the RoBERTa-base backbone (81.5%).
- Error Rate Reduction: On the SciTail dataset, DELE improved the error rate by +11.29% compared to the backbone.
- Clustering Quality: T-SNE visualizations (Fig. 6 in the paper) show that DELE produces much tighter, more distinct clusters for different classes compared to standard BERT or other label embedding methods like EXAM.
Limitations and Future Work
- Manual Selection: Currently, the authors manually select the top 3 noun descriptions from WordNet. They acknowledge that a more automated, end-to-end knowledge selection process would be more scalable.
- Extreme Multi-label Classification (XMC): While DELE works well for tasks with 2–10 labels, the computational complexity of the mutual interaction module might become a bottleneck for tasks with thousands of labels.
- Hierarchical Knowledge: The authors suggest that future work could incorporate Graph Neural Networks (GNNs) to model the hierarchical relationships between labels (e.g., knowing that "Biology" is a sub-type of "Science") to further boost performance.
Summary updated