Description-Enhanced Label Embedding Contrastive Learning for Text Classification

Kun Zhang, Le Wu, Guangyi Lv, Enhong Chen, Shulan Ruan, Jing Liu, Zhiqiang Zhang, Jun Zhou, Meng Wang

Abstract

Standard text classification models typically treat labels as discrete, independent indices, ignoring the inherent semantic information they contain. This paper proposes the Description-Enhanced Label Embedding (DELE) framework, which enriches label representations using multi-aspect definitions from WordNet. By employing a mutual interaction module and contrastive learning, DELE aligns input text with the most relevant label descriptions, effectively denoising dictionary definitions and capturing complex semantic patterns.

The Semantic Gap in Text Classification

In standard text classification, labels are typically treated as one-hot vectors—essentially meaningless indices. This approach assumes that labels are independent and carry no internal meaning. However, a label like "Contradiction" or "Business" has deep semantic roots that could guide a model's understanding of the input text.

Existing "label embedding" methods often use only the label name or a single, coarse description. The authors identify two main challenges:

  1. Semantic Sparsity: A single word (the label) isn't enough to capture the complex patterns it represents.
  2. Noise: Dictionary definitions (like those in WordNet) often contain multiple meanings. If a model uses the wrong definition for a specific context, it introduces noise that can degrade performance.
Figure 1: Examples of label ambiguity and context-dependent meanings for the label 'Business' across different datasets.

The DELE Framework

The authors propose two nested architectures: $R^2$-Net and its evolution, DELE.

1. $R^2$-Net (Relation of Relation Learning Network)

This preliminary model focuses on better label utilization through self-supervision. It introduces a Relation of Relation ($R^2$) classification task, where the model must predict if two different input sentences share the same label. It also employs a triplet loss to pull samples of the same class together and push different classes apart in the embedding space.

2. DELE (Description-Enhanced Label Embedding)

DELE extends $R^2$-Net by incorporating external knowledge:

  • Label Encoder: For every label, the model retrieves up to three fine-grained noun descriptions from WordNet.
  • Mutual Interaction Module: This uses a bidirectional attention mechanism:
  • S2L (Sentence-to-Label) Attention: Uses the input text to select the most relevant parts of the label descriptions, effectively "denoising" the WordNet definitions.
  • L2S (Label-to-Sentence) Attention: Uses label semantics to highlight the most important keywords in the input text.
  • Contrastive Learning (CL): The model uses a projection head and a contrastive loss (NT-Xent) to maximize the agreement between the text representation and its corresponding "text-supervised" label representation.
Figure 2: The architecture of $R^2$-Net and its detailed encoding and identification components.

Experimental Evaluation and Insights

The models were evaluated across several diverse text classification tasks, including Natural Language Inference (SNLI, SICK, SciTail), Paraphrase Identification (Quora, MSRP), QA Topic Classification (Yahoo! Answers), and Sentiment Analysis (SST-5).

Key findings include:

  • Labels as Guidance: By treating labels as semantically rich objects, the model uses them to "probe" the input text for relevant features.
  • The Power of Denoising: The Mutual Interaction Module is crucial for filtering out irrelevant dictionary definitions. For example, the word "Business" can mean a "commercial enterprise" or an "immediate objective"—the model uses the input sentence to decide which one fits.
  • Self-Supervision via $R^2$: Identifying whether two samples share a label forces the model to learn much more robust inter-class boundaries than standard cross-entropy alone.

DELE consistently outperformed strong backbones like BERT and RoBERTa. On the SNLI Hard Test, DELE-RoBERTa achieved 83.2% accuracy, compared to the 81.5% of the RoBERTa-base backbone. On the SciTail dataset, DELE improved the error rate by +11.29% compared to the backbone.

Conclusion and Future Directions

While DELE shows significant promise, there are areas for improvement. Currently, the top 3 noun descriptions from WordNet are manually selected; an automated selection process would improve scalability. Additionally, the computational complexity of the mutual interaction module may be a bottleneck for Extreme Multi-label Classification (XMC). Future work could incorporate Graph Neural Networks (GNNs) to model hierarchical relationships between labels.

Summary updated

BibTeX
@article{zhang2024description,
  author = {Kun Zhang and Le Wu and Guangyi Lv and Enhong Chen and Shulan Ruan and Jing Liu and Zhiqiang Zhang and Jun Zhou and Meng Wang},
  title = {{Description-Enhanced Label Embedding Contrastive Learning for Text Classification}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2024},
  url = {https://openreview.net/forum?id=RsNkzNJVLd},
}
Figure