Description-Enhanced Label Embedding Contrastive Learning for Text Classification

Kun Zhang, Le Wu, Guangyi Lv, Enhong Chen, Shulan Ruan, Jing Liu, Zhiqiang Zhang, Jun Zhou, Meng Wang

TL;DR

Most text classification models treat labels as arbitrary numbers, ignoring the rich semantic information they carry. This paper introduces DELE, a framework that enriches label embeddings with multi-aspect definitions from WordNet and uses a mutual interaction module with contrastive learning to align text with the most relevant label descriptions.

What Problem Does This Paper Address?

In standard text classification, labels are typically treated as one-hot vectors—essentially meaningless indices. This approach assumes that labels are independent and carry no internal meaning. However, a label like "Contradiction" or "Business" has deep semantic roots that could guide a model's understanding of the input text.

Existing "label embedding" methods often use only the label name or a single, coarse description. The authors identify two main challenges:

  1. Semantic Sparsity: A single word (the label) isn't enough to capture the complex patterns it represents.
  2. Noise: Dictionary definitions (like those in WordNet) often contain multiple meanings. If a model uses the wrong definition for a specific context, it introduces noise that can degrade performance.

Methods

The authors propose two nested architectures: $R^2$-Net and its evolution, DELE.

1. $R^2$-Net (Relation of Relation Learning Network)

This preliminary model focuses on better label utilization through self-supervision. It introduces a Relation of Relation ($R^2$) classification task, where the model must predict if two different input sentences share the same label. It also employs a triplet loss to pull samples of the same class together and push different classes apart in the embedding space.

2. DELE (Description-Enhanced Label Embedding)

DELE extends $R^2$-Net by incorporating external knowledge:

  • Label Encoder: For every label, the model retrieves up to three fine-grained noun descriptions from WordNet.
  • Mutual Interaction Module: This is the "secret sauce." It uses a bidirectional attention mechanism:
  • S2L (Sentence-to-Label) Attention: Uses the input text to select the most relevant parts of the label descriptions, effectively "denoising" the WordNet definitions.
  • L2S (Label-to-Sentence) Attention: Uses label semantics to highlight the most important keywords in the input text.
  • Contrastive Learning (CL): The model uses a projection head and a contrastive loss (NT-Xent) to maximize the agreement between the text representation and its corresponding "text-supervised" label representation.

Datasets

The models were evaluated across several diverse text classification tasks:

  • Natural Language Inference (NLI): SNLI (Full and Hard sets), SICK, and SciTail.
  • Paraphrase Identification (PI): Quora Question Pair (Quora) and MSRP.
  • QA Topic Classification: Yahoo! Answers (10-class classification).
  • Sentiment Analysis: SST-5 (5-class fine-grained sentiment).

Key Insights and Ideas

  • Labels as Guidance, Not Just Targets: By treating labels as semantically rich objects, the model uses them to "probe" the input text for relevant features, rather than just trying to map text to a number at the very last layer.
  • The Power of Denoising: The authors found that simply adding WordNet descriptions isn't enough; you need the Mutual Interaction Module to filter out irrelevant dictionary definitions. For example, the word "Business" can mean a "commercial enterprise" or an "immediate objective"—the model uses the input sentence to decide which one fits.
  • Self-Supervision via $R^2$: The task of identifying whether two samples share a label forces the model to learn much more robust inter-class boundaries than standard cross-entropy alone.

Key Results

DELE consistently outperformed strong backbones like BERT and RoBERTa, as well as previous state-of-the-art label embedding methods:

  • SNLI Hard Test: DELE-RoBERTa achieved 83.2% accuracy, a significant improvement over the RoBERTa-base backbone (81.5%).
  • Error Rate Reduction: On the SciTail dataset, DELE improved the error rate by +11.29% compared to the backbone.
  • Clustering Quality: T-SNE visualizations (Fig. 6 in the paper) show that DELE produces much tighter, more distinct clusters for different classes compared to standard BERT or other label embedding methods like EXAM.

Limitations and Future Work

  • Manual Selection: Currently, the authors manually select the top 3 noun descriptions from WordNet. They acknowledge that a more automated, end-to-end knowledge selection process would be more scalable.
  • Extreme Multi-label Classification (XMC): While DELE works well for tasks with 2–10 labels, the computational complexity of the mutual interaction module might become a bottleneck for tasks with thousands of labels.
  • Hierarchical Knowledge: The authors suggest that future work could incorporate Graph Neural Networks (GNNs) to model the hierarchical relationships between labels (e.g., knowing that "Biology" is a sub-type of "Science") to further boost performance.

Summary updated

Abstract

Text classification is one of the fundamental tasks in natural language processing, which requires an agent to determine the most appropriate category for input sentences. Recently, deep neural networks have achieved impressive performance in this area, especially pretrained language models (PLMs). Usually, these methods concentrate on input sentences and corresponding semantic embedding generation. However, for another essential component: labels, most existing works either treat them as meaningless one-hot vectors or use vanilla embedding methods to learn label representations along with model training, underestimating the semantic information and guidance that these labels reveal. To alleviate this problem and better exploit label information, in this article, we employ self-supervised learning (SSL) in model learning process and design a novel self-supervised relation of relation ( $\text{R}^{2}$ ) classification task for label utilization from a one-hot manner perspective. Then, we propose a novel relation of relation learning network( $\text{R}^{2}$ -Net) for text classification, in which text classification and $\text{R}^{2}$ classification are treated as optimization targets. Meanwhile, triplet loss is employed to enhance the analysis of differences and connections among labels. Moreover, considering that one-hot usage is still short of exploiting label information, we incorporate external knowledge from WordNet to obtain multiaspect descriptions for label semantic learning and extend $\text{R}^{2}$ -Net to a novel description-enhanced label embedding network(DELE) from a label embedding perspective. One step further, since these fine-grained descriptions may introduce unexpected noise, we develop a mutual interaction module to select appropriate parts from input sentences and labels simultaneously based on contrastive learning (CL) for noise mitigation. Extensive experiments on different text classification tasks reveal that $\text{R}^{2}$ -Net can effectively improve the classification performance and DELE can make better use of label information and further improve the performance. As a byproduct, we have released the codes to facilitate other research.
BibTeX
@article{zhang2024description,
  author = {Kun Zhang and Le Wu and Guangyi Lv and Enhong Chen and Shulan Ruan and Jing Liu and Zhiqiang Zhang and Jun Zhou and Meng Wang},
  title = {{Description-Enhanced Label Embedding Contrastive Learning for Text Classification}},
  journal = {IEEE Trans. Neural Networks Learn. Syst.},
  year = {2024},
  url = {https://openreview.net/forum?id=RsNkzNJVLd},
}
Figure