An Effective Deployment of Contrastive Learning in Multi-label Text Classification

Nankai Lin, Guanqiu Qin, Jigang Wang, Aimin Yang, Dong Zhou

TL;DR

Contrastive learning is highly effective for single-label classification, but defining positive and negative pairs gets messy when a text has multiple labels. This paper introduces five novel multi-label contrastive loss functions that intelligently reshape the semantic space, consistently boosting performance across diverse text classification tasks.

What Problem Does This Paper Address?

In multi-label text classification (MLTC), a single piece of text can be assigned multiple categories simultaneously—for instance, a tweet might express both "happy" and "surprise." This overlap creates a major challenge: it muddies the model's semantic representation space.

When samples share some labels but not others, the boundaries between different classes blur. A shared label can indirectly pull completely unrelated classes closer together in the embedding space. While contrastive learning (CL) is typically used to pull similar samples together and push distinct ones apart, applying it to MLTC is difficult because it is hard to define what truly constitutes a "positive" or "negative" pair when labels partially overlap.

Methods

The authors propose five plug-and-play contrastive loss functions designed to define positive and negative samples from different perspectives:

  • Strict Contrastive Loss (SCL): A conservative approach where two samples are considered a positive pair only if they share the exact same set of labels.
  • Jaccard Similarity Contrastive Loss (JSCL): A softer approach that scales the contrastive pull based on the Jaccard similarity (the degree of overlap) between the label sets of two samples.
  • Jaccard Similarity Probability Contrastive Loss (JSPCL): Similar to JSCL, but instead of operating on hidden semantic embeddings, it operates on the model's final output probability distributions.
  • Stepwise Label Contrastive Loss (SLCL): Deconstructs the multi-label problem by calculating a separate contrastive loss for each individual label independently, then combining them.
  • Intra-label Contrastive Loss (ICL): Focuses internally on a single sample, pulling the representations of its co-occurring labels closer together.

These contrastive losses are combined with standard cross-entropy loss to train the model jointly.

Datasets

The authors evaluated their methods on two distinct multi-label tasks:

  1. SemEval2018 Multi-label Emotion Classification (MEC): Evaluated across English, Arabic, and Spanish datasets. It contains 11 discrete emotion labels (e.g., anger, joy, sadness, trust).
  2. Indonesian Multi-label News Classification (MNC): An open-source dataset containing 8 labels (e.g., politics, economy, technology, military), where each article is associated with up to two categories.

Key Insights and Ideas

  • Strictness Often Wins: Counterintuitively, being strict (SCL) frequently outperforms being flexible (JSCL). Treating only exact label-set matches as positive pairs prevents the semantic space from becoming overly convoluted.
  • Features Over Probabilities: Feature-based contrastive learning (optimizing the hidden embedding space directly, like JSCL) proved superior to probability-based contrastive learning (optimizing output distributions, like JSPCL). The authors note that because multi-label semantic spaces are highly complex, targeting the embeddings directly is more effective.
  • Alleviating Class Imbalance: By enforcing clearer boundaries in the semantic space, these contrastive losses naturally help the model better distinguish minority classes that suffer from data imbalance.

Key Results

Using SpanEmo as their baseline SOTA architecture, the authors demonstrated that integrating their contrastive losses consistently improved performance across Micro $F_1$, Macro $F_1$, and Jaccard Index (JS) metrics.

Experimental results on English dataset
  • English MEC: JSCL achieved the top performance, pushing the Macro $F_1$ to 57.68 and JS to 59.05 (up from the baseline SpanEmo's 57.00 and 58.30).
  • Arabic & Spanish MEC: SCL and SLCL led the pack. On Arabic, SLCL hit a Macro $F_1$ of 54.88, while SCL achieved the highest Jaccard Index on Spanish (53.52).
  • Indonesian MNC: SCL dominated the news classification task, reaching a Macro $F_1$ of 74.29 and a JS of 84.06, noticeably outperforming standard BERT and SGM baselines.

Limitations and Future Work

The authors acknowledge a few limitations in their current work:

  1. Task Scope: The evaluation was restricted to emotion and news classification datasets.
  2. Modality: The framework is strictly single-modal (text-only) and has not yet been extended to multi-modal settings (such as image-text classification).
  3. Backbone Dependency: The experiments relied primarily on SpanEmo as the backbone model. Future work will explore validating these losses across a wider variety of multi-label model architectures.

Summary updated

Abstract

The effectiveness of contrastive learning technology in natural language processing tasks is yet to be explored and analyzed. How to construct positive and negative samples correctly and reasonably is the core challenge of contrastive learning. It is even harder to discover contrastive objects in multi-label text classification tasks. There are very few contrastive losses proposed previously. In this paper, we investigate the problem from a different angle by proposing five novel contrastive losses for multi-label text classification tasks. These are Strict Contrastive Loss (SCL), Intra-label Contrastive Loss (ICL), Jaccard Similarity Contrastive Loss (JSCL), Jaccard Similarity Probability Contrastive Loss (JSPCL), and Stepwise Label Contrastive Loss (SLCL). We explore the effectiveness of contrastive learning for multi-label text classification tasks by the employment of these novel losses and provide a set of baseline models for deploying contrastive learning techniques on specific tasks. We further perform an interpretable analysis of our approach to show how different components of contrastive learning losses play their roles. The experimental results show that our proposed contrastive losses can bring improvement to multi-label text classification tasks. Our work also explores how contrastive learning should be adapted for multi-label text classification tasks.
BibTeX
@inproceedings{lin2023effective,
  author = {Nankai Lin and Guanqiu Qin and Jigang Wang and Aimin Yang and Dong Zhou},
  title = {{An Effective Deployment of Contrastive Learning in Multi-label Text Classification}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.556},
}
Figure