TL;DR
Contrastive learning is highly effective for single-label classification, but defining positive and negative pairs gets messy when a text has multiple labels. This paper introduces five novel multi-label contrastive loss functions that intelligently reshape the semantic space, consistently boosting performance across diverse text classification tasks.
What Problem Does This Paper Address?
In multi-label text classification (MLTC), a single piece of text can be assigned multiple categories simultaneously—for instance, a tweet might express both "happy" and "surprise." This overlap creates a major challenge: it muddies the model's semantic representation space.
When samples share some labels but not others, the boundaries between different classes blur. A shared label can indirectly pull completely unrelated classes closer together in the embedding space. While contrastive learning (CL) is typically used to pull similar samples together and push distinct ones apart, applying it to MLTC is difficult because it is hard to define what truly constitutes a "positive" or "negative" pair when labels partially overlap.
Methods
The authors propose five plug-and-play contrastive loss functions designed to define positive and negative samples from different perspectives:
- Strict Contrastive Loss (SCL): A conservative approach where two samples are considered a positive pair only if they share the exact same set of labels.
- Jaccard Similarity Contrastive Loss (JSCL): A softer approach that scales the contrastive pull based on the Jaccard similarity (the degree of overlap) between the label sets of two samples.
- Jaccard Similarity Probability Contrastive Loss (JSPCL): Similar to JSCL, but instead of operating on hidden semantic embeddings, it operates on the model's final output probability distributions.
- Stepwise Label Contrastive Loss (SLCL): Deconstructs the multi-label problem by calculating a separate contrastive loss for each individual label independently, then combining them.
- Intra-label Contrastive Loss (ICL): Focuses internally on a single sample, pulling the representations of its co-occurring labels closer together.
These contrastive losses are combined with standard cross-entropy loss to train the model jointly.
Datasets
The authors evaluated their methods on two distinct multi-label tasks:
- SemEval2018 Multi-label Emotion Classification (MEC): Evaluated across English, Arabic, and Spanish datasets. It contains 11 discrete emotion labels (e.g., anger, joy, sadness, trust).
- Indonesian Multi-label News Classification (MNC): An open-source dataset containing 8 labels (e.g., politics, economy, technology, military), where each article is associated with up to two categories.
Key Insights and Ideas
- Strictness Often Wins: Counterintuitively, being strict (SCL) frequently outperforms being flexible (JSCL). Treating only exact label-set matches as positive pairs prevents the semantic space from becoming overly convoluted.
- Features Over Probabilities: Feature-based contrastive learning (optimizing the hidden embedding space directly, like JSCL) proved superior to probability-based contrastive learning (optimizing output distributions, like JSPCL). The authors note that because multi-label semantic spaces are highly complex, targeting the embeddings directly is more effective.
- Alleviating Class Imbalance: By enforcing clearer boundaries in the semantic space, these contrastive losses naturally help the model better distinguish minority classes that suffer from data imbalance.
Key Results
Using SpanEmo as their baseline SOTA architecture, the authors demonstrated that integrating their contrastive losses consistently improved performance across Micro $F_1$, Macro $F_1$, and Jaccard Index (JS) metrics.
- English MEC: JSCL achieved the top performance, pushing the Macro $F_1$ to 57.68 and JS to 59.05 (up from the baseline SpanEmo's 57.00 and 58.30).
- Arabic & Spanish MEC: SCL and SLCL led the pack. On Arabic, SLCL hit a Macro $F_1$ of 54.88, while SCL achieved the highest Jaccard Index on Spanish (53.52).
- Indonesian MNC: SCL dominated the news classification task, reaching a Macro $F_1$ of 74.29 and a JS of 84.06, noticeably outperforming standard BERT and SGM baselines.
Limitations and Future Work
The authors acknowledge a few limitations in their current work:
- Task Scope: The evaluation was restricted to emotion and news classification datasets.
- Modality: The framework is strictly single-modal (text-only) and has not yet been extended to multi-modal settings (such as image-text classification).
- Backbone Dependency: The experiments relied primarily on SpanEmo as the backbone model. Future work will explore validating these losses across a wider variety of multi-label model architectures.
Summary updated