Dictionary-Assisted Supervised Contrastive Learning

Patrick Y. Wu, Richard Bonneau, Joshua A. Tucker, Jonathan NaglerEMNLP 2022 · DOI

Abstract

Dictionary-Assisted Supervised Contrastive Learning (DASCL) is a novel framework designed to enhance text classification in data-scarce scenarios by integrating domain-specific dictionaries with modern transformer models. By employing a keyword simplification preprocessing step and a supervised contrastive loss, DASCL bridges the gap between expert-crafted lexicons and deep learning. Experimental results across sentiment analysis and abusive language detection show significant performance gains in few-shot settings, often outperforming specialized pre-trained models.

Introduction

In the social sciences, researchers frequently encounter "small corpora"—datasets where the number of labeled examples is insufficient for deep learning models like BERT or RoBERTa to generalize effectively. Historically, these fields relied on expert-crafted dictionaries to identify concepts like "economic anxiety" or "sentiment." While these dictionaries capture valuable domain expertise, they are often overlooked in modern supervised machine learning workflows.

The primary difficulty lies in lexical diversity: two documents may share the same label (e.g., "positive sentiment") but use entirely different vocabularies (e.g., "spectacular" vs. "breathtaking"). In low-data regimes, models struggle to recognize these as semantically equivalent. DASCL proposes to leverage these "old-school" dictionaries to guide "new-school" language models toward more robust conceptual representations.

Methods

The DASCL framework operates in two distinct stages:

1. Keyword Simplification

This preprocessing step uses a task-relevant dictionary to reduce lexical noise. Any word in the input text found in the dictionary is replaced by a fixed, common token (e.g., all positive adjectives become <positive>). This artificially increases the textual similarity between documents belonging to the same class, emphasizing the core signal.

2. Supervised Contrastive Learning (SCL)

During the fine-tuning phase, the model processes both the original text and its keyword-simplified counterpart. The DASCL objective treats these as a "positive pair." The loss function is designed to pull together the embeddings of original and simplified texts within the same class while pushing apart embeddings from different classes.

The DASCL architecture utilizes a dual-stream approach with RoBERTa, applying a contrastive loss between the original and keyword-simplified text representations.

The final training objective combines this contrastive loss with standard cross-entropy. Crucially, at inference time, the dictionary is no longer required; the model classifies the original, un-simplified text directly.

Experimental Evaluation

The authors evaluated DASCL across three domains: SST-2 (Sentiment), NYT Economic Articles (Economic Tone), and AbusEval (Abusive Language). Experiments specifically focused on few-shot settings, testing performance with training sets as small as N=20, 100, or 1,000 examples.

Key Results

DASCL demonstrates its greatest utility in extremely data-constrained environments. On the SST-2 dataset with only 20 training examples, DASCL achieved an accuracy of 77.7%, representing a 10.2 percentage point improvement over standard cross-entropy training.

Experimental results on the SST-2 dataset show that DASCL (CE+DASCL) significantly outperforms standard cross-entropy (CE) and other baselines in few-shot settings.

Furthermore, in the AbusEval task, a general BERT model augmented with DASCL performed comparably to or better than HateBERT—a model specifically pre-trained on massive amounts of abusive language data. This suggests that DASCL can effectively inject domain knowledge without the need for expensive specialized pre-training.

Discussion and Conclusion

By treating dictionaries as domain priors rather than rigid classification tools, DASCL reduces lexical noise and forces the model to learn why documents are similar. Visualizations of the embedding space via t-SNE confirm that DASCL creates tighter, more distinct clusters compared to the overlapping "blobs" often produced by standard training.

While the current study focuses on binary classification and is sensitive to dictionary quality, it offers a computationally efficient path for researchers to leverage expert knowledge. Future work may extend this to multiclass settings or semi-supervised learning, further exploring how keyword simplification can help models discover latent concepts in unlabeled text.

Summary updated

BibTeX
@inproceedings{wu2022dictionary,
  author = {Patrick Y. Wu and Richard Bonneau and Joshua A. Tucker and Jonathan Nagler},
  title = {{Dictionary-Assisted Supervised Contrastive Learning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.emnlp-main.696},
}
Figure