Efficient Contrastive Learning via Novel Data Augmentation and Curriculum Learning

Seonghyeon Ye, Jiseon Kim, Alice Oh

Abstract

EfficientCL is a memory-efficient method for the continual pretraining of language models. By combining a novel data augmentation strategy—Cutoff and PCA jittering—with a curriculum learning schedule that gradually increases augmentation noise, the proposed method achieves state-of-the-art performance on the GLUE benchmark. Notably, EfficientCL utilizes 30% less computational memory than previous contrastive learning baselines, making it a more accessible and "green" alternative for language model optimization.

Introduction

Continual pretraining—further training an existing model like RoBERTa on new data—is a cost-effective alternative to training massive language models from scratch. Contrastive Learning (CL) has emerged as a powerful technique in this domain, teaching models to align embeddings of similar text spans (positive pairs) while distancing dissimilar ones (negative pairs). However, CL in Natural Language Processing (NLP) faces significant challenges: the difficulty of translating visual augmentations to discrete text and the high resource intensity required to sample multiple long text spans, which often limits document length and consumes excessive memory.

Methodology

EfficientCL addresses these hurdles through three primary innovations: stacked data augmentation, a curriculum learning schedule, and a hybrid objective function.

Stacked Data Augmentation

Instead of sampling two distinct text spans to create a positive pair, EfficientCL uses a single "anchor" span. It generates a positive counterpart by applying two operations directly to the hidden states:

  1. Cutoff: Randomly zeroing out continuous portions of the embedding sequence, analogous to image cropping.
  2. PCA Jittering: Inspired by color jittering in computer vision, this adds noise to hidden states based on the data's principal components (eigenvectors and eigenvalues). This encourages the model to be robust against "shallow" variations in sentence representation.
Overview of the EfficientCL framework featuring stacked data augmentation and curriculum learning.

Curriculum Learning and Hybrid Objective

The difficulty of the contrastive task is modulated by a noise level parameter, $\gamma$. Training follows an "easy-to-hard" curriculum, starting with light augmentation ($\gamma = 0.01$) and scaling to harder examples ($\gamma = 0.1$). The authors found that starting with high noise immediately leads to poor convergence. To maintain token-level knowledge and prevent catastrophic forgetting, the model is trained with a joint loss function combining standard Masked Language Modeling (MLM) and the new contrastive loss.

Experimental Setup

The model was continually pretrained on OpenWebText, a corpus comprising approximately 495,000 web documents. Evaluation was conducted using the GLUE (General Language Understanding Evaluation) benchmark, covering tasks such as sentiment analysis (SST-2), linguistic acceptability (CoLA), and natural language inference (MNLI, RTE).

Results

EfficientCL achieved an average score of 85.34 on the GLUE development set, surpassing the original RoBERTa-base (85.09) and the previous state-of-the-art contrastive baseline, DeCLUTR (84.93). The model demonstrated particular strength in sentence-level tasks, with gains of +0.61 on MRPC and +0.72 on RTE over the RoBERTa baseline.

Comparison of average accuracy and memory usage between DeCLUTR and EfficientCL.

Crucially, EfficientCL is significantly more memory-efficient. By generating positive pairs through mathematical augmentation of hidden states rather than additional text sampling, it requires only 70% of the memory used by DeCLUTR. This efficiency allows the model to process a wider range of document lengths without the need for extremely long source texts.

Discussion and Limitations

While EfficientCL excels in most areas, it showed a slight performance dip on the CoLA task. This is likely because CoLA requires sensitivity to subtle grammatical errors, which the model's learned "noise-robustness" may inadvertently overlook.

Future work includes exploring "hard negative mining" by sampling multiple anchors per document—though this would trade off some memory efficiency—and testing the framework on larger architectures like RoBERTa-large or linear-attention Transformers.

Summary updated

Abstract

We introduce EfficientCL, a memory-efficient continual pretraining method that applies contrastive learning with novel data augmentation and curriculum learning. For data augmentation, we stack two types of operation sequentially: cutoff and PCA jittering. While pretraining steps proceed, we apply curriculum learning by incrementing the augmentation degree for each difficulty step. After data augmentation is finished, contrastive learning is applied on projected embeddings of original and augmented examples. When finetuned on GLUE benchmark, our model outperforms baseline models, especially for sentence-level tasks. Additionally, this improvement is capable with only 70% of computational memory compared to the baseline model.
BibTeX
@inproceedings{ye2021efficient,
  author = {Seonghyeon Ye and Jiseon Kim and Alice Oh},
  title = {{Efficient Contrastive Learning via Novel Data Augmentation and Curriculum Learning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=n0hvEWK26B},
}
Figure