Efficient Contrastive Learning via Novel Data Augmentation and Curriculum Learning

Seonghyeon Ye, Jiseon Kim, Alice Oh

Abstract

We introduce EfficientCL, a memory-efficient continual pretraining method that applies contrastive learning with novel data augmentation and curriculum learning. For data augmentation, we stack two types of operation sequentially: cutoff and PCA jittering. While pretraining steps proceed, we apply curriculum learning by incrementing the augmentation degree for each difficulty step. After data augmentation is finished, contrastive learning is applied on projected embeddings of original and augmented examples. When finetuned on GLUE benchmark, our model outperforms baseline models, especially for sentence-level tasks. Additionally, this improvement is capable with only 70% of computational memory compared to the baseline model.

Introduction

Training state-of-the-art language models from scratch is prohibitively expensive. Continual pretraining—taking an existing model and training it further on more data—is a popular alternative. However, it faces two major hurdles: the difficulty of creating meaningful "positive pairs" for contrastive learning in NLP without altering semantic meaning, and the high resource intensity of existing methods that require sampling multiple long text spans.

EfficientCL addresses these challenges by combining contrastive learning with a novel "hidden-state" data augmentation stack and a curriculum that gradually increases task difficulty. This approach allows the model to outperform standard RoBERTa and DeCLUTR baselines on the GLUE benchmark while requiring 30% less computational memory.

Methodology

The EfficientCL framework utilizes a three-pronged strategy to enhance pretraining efficiency and performance:

Hidden-State Augmentation

Instead of modifying raw input text, EfficientCL applies two sequential operations to the hidden states (latent vector representations) of the model:

  1. Cutoff: Randomly selecting a continuous span of the embedding and setting it to zero.
  2. PCA Jittering: Adding noise to the hidden states along the directions of greatest variance (eigenvectors), inspired by computer vision techniques.

Simple Curriculum Learning

The model implements a curriculum where the "noise level" ($\gamma$), representing the degree of cutoff and jittering, starts low (0.01) and increases to a maximum (0.1) as training progresses. A discrete 10-step increase was found to be more effective than a continuous one.

Contrastive Framework

The model takes an "anchor" (a 512-token sequence) and creates a "positive" version via the augmentation stack. A contrastive loss function pulls the anchor and its augmented version closer in vector space while pushing other sequences in the batch away. Standard Masked Language Modeling (MLM) loss is retained to preserve basic linguistic patterns.

Overview of the EfficientCL architecture, illustrating the augmentation stack and the curriculum-based contrastive learning process.

Experimental Results

The model was continually pretrained on OpenWebText (approx. 495k documents) and evaluated on the GLUE benchmark.

Performance and Efficiency

EfficientCL achieved an average GLUE score of 85.34, outperforming RoBERTa-base (85.09) and the previous state-of-the-art contrastive model, DeCLUTR (84.93). Notably, these results were achieved using only 70% of the memory required by DeCLUTR. Because EfficientCL only requires 512 tokens to create a training pair (compared to 2048 for DeCLUTR), it can utilize a much larger portion of the OpenWebText corpus—roughly 8 times more documents are eligible for training.

Comparison of average accuracy on the GLUE dev set and memory consumption between DeCLUTR and EfficientCL.

Robustness vs. Sensitivity

The combination of Cutoff and PCA Jittering makes the model significantly more robust to sentence-level noise. However, this increased robustness leads to a slight performance dip in tasks like CoLA (linguistic acceptability), as the model becomes more forgiving of the small grammatical errors that CoLA is designed to detect.

Conclusion

EfficientCL demonstrates that hidden-state augmentation combined with curriculum learning can significantly improve the efficiency and effectiveness of continual pretraining. Future work will explore scaling these techniques to larger architectures and maintaining sensitivity to fine-grained syntax while building sentence-level robustness.

Summary updated

Abstract

We introduce EfficientCL, a memory-efficient continual pretraining method that applies contrastive learning with novel data augmentation and curriculum learning. For data augmentation, we stack two types of operation sequentially: cutoff and PCA jittering. While pretraining steps proceed, we apply curriculum learning by incrementing the augmentation degree for each difficulty step. After data augmentation is finished, contrastive learning is applied on projected embeddings of original and augmented examples. When finetuned on GLUE benchmark, our model outperforms baseline models, especially for sentence-level tasks. Additionally, this improvement is capable with only 70% of computational memory compared to the baseline model.
BibTeX
@inproceedings{ye2021efficient,
  author = {Seonghyeon Ye and Jiseon Kim and Alice Oh},
  title = {{Efficient Contrastive Learning via Novel Data Augmentation and Curriculum Learning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=n0hvEWK26B},
}
Figure