CIAug: Equipping Interpolative Augmentation with Curriculum Learning

Ramit Sawhney, Ritesh Soun, Shrey Pandit, Megh Thakkar, Sarvagya Malaviya, Yuval PinterNAACL 2022 · OpenReview

Abstract

Ramit Sawhney, Ritesh Soun, Shrey Pandit, Megh Thakkar, Sarvagya Malaviya, Yuval Pinter. Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.

2022.

Introduction

Data augmentation is a cornerstone of modern NLP, especially when training data is scarce. A popular technique is Mixup, which creates synthetic training examples by interpolating (blending) two random samples and their labels. While effective, standard Mixup has two major flaws: it picks samples to mix at random, which can introduce noise early in training, and it typically relies on Euclidean distance, which fails to capture the hierarchical nature of linguistic data.

CIAug (Curriculum Interpolative Augmentation) addresses these issues by replacing random selection with a structured curriculum. By gradually mixing increasingly diverse samples based on their distance in hyperbolic space, the method achieves state-of-the-art results and converges significantly faster than standard approaches.

Methods

CIAug introduces a novel selection strategy for interpolative augmentation based on two key pillars:

  • Curriculum Learning: Instead of random mixing, CIAug starts by mixing samples that are "close" to each other (easy). As training progresses, it gradually increases the "complexity" by allowing the model to mix samples that are further apart (hard/diverse). This is managed by a dynamic threshold that updates every epoch.
  • Hyperbolic Geometry: To determine which samples are "close," the authors map sentence embeddings into hyperbolic space using the Poincaré ball model. This allows the model to better understand the underlying hierarchy and semantic similarity of text compared to flat Euclidean spaces.
Comparison of sample selection in Mixup (random) vs. CIAug (curriculum-based) within the hyperbolic Poincaré ball.

Additionally, the system maintains a Learnable Distance Matrix. While initialized with hyperbolic distances, this matrix updates alongside the model to better capture sample-specific relationships as the training progresses.

Datasets and Evaluation

The researchers evaluated CIAug on a diverse set of 10 benchmark datasets spanning four languages: English (SST-2, TREC, CoLA, RTE, MRPC, CoNLL-2003), Arabic (AHS), Turkish (TTC), and German (CoNLL-2003). This multi-lingual approach demonstrates that the benefits of hyperbolic curriculum learning are not limited to English-centric syntax.

Key Results

CIAug consistently outperformed standard Mixup and other advanced baselines (like SMix and HMix) across all 10 datasets. For example, on the CoLA dataset, it achieved an F1 score of 95.32 compared to the baseline's 84.91.

Performance comparison of CIAug against baselines across multiple NLP benchmarks, showing consistent state-of-the-art results.

Beyond raw performance, the method demonstrated two critical advantages:

  1. Efficiency: CIAug reached benchmark F1 scores up to three times faster than traditional Mixup, significantly reducing the computational cost of training.
  2. Robustness: The model proved more resilient against adversarial "synonym substitution" attacks, maintaining higher accuracy than its competitors when faced with perturbed input.

Conclusion and Future Work

CIAug demonstrates that the geometry of embedding spaces should match the nature of the data, and that "staggering" the difficulty of synthetic samples leads to more stable and efficient learning. The primary limitation is the computational overhead of maintaining an $N \times N$ distance matrix. Future work will explore the use of sparse matrices to scale this technique to massive datasets with millions of examples, making sophisticated augmentation viable for large-scale production applications.

Summary updated

BibTeX
@inproceedings{sawhney2022ciaug,
  author = {Ramit Sawhney and Ritesh Soun and Shrey Pandit and Megh Thakkar and Sarvagya Malaviya and Yuval Pinter},
  title = {{CIAug: Equipping Interpolative Augmentation with Curriculum Learning}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=gpGZSKEdKie},
}
Figure