BatchMixup: Improving Training by Interpolating Hidden States of the Entire Mini-batch

Wenpeng Yin, Huan Wang, Jin Qu, Caiming Xiong

Abstract

Data augmentation is essential for training robust machine learning models, yet traditional techniques in Natural Language Processing (NLP) often lack efficiency or spatial coverage. While "Mixup" interpolates pairs of examples to create synthetic data, it is limited to linear paths between points and incurs significant computational overhead. This paper introduces BatchMixup, a novel approach that interpolates the hidden states of an entire mini-batch simultaneously. By moving from 1D to N-dimensional interpolation, BatchMixup generates synthetic data that better covers the representation space while training up to 13x faster than standard pair-wise Mixup variants.

Introduction

In modern NLP, data augmentation typically involves discrete transformations like synonym swapping or back-translation. Mixup offers a continuous alternative by creating convex combinations of input pairs and their labels. However, standard Mixup has two primary drawbacks: it only explores the thin lines connecting pairs of points, leaving vast "inner" regions of the representation space untouched, and it is computationally expensive, often requiring multiple synthetic pairs per original sample to be effective.

Methodology: From Edges to Volumes

BatchMixup addresses these limitations by performing interpolation across the entire mini-batch. Instead of selecting pairs, the method considers all points in a batch to create a richer signal for learning class boundaries.

The process involves:

  1. Hidden State Extraction: Inputs are passed through a shared encoder (e.g., RoBERTa) to obtain hidden representation vectors.
  2. Mixing Policy: A mixing matrix $\mathcal{M}$ is generated using values sampled from a Beta distribution.
  3. Softmax Interpolation: A softmax is applied across the batch dimension of $\mathcal{M}$ to ensure interpolation weights sum to one.
  4. Parallel Generation: The new synthetic batch is computed as a weighted sum of all original representations in the mini-batch via a single matrix operation.

This parallelized approach allows the model to generate $n$ new points simultaneously, significantly reducing training time compared to sequential pair-wise methods.

Experimental Results

The authors evaluated BatchMixup on three NLP tasks: Textual Entailment (GLUE RTE), Relation Classification (FewRel), and Intent Classification (BANKING77), across varying data scales from 1% to 100%.

Performance and Efficiency

BatchMixup consistently outperformed the base RoBERTa-large model and other Mixup variants. The performance gains were particularly notable in low-resource settings. For instance, on the FewRel task with only 1% of training data, BatchMixup achieved 60.29% accuracy, a massive improvement over the 3.41% achieved by the base model.

Performance comparison of BatchMixup against baselines across various NLP tasks and data availability levels.

Beyond accuracy, BatchMixup offers a substantial speed advantage. On the RTE task, it required only 1.5 minutes per epoch, compared to 20 minutes for standard Mixup to reach comparable augmentation levels.

Decision Boundary Regularization

Qualitative analysis through toy experiments demonstrates that BatchMixup acts as a powerful implicit regularizer. In few-shot scenarios, it helps the model learn a cleaner, more generalized decision boundary (hyperplane) compared to models trained without it, which often overfit to local noise.

Visualization of the clean circular decision boundary (hyperplane) learned using BatchMixup in a 5-shot classification task.

Conclusion and Limitations

BatchMixup provides a more efficient and spatially aware method for data augmentation in NLP. By interpolating across entire mini-batches, it improves both training speed and model performance, especially in data-scarce environments. However, the method faces a "cold start" problem: if the initial model performance is near random (as seen in the 1% RTE task), the underlying representations are too noisy for interpolation to provide meaningful benefits. Future research may explore its application to non-Transformer architectures and its interaction with other regularization techniques.

Summary updated

BibTeX
@inproceedings{yin2021batchmixup,
  author = {Wenpeng Yin and Huan Wang and Jin Qu and Caiming Xiong},
  title = {{BatchMixup: Improving Training by Interpolating Hidden States of the Entire Mini-batch}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  doi = {10.18653/v1/2021.findings-acl.434},
}
Figure