HiddenCut: Simple Data Augmentation for Natural Language Understanding with Better Generalization

Jiaao Chen, Dinghan Shen, Weizhu Chen, Diyi Yang

Abstract

Fine-tuning large pre-trained models with task-specific data has achieved great success in NLP. However, it has been demonstrated that the majority of information within the self-attention networks is redundant and not utilized effectively during the fine-tuning stage. This leads to inferior results when generalizing the obtained models to out-of-domain distributions. To this end, we propose a simple yet effective data augmentation technique, HiddenCut, to better regularize the model and encourage it to learn more generalizable features. Specifically, contiguous spans within the hidden space are dynamically and strategically dropped during training. Experiments show that our HiddenCut method outperforms the state-of-the-art augmentation methods on the GLUE benchmark, and consistently exhibits superior generalization performances on out-of-distribution and challenging counterexamples.

Introduction

Pre-trained Language Models (PLMs) like RoBERTa have revolutionized NLP, yet they often struggle with a "generalization gap" when fine-tuned on relatively small task-specific datasets. These powerful models frequently rely on spurious correlations—statistical artifacts in training data that do not reflect the true underlying logic of a task. While standard regularization like Dropout attempts to mitigate this by randomly zeroing out neurons, it is often insufficient for modern Transformers. Due to the high redundancy in self-attention networks, models can easily reconstruct missing random neurons from their neighbors, failing to learn deeper, more robust linguistic features.

Methodology: The HiddenCut Approach

HiddenCut addresses these limitations by shifting the "cutting" of information from the input space to the model's internal hidden representations. Unlike standard dropout, which acts on isolated units, HiddenCut masks contiguous blocks of hidden representations corresponding to token sequences.

Comparison of standard Dropout versus HiddenCut, and the placement of HiddenCut within a Transformer block.

The method employs three key components:

  1. Contiguous Spans: By masking contiguous blocks, HiddenCut prevents the model from easily interpolating missing information from adjacent neurons, a common failure mode of standard dropout.
  2. Strategic Sampling: Instead of random masking, an attention-based strategy identifies "high-importance" tokens. By strategically masking these informative spans, the model is forced to look beyond its primary "shortcuts" and explore the "long tail" of linguistic features.
  3. Consistency Training: The model is trained to maintain consistent predictions between original and "cut" versions of the data using Jensen-Shannon Divergence loss.

Experimental Evaluation

The researchers evaluated HiddenCut across a diverse set of benchmarks:

  • In-Distribution: The standard GLUE benchmark (SST-2, MNLI, QQP, etc.).
  • Out-of-Distribution (OOD) & Counterfactuals: Challenging sets including IMDB Contrast/Counterfactual Sets, PAWS-QQP, HANS (lexical overlap heuristics), and Adversarial NLI (AdvNLI).

Results and Discussion

HiddenCut consistently outperformed RoBERTa-base and other state-of-the-art augmentation methods like ALUM and Span Cutoff. It achieved a higher average score on GLUE (87.8 vs. 86.3 base) and showed significant gains in OOD scenarios.

Performance of HiddenCut compared to RoBERTa-base and Span Cutoff across various OOD and challenging benchmarks.

As shown in the results, the improvements are most pronounced on challenging datasets like HANS and IMDB-CAD, where the model must move beyond simple heuristics. Furthermore, HiddenCut is computationally efficient; its attention-based sampling leverages existing model calculations, avoiding the expensive backward passes required by adversarial training methods.

Conclusion

HiddenCut provides a simple yet powerful regularization mechanism for PLMs. By forcing models to look beyond the most obvious hidden features through strategic span masking, it encourages the learning of more generalizable linguistic patterns. While hyperparameter sensitivity remains a consideration, the method's efficiency and superior performance on OOD data make it a compelling tool for robust natural language understanding.

Summary updated

Abstract

Fine-tuning large pre-trained models with task-specific data has achieved great success in NLP. However, it has been demonstrated that the majority of information within the self-attention networks is redundant and not utilized effectively during the fine-tuning stage. This leads to inferior results when generalizing the obtained models to out-of-domain distributions. To this end, we propose a simple yet effective data augmentation technique, HiddenCut, to better regularize the model and encourage it to learn more generalizable features. Specifically, contiguous spans within the hidden space are dynamically and strategically dropped during training. Experiments show that our HiddenCut method outperforms the state-of-the-art augmentation methods on the GLUE benchmark, and consistently exhibits superior generalization performances on out-of-distribution and challenging counterexamples. We have publicly released our code at https://github.com/GT-SALT/HiddenCut.
BibTeX
@inproceedings{chen2021hiddencut,
  author = {Jiaao Chen and Dinghan Shen and Weizhu Chen and Diyi Yang},
  title = {{HiddenCut: Simple Data Augmentation for Natural Language Understanding with Better Generalization}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=KIogqW7xn-H},
}
Figure