Abstract
Fine-tuning large pre-trained models with task-specific data has achieved great success in NLP. However, it has been demonstrated that the majority of information within the self-attention networks is redundant and not utilized effectively during the fine-tuning stage. This leads to inferior results when generalizing the obtained models to out-of-domain distributions. To this end, we propose a simple yet effective data augmentation technique, HiddenCut, to better regularize the model and encourage it to learn more generalizable features. Specifically, contiguous spans within the hidden space are dynamically and strategically dropped during training. Experiments show that our HiddenCut method outperforms the state-of-the-art augmentation methods on the GLUE benchmark, and consistently exhibits superior generalization performances on out-of-distribution and challenging counterexamples.
Introduction
Pre-trained Language Models (PLMs) like RoBERTa have revolutionized NLP, yet they often struggle with a "generalization gap" when fine-tuned on relatively small task-specific datasets. These powerful models frequently rely on spurious correlations—statistical artifacts in training data that do not reflect the true underlying logic of a task. While standard regularization like Dropout attempts to mitigate this by randomly zeroing out neurons, it is often insufficient for modern Transformers. Due to the high redundancy in self-attention networks, models can easily reconstruct missing random neurons from their neighbors, failing to learn deeper, more robust linguistic features.
Methodology: The HiddenCut Approach
HiddenCut addresses these limitations by shifting the "cutting" of information from the input space to the model's internal hidden representations. Unlike standard dropout, which acts on isolated units, HiddenCut masks contiguous blocks of hidden representations corresponding to token sequences.
The method employs three key components:
- Contiguous Spans: By masking contiguous blocks, HiddenCut prevents the model from easily interpolating missing information from adjacent neurons, a common failure mode of standard dropout.
- Strategic Sampling: Instead of random masking, an attention-based strategy identifies "high-importance" tokens. By strategically masking these informative spans, the model is forced to look beyond its primary "shortcuts" and explore the "long tail" of linguistic features.
- Consistency Training: The model is trained to maintain consistent predictions between original and "cut" versions of the data using Jensen-Shannon Divergence loss.
Experimental Evaluation
The researchers evaluated HiddenCut across a diverse set of benchmarks:
- In-Distribution: The standard GLUE benchmark (SST-2, MNLI, QQP, etc.).
- Out-of-Distribution (OOD) & Counterfactuals: Challenging sets including IMDB Contrast/Counterfactual Sets, PAWS-QQP, HANS (lexical overlap heuristics), and Adversarial NLI (AdvNLI).
Results and Discussion
HiddenCut consistently outperformed RoBERTa-base and other state-of-the-art augmentation methods like ALUM and Span Cutoff. It achieved a higher average score on GLUE (87.8 vs. 86.3 base) and showed significant gains in OOD scenarios.
As shown in the results, the improvements are most pronounced on challenging datasets like HANS and IMDB-CAD, where the model must move beyond simple heuristics. Furthermore, HiddenCut is computationally efficient; its attention-based sampling leverages existing model calculations, avoiding the expensive backward passes required by adversarial training methods.
Conclusion
HiddenCut provides a simple yet powerful regularization mechanism for PLMs. By forcing models to look beyond the most obvious hidden features through strategic span masking, it encourages the learning of more generalizable linguistic patterns. While hyperparameter sensitivity remains a consideration, the method's efficiency and superior performance on OOD data make it a compelling tool for robust natural language understanding.
Summary updated