TL;DR
This paper introduces HiddenCut, a novel data augmentation technique for natural language understanding (NLU) models. By strategically dropping contiguous spans of hidden representations within transformer layers, HiddenCut encourages models to learn more robust and generalizable features, leading to improved performance on out-of-distribution datasets.
What Problem Does This Paper Address?
Large pre-trained language models (PLMs) have achieved remarkable success in NLU tasks. However, a significant challenge arises from the vast number of parameters in these models compared to the often limited task-specific data available for fine-tuning. This data scarcity can lead to overfitting, where models learn spurious patterns from the training data, resulting in poor generalization to unseen or out-of-domain distributions. Existing regularization techniques like dropout, while helpful, often ignore the linguistic relationships between words, making the dropped information easily inferable and thus less effective for true regularization.
Methods
The core of HiddenCut lies in its approach to data augmentation. Instead of randomly dropping individual hidden units (like standard dropout), HiddenCut dynamically masks entire contiguous spans of hidden representations after each transformer layer. This "cutting" is guided by an attention-based sampling strategy. The intuition is that adjacent words often share similar or redundant information, and by removing these contiguous blocks, the model is forced to rely on a broader range of features, thus mitigating reliance on spurious correlations.
The authors propose several sampling strategies to determine which spans to cut:
- Attention-based Sampling: This is the primary strategy, where spans are selected based on the attention weights assigned to tokens in the self-attention layers. The idea is to drop spans that are highly attended to, forcing the model to explore less-attended but potentially crucial information.
- Random Sampling: A baseline where spans are chosen randomly.
- LIME, GEM, Gradient: These are other established methods for identifying important tokens, adapted to select spans for cutting.
HiddenCut is applied after the feed-forward network in each transformer layer. The authors also incorporate a Jensen-Shannon Divergence consistency regularization to encourage consistency between the original and augmented examples.
Datasets
The paper evaluates HiddenCut on a comprehensive set of datasets:
- In-Distribution Datasets (GLUE Benchmark): This includes a wide range of NLU tasks:
- Single-Sentence Tasks: Stanford Sentiment Treebank (SST-2), Corpus of Linguistic Acceptability (CoLA).
- Similarity and Paraphrase Tasks: Quora Question Pairs (QQP), Semantic Textual Similarity Benchmark (STS-B), Microsoft Research Paraphrase Corpus (MRPC).
- Inference Tasks: Multi-Genre Natural Language Inference (MNLI), Question Natural Language Inference (QNLI), Recognizing Textual Entailment (RTE).
- Out-Of-Distribution Datasets: To specifically test generalization, models fine-tuned on GLUE were evaluated on:
- Single Sentence Tasks: IMDB Contrast Set, IMDB Counterfactually Augmented Dataset.
- Similarity and Paraphrase Tasks: PAWS-QQP.
- Inference Tasks: HANS, Adversarial NLI (Adversarial sets).
Key Insights and Ideas
- Linguistic Structure Matters for Regularization: The paper strongly suggests that simply dropping random units (dropout) is less effective than strategically removing linguistically related spans of information. This is because the remaining information can often compensate for randomly dropped units.
- Attention as a Guide for Augmentation: Leveraging attention weights to identify informative spans for removal is a key innovation. This allows for a more targeted and linguistically informed regularization.
- Balancing Perturbation and Noise: The "HiddenCut ratio" ($\alpha$) is crucial. Too little cutting might not provide sufficient regularization, while too much can introduce excessive noise and degrade performance. Finding the right balance is key.
- Out-of-Distribution Performance is Key: The paper emphasizes that true generalization is best measured on challenging, out-of-distribution datasets. HiddenCut shows significant improvements here, suggesting it addresses the core problem of spurious correlations.
Key Results
- In-Distribution: HiddenCut consistently outperformed baselines, including state-of-the-art methods like ALUM and Cutoff, on 7 out of 8 GLUE tasks. It achieved the best average score across the GLUE benchmark.
- Out-of-Distribution: HiddenCut demonstrated substantial improvements over RoBERTa and Span Cutoff on challenging out-of-distribution datasets. For instance, it outperformed Span Cutoff by 2.3% on IMDB-Conts and 2.8% on HANS.
- Sampling Strategy: The attention-based sampling strategy proved to be the most effective among the explored methods, outperforming random sampling and other token importance discovery techniques.
- Hyperparameter Sensitivity: The paper found that a HiddenCut ratio ($\alpha$) of 0.1 and a sampling ratio ($\beta$) of 0.4 yielded the best results, indicating the importance of careful hyperparameter tuning.
Limitations and Future Work
- Hyperparameter Tuning: While the paper explores hyperparameter sensitivity, finding the optimal $\alpha$ and $\beta$ might still require significant grid search for new tasks or datasets.
- Computational Overhead: Although HiddenCut is presented as computationally efficient (no extra backward passes like adversarial training), the dynamic span selection and augmentation process still adds some overhead compared to standard fine-tuning.
- Future Work: The authors suggest exploring more sophisticated sampling strategies and investigating the application of HiddenCut to other domains beyond NLU. They also hint at the potential for HiddenCut to be combined with other regularization techniques for even greater gains.
Summary updated