TL;DR
This paper introduces SimReg, a novel method for mitigating dataset bias in NLP models by regularizing internal representations. Instead of just making models behave differently from biased ones, SimReg encourages representations to be either similar to those of an unbiased model or dissimilar to those of a biased model, leading to improved out-of-distribution performance with minimal in-distribution degradation.
What Problem Does This Paper Address?
Modern Natural Language Understanding (NLU) models are prone to learning "shortcuts" – spurious correlations between input features and labels that are not causally related to the true task. For example, a model might learn that the presence of a negation word like "not" strongly correlates with a "contradiction" label in Natural Language Inference (NLI) tasks, even if the overall sentence meaning doesn't support this. This leads to poor generalization on out-of-distribution (OOD) datasets where these shortcuts don't hold. Existing debiasing methods often operate extrinsically, on the model's output, and surprisingly, can sometimes increase the bias in the model's internal representations. This paper tackles the problem of mitigating dataset bias by focusing on the model's internal representations, aiming for a more robust and intrinsically debiased model.
Methods
The core of SimReg is a three-stage process:
- Train a Bias Model ($f_b$): A model is trained on the original dataset to capture specific, known biases (e.g., lexical overlap, partial input) or unknown biases using limited-capacity models.
- Obtain an Unbiased Guidance Model ($f_g$): The training set is filtered based on the bias model's confident predictions to create a cleaner, "unbiased" subset ($D_U$). A guidance model ($f_g$) is then trained on this subset.
- Train the Main Model ($f_m$): The main model is trained on the full dataset, but its internal representations are regularized to be either:
- Similar to $f_g$: Encouraging the main model to learn representations that align with the unbiased guidance model.
- Dissimilar to $f_b$: Pushing the main model's representations away from those learned by the biased model.
A key technical innovation is the use of second-order similarity measures, specifically Centered Kernel Alignment (CKA), to compare representations between models. This is crucial because models might have different architectures and dimensionalities, making direct vector comparison impossible. The regularization is applied at the batch level, and importantly, is applied only on the biased subset of data ($D_B$) to avoid discarding useful information from unbiased samples.
Datasets
The paper evaluates SimReg on several NLU tasks and datasets:
- Natural Language Inference (NLI):
- MNLI: The primary training dataset for NLI tasks.
- HANS: An OOD challenge set designed to evaluate lexical overlap bias.
- MNLI-hard: A subset of MNLI's dev set for evaluating hypothesis-only bias.
- Synthetic MNLI: A custom dataset created to introduce synthetic spurious correlations for controlled experiments.
- Fact Verification:
- FEVER: A dataset for fact verification.
- FEVER-Symmetric: An OOD evaluation set for claim-only bias.
- Question Answering:
- QQP: A dataset for paraphrase identification.
- PAWS: A dataset designed to be adversarial to lexical overlap bias.
Key Insights and Ideas
- Intrinsic vs. Extrinsic Debiasing: The paper argues that focusing on internal representations (intrinsic) is more effective than solely modifying outputs (extrinsic), especially given the counter-intuitive finding that extrinsic debiasing can sometimes increase representation bias.
- Learning from "Good" Models: Instead of just learning to be different from biased models, SimReg proposes learning to be similar to unbiased models, leveraging the knowledge from a cleaner data subset.
- Second-Order Similarity for Representation Comparison: The use of CKA is a critical technical contribution, enabling robust comparison of representations across models with varying architectures and dimensions.
- Regularizing Biased Subsets: Applying regularization only on the subset of data identified as biased ($D_B$) is shown to be more effective than regularizing the entire dataset, suggesting a targeted approach to debiasing.
- Top Layers Matter Most: Analysis shows that fine-tuning and debiasing primarily affect the top layers of a model, and SimReg successfully aligns these layers with the unbiased guidance model.
Key Results
- Improved OOD Performance: SimReg consistently outperforms baseline models and other debiasing methods (like PoE and ConfReg) on OOD challenge sets across various tasks and biases (lexical overlap, partial input, hypothesis-only, unknown biases).
- Minimal In-Distribution Degradation: While improving OOD performance, SimReg incurs only a small decrease in in-distribution (ID) performance, indicating a good ID-OOD tradeoff.
- Reduced Representation Bias: Experiments involving retraining a classifier on debiased representations show that SimReg models exhibit the weakest signal of spurious correlations, suggesting more robust internal representations.
- Effectiveness on Stronger Models: SimReg also shows improvements when applied to stronger pre-trained models like DeBERTa-V3, not just BERT.
Limitations and Future Work
- Reliance on Unbiased Samples: The success of SimReg, like most debiasing methods, depends on the availability of a sufficient number of non-biased samples to train the guidance model.
- Unknown Bias Challenges: For unknown biases, there's a risk that the chosen "weak" bias model might either miss the bias or overfit to non-biased samples, requiring careful inspection.
- Future Directions:
- Investigating the effect of simultaneously learning from both unbiased and biased models.
- Extending the approach to generation tasks, which would require developing different similarity measures.
- Testing SimReg on other types of biases, such as social biases (gender, racial).
Summary updated