TL;DR
This paper investigates whether models trained on adversarially collected data become overconfident and poorly calibrated, especially on ambiguous inputs. They found that models trained only on adversarial data exhibit worse calibration and overconfidence, but this issue is mitigated when naturalistic data is also included in training.
What Problem Does This Paper Address?
Modern machine learning models, particularly in NLP, often learn spurious correlations from data rather than true underlying task logic. This leads to poor generalization when faced with out-of-distribution examples, such as those encountered in domain shifts or adversarial attacks. Adversarial data collection, where annotators deliberately try to "fool" a model, is a promising technique to create more robust models by exposing these weaknesses. However, the process of creating adversarial data might itself introduce biases, particularly by over-sampling ambiguous or contentious examples where annotator disagreement is high. This raises a crucial question: does training on such adversarially collected data lead to models that are miscalibrated, especially when dealing with these ambiguous inputs?
Methods
The authors investigate this by training Natural Language Inference (NLI) models under three conditions:
- CLASSICAL: Trained on standard, naturalistic NLI datasets (SNLI, MultiNLI).
- ADVERSARIAL: Trained on adversarially collected NLI data (Adversarial NLI - ANLI).
- ALL: Trained on a combination of both naturalistic and adversarial data.
They then evaluate these models on a challenging benchmark called ChaosNLI, which features examples with comprehensive annotator distributions (100 annotations per example). This allows them to analyze model performance across different levels of ambiguity, measured by human annotator agreement. Key metrics include:
- Accuracy in Expectation: Model accuracy against a randomly sampled human annotator.
- Accuracy against Plurality: Model accuracy against the majority vote of human annotators.
- Model Perplexity: A measure of the model's uncertainty.
- KL-Divergence: How well the model's predicted distribution matches the human annotator distribution, serving as a measure of calibration.
Datasets
- Training Data:
- SNLI (Stanford Natural Language Inference)
- MultiNLI (Multi-Genre Natural Language Inference)
- ANLI (Adversarial Natural Language Inference) - specifically, the adversarial annotations.
- Evaluation Data:
- ChaosNLI evaluation sets for SNLI and MultiNLI. These sets are notable for having 100 independent annotations per example, providing a rich signal of annotator disagreement and ambiguity.
Key Insights and Ideas
- Adversarial Data Can Introduce Overconfidence: The most striking finding is that models trained exclusively on adversarial data (ADVERSARIAL) become significantly more overconfident and exhibit worse calibration, particularly on ambiguous inputs. This suggests that the adversarial data collection process, even with filtering for agreement, can lead to a predictable blind spot.
- Ambiguity is a Challenge for All Models: Regardless of training data, all models perform worse on ambiguous examples. This highlights the inherent difficulty of modeling tasks with high annotator disagreement.
- Naturalistic Data is Key for Calibration: The "ALL" model, which combines naturalistic and adversarial data, mitigates the overconfidence and calibration issues seen in the purely adversarial model. This strongly suggests that including a substantial amount of naturalistic data in training is crucial for maintaining good calibration, even when adversarial data is used.
- Adversarial Data May Not Necessarily Improve Accuracy: The study found no clear difference in accuracy between models trained on naturalistic versus adversarial data, but it did reveal significant differences in calibration and confidence.
Key Results
- The ADVERSARIAL model showed much higher KL-Divergence from gold (annotator) distributions compared to CLASSICAL and ALL models, indicating a poorer fit and potentially overconfidence.
- The ADVERSARIAL model exhibited extremely low perplexity on many examples, suggesting it was more confident, and its perplexity grew less with respect to annotator perplexity, implying it was less sensitive to ambiguity.
- Calibration curves (Figure 5 and 6) showed that the ADVERSARIAL model was more confident more often and had significantly worse calibration at high confidence levels, especially on ambiguous inputs.
- The ALL model, trained on both naturalistic and adversarial data, mitigated these overconfidence and calibration issues, performing comparably to the CLASSICAL model in these aspects.
Limitations and Future Work
- Domain Shift Hypothesis: The authors acknowledge that the observed miscalibration in the ADVERSARIAL model might be due to a domain shift between the ANLI dataset and the SNLI/MultiNLI evaluation sets. While the ALL model mitigates this, it's unclear how much naturalistic data is minimally required to achieve this mitigation.
- Scarcity of Ambiguous Examples: The study notes that all models perform worse on ambiguous examples, and this might be due to their relative scarcity in training data or the inherent difficulty of modeling them.
- Future Work: The authors suggest investigating how much naturalistic data is sufficient to mitigate overconfidence, exploring full annotator distributions during training, and developing model calibration techniques specifically for ambiguous inputs. They also highlight the need for future work on measuring, controlling, managing, or representing ambiguity in adversarial annotation itself.
Summary updated