Abstract
Current AI benchmarks often suffer from spurious dataset biases—unintended correlations between inputs and labels—that allow models to achieve high scores without learning the underlying task. We present AFLITE, a "bottom-up" algorithmic approach to identify and remove these artifacts. By iteratively filtering out examples that are easily predictable by simple models, AFLITE creates more challenging benchmarks that demand true generalization. Models trained on AFLITE-filtered data demonstrate significantly improved robustness in out-of-distribution (OOD) scenarios, despite being trained on smaller datasets.
Introduction: The Mirage of Superhuman Performance
While modern AI models frequently reach "superhuman" performance on standard benchmarks like ImageNet or SNLI, this success often proves brittle. When tested on out-of-distribution or adversarial data, performance typically collapses. This discrepancy suggests that models are not learning robust features or logical reasoning; instead, they are exploiting "shortcuts" or spurious biases introduced during data collection. For instance, in Natural Language Inference (NLI), the presence of negation words like "not" is often a strong but superficial predictor of a "contradiction" label.
Traditional methods for mitigating bias are "top-down," requiring researchers to manually identify and target specific artifacts. AFLITE offers an automated, systematic alternative for discovering and removing these biases.
AFLITE: Lightweight Adversarial Filtering
AFLITE is a practical approximation of the theoretically optimal (but computationally prohibitive) AFOPT framework. It employs an iterative, greedy filtering process to sanitize datasets:
- Feature Representation: Data is transformed into high-dimensional embeddings (e.g., RoBERTa for text, EfficientNet for images).
- Ensemble Training: An ensemble of $m$ simple linear classifiers is trained on random partitions of the data.
- Predictability Scoring: Each instance is assigned a score based on the fraction of times the ensemble correctly predicts its label when it is in the test partition.
- Filtering: The $k$ most "predictable" (easiest) instances are removed.
- Iteration: The process repeats on the remaining data until the dataset reaches a target size or a predictability threshold.
By using simple linear models as the "adversary," AFLITE targets examples where labels can be inferred from surface-level features rather than complex, task-relevant patterns.
Experimental Results
The authors evaluated AFLITE across synthetic, NLP, and vision datasets, revealing significant inflation in current benchmark scores.
Natural Language Inference (SNLI)
When the SNLI dataset was filtered by AFLITE (reducing it from 550k to 92k examples), RoBERTa’s performance dropped from 92.6% to 62.6%. Crucially, human performance remained relatively high (~80%), indicating that AFLITE successfully removed "easy" examples that models exploited through artifacts, while retaining the core difficulty of the task for humans.
ImageNet
Filtering ImageNet to 40% of its original size (ImageNet-AFLITE) significantly increased the difficulty of the validation set. Top-1 accuracy for an EfficientNet-B7 model fell from 84.4% to 63.5%.
Generalization and Robustness
A key finding is that "less is more" for generalization. Models trained on smaller, AFLITE-sanitized datasets often outperform those trained on the full, biased datasets when tested on OOD benchmarks.
- HANS Benchmark: A model trained on AFLITE-filtered SNLI achieved 94.1% accuracy on the HANS "Lexical Overlap" test, compared to 88.4% for the model trained on the full dataset.
- ImageNet-A: Despite seeing far fewer training examples, the model trained on AFLITE-filtered data showed a 2% improvement on the ImageNet-A adversarial set.
Conclusion
AFLITE demonstrates that benchmark performance is often a poor proxy for true model capability. By automating the discovery and removal of dataset artifacts, AFLITE provides a path toward building more robust and generalizable AI systems. Future work will focus on refining the filtering thresholds to ensure that "genuinely easy" examples—those easy for both humans and models—are preserved, further aligning AI benchmarks with human-level reasoning.
Summary updated