Abstract
Classifiers are biased when trained on biased datasets. As a remedy, we propose Learning to Split (ls), an algorithm for automatic bias detection. Given a dataset with input-label pairs, ls learns to split this dataset so that predictors trained on the training split cannot generalize to the testing split. This performance gap suggests that the testing split is under-represented in the dataset, which is a signal of potential bias. Identifying non-generalizable splits is challenging since we have no annotations about the bias. In this work, we show that the prediction correctness of each example in the testing split can be used as a source of weak supervision: generalization performance will drop if we move examples that are predicted correctly away from the testing split, leaving only those that are mis-predicted. ls is task-agnostic and can be applied to any supervised learning problem, ranging from natural language understanding and image classification to molecular property prediction. Empirical results show that ls is able to generate astonishingly challenging splits that correlate with human-identified biases. Moreover, we demonstrate that combining robust learning algorithms (such as group DRO) with splits identified by ls enables automatic de-biasing. Compared to previous state-of-the-art, we substantially improve the worst-group performance (23.4% on average) when the source of biases is unknown during training and validation.
Introduction
Machine learning models often inherit biases from their training data, leading to a phenomenon known as "bias in, bias out." If a dataset contains hidden correlations—such as polar bears always appearing against snowy backgrounds—a model may learn to identify the background rather than the animal. Traditionally, mitigating these biases required experts to pre-identify sensitive attributes (e.g., gender or race). However, real-world biases are often "unknown unknowns," consisting of subtle annotation artifacts or spurious correlations that even domain experts might miss.
The Learning to Split (ls) Framework
The proposed Learning to Split (ls) algorithm automates bias detection without prior knowledge of the bias type. It sets up a game between two components:
- The Splitter: This model learns to partition the dataset into a "training split" and a "testing split" to maximize the performance gap.
- The Predictor: A standard classifier trained on the training split and evaluated on the testing split.
The Splitter uses the Predictor's mistakes as a source of weak supervision. If the Predictor correctly classifies a test example, the Splitter moves it to the training set. If it fails, the example remains in the test set. This iterative process refines the test split to contain the most challenging, under-represented groups.
Generalization Gap as a Signal
Figure 1: The ls algorithm identifies splits where spurious correlations (e.g., background color) lead to high training accuracy but zero testing accuracy. Source, p. 2
The core indicator of bias is the generalization gap. A large disparity between training and testing performance suggests the model has relied on features present in the training set that do not hold in the test set. As shown in Figure 1, if the training set only contains Samoyeds on grass and Polar Bears on snow, a model might use background color as a shortcut. When tested on a split where these correlations are reversed, the model fails completely. To ensure splits are meaningful, the authors impose regularity constraints, requiring that splits are of comparable size and maintain similar label distributions.
Empirical Evaluation and De-biasing
The authors evaluated ls across diverse domains, including computer vision (Waterbirds, CelebA), Natural Language Processing (Beer Reviews, MNLI), and molecular chemistry (Tox21).
- Bias Discovery: The splits discovered by
lscorrelated strongly with human-identified biases. In the Waterbirds dataset,lsautomatically isolated "landbirds on water backgrounds" into the test split. - Performance Impact: On the MNLI dataset, a standard BERT model's accuracy plummeted from 79.4% on a random split to just 27.8% on the split identified by
ls. - Automatic De-biasing: By combining
lswith robust optimization algorithms like Group DRO, the system can find and fix biases in a single pipeline. This approach improved worst-group performance by 23.4% on average compared to previous state-of-the-art methods when the bias source was unknown.
Limitations and Future Work
While powerful, ls can only detect biases present in the data; it cannot address "support" issues where a minority group is entirely missing. Additionally, the iterative nature of the algorithm increases computational costs, as the Predictor must be re-trained in each outer-loop iteration. Future research may focus on identifying multiple, independent sources of bias simultaneously.
Summary updated