Learning to Split for Automatic Bias Detection

Yujia Bao, Regina Barzilay

TL;DR

Learning to Split (ls) is an algorithm that automatically uncovers hidden dataset biases without requiring human bias annotations. By learning to partition data into training and test splits where a predictor fundamentally fails to generalize, ls isolates under-represented slices and spurious correlations. When paired with group distributionally robust optimization (Group DRO), it delivers massive gains in worst-group robustness (an average 23.4% improvement) under realistic settings where bias sources are entirely unknown during both training and model selection.

What Problem Does This Paper Address?

Supervised models frequently latch onto spurious shortcuts rather than robust features—such as associating polar bears strictly with snow backgrounds or natural language inference contradictions with the presence of negation words.

Existing debiasing algorithms typically assume that bias attributes (e.g., race, gender, background) are known and labeled, or they quietly rely on bias annotations in the validation set to tune hyperparameters and choose models. When bias attributes are completely unknown across both training and validation sets, existing debiasing techniques often collapse back to standard Empirical Risk Minimization (ERM). The authors address a fundamental challenge: How can we automatically detect and mitigate hidden biases in an arbitrary dataset using only input-label pairs $(x, y)$?

Methods

The paper introduces Learning to Split (ls), framing bias detection as an adversarial data partitioning problem. It operates through an iterative game between two components:

  1. Splitter: A model that outputs the probability $P_{\text{Splitter}}(z_i \mid x_i, y_i)$ of assigning each example to the training split ($z_i = 1$) or the test split ($z_i = 0$).
  2. Predictor: A task classifier trained from scratch on the proposed training split $D_{\text{train}}$ via ERM (using an early-stopping split to prevent overfitting).
   ┌────────────────────────────────────────────────────────┐
   │                        Splitter                        │
   │  Partitions dataset into D_train (z=1) & D_test (z=0)  │
   └─────────────┬────────────────────────────▲─────────────┘
                 │ Samples splits             │ Updates Splitter
                 ▼                            │ via L_gap + Ω_1 + Ω_2
   ┌─────────────────────────────┐            │
   │          Predictor          │            │
   │  Trains on D_train;         ├────────────┘
   │  evaluates on D_test        │ Weak supervision: correctly predicted
   └─────────────────────────────┘ test samples are routed to D_train

Key Technical Contributions:

  • Prediction Correctness as Weak Supervision: Instead of computing expensive meta-gradients or relying on sparse reinforcement learning rewards, ls updates the Splitter using cross-entropy against the Predictor’s correctness on test points. Correctly predicted test instances are shifted toward the training set, leaving hard, mispredicted instances in the test set.
  • Regularity Constraints: Poor generalization can trivially occur if the training set is too small or if labels are completely separated between splits. The authors enforce two soft KL-divergence regularizers:
  • $\Omega_1$: Constrains the overall split ratio toward a target balance (setting training size around 75%).
  • $\Omega_2$: Matches conditional label distributions across splits to prevent degenerate label shifts.
  • Unsupervised De-biasing: For downstream training, the learned binary splitting variable $z$ is crossed with the task label $y$ to form latent groups (e.g., $y \times z$) for Group DRO. The Splitter is also evaluated on the validation set as a surrogate metric for model selection without needing bias labels.

Datasets

The authors evaluated ls across three distinct domains:

  • Vision:
  • Waterbirds: Classifying waterbirds vs. landbirds (spurious correlation with land/water backgrounds).
  • CelebA: Hair color classification (blond vs. non-blond, spuriously correlated with gender).
  • Natural Language Understanding:
  • MNLI: Natural language inference (contradiction, entailment, neutral), where contradictions correlate spuriously with negation words.
  • Beer Reviews (BeerAdvocate): Aspect sentiment classification for "look" and "aroma."
  • Molecular Property Prediction:
  • Tox21: Predicting the activity of the Androgen Receptor (AR) across 12,707 chemical compounds.

Key Insights and Ideas

  • Generalization gaps expose data under-representation: When a model fits a training split but fails on a regularized test split, the failure reflects differing predictive mechanisms (shortcuts) rather than capacity issues or simple label imbalances.
  • Human-aligned biases emerge spontaneously: Without ever seeing human bias annotations, ls concentrated spurious features (e.g., landbirds on land or male blonds) into skewed splits. On MNLI, for example, ls funneled examples with negation words into contradiction labels on train, while placing negation words with entailment/neutral labels in test.
  • Exposing the "validation leakage" in previous work: Prior methods claiming to mitigate "unknown" biases often used ground-truth bias labels in the validation set to pick their best checkpoint. Without this hidden supervision, prior methods fell back to baseline accuracy. ls closes this gap by using the learned Splitter directly for validation model selection.

Key Results

  • Creating Non-Generalizable Splits: On MNLI, a standard BERT model achieves 79.4% test accuracy under random splitting, but plummets to 27.8% on the ls split (a 65.8% generalization gap). On Tox21, AUC-ROC drops from 78.1% to 33.4%.
  • Worst-Group Robustness Without Bias Labels: Under the fully blind setting (no bias labels in train or validation), Group DRO trained with ls splits outperformed prior state-of-the-art methods across benchmarks:
  • Waterbirds: 86.1% worst-group accuracy (vs. 64.8% for ERM, 62.5% for JTT).
  • CelebA: 83.3% worst-group accuracy (vs. 41.1% for ERM, 40.6% for JTT).
  • MNLI: 72.1% worst-group accuracy (vs. 60.4% for ERM, 64.4% for JTT).
  • Across these tasks, ls improved worst-group performance by 23.4% on average over previous baselines.

Limitations and Future Work

  • Reliance on Data Support: ls can only isolate bias if minority groups exist in the dataset. If all polar bears appear on snow and none on grass, no split can expose the background shortcut.
  • Sensitivity to Label Noise: Because the Splitter observes input-label pairs, it can optimize the generalization gap by dumping mislabeled instances into the test set, acting as a label-noise detector rather than a bias detector.
  • Computational Overhead: Retraining a predictor from scratch at each outer loop (typically converging in ~50 iterations) is computationally heavier than standard ERM.
  • Single-Split Assumption: Datasets often contain multiple distinct, interacting biases (e.g., race, gender, and age). The authors leave finding multiple, diverse splits to future work.

Summary updated

Abstract

Classifiers are biased when trained on biased datasets. As a remedy, we propose Learning to Split (ls), an algorithm for automatic bias detection. Given a dataset with input-label pairs, ls learns to split this dataset so that predictors trained on the training split cannot generalize to the testing split. This performance gap suggests that the testing split is under-represented in the dataset, which is a signal of potential bias. Identifying non-generalizable splits is challenging since we have no annotations about the bias. In this work, we show that the prediction correctness of each example in the testing split can be used as a source of weak supervision: generalization performance will drop if we move examples that are predicted correctly away from the testing split, leaving only those that are mispredicted. ls is task-agnostic and can be applied to any supervised learning problem, ranging from natural language understanding and image classification to molecular property prediction. Empirical results show that ls is able to generate astonishingly challenging splits that correlate with human-identified biases. Moreover, we demonstrate that combining robust learning algorithms (such as group DRO) with splits identified by ls enables automatic de-biasing. Compared to previous state-of-the-art, we substantially improve the worst-group performance (23.4% on average) when the source of biases is unknown during training and validation. Our code is included in the supplemental materials and will be publicly available.
BibTeX
@inproceedings{bao2023learning,
  author = {Yujia Bao and Regina Barzilay},
  title = {{Learning to Split for Automatic Bias Detection}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=IA96Pn7A08h},
}

Loading related content...

Figure