MetaShift: A Dataset of Datasets for Evaluating Contextual Distribution Shifts and Training Conflicts

Weixin Liang, James Zou

TL;DR

This paper introduces MetaShift, a novel dataset comprising over 12,000 distinct image subsets, each representing a unique real-world data distribution. MetaShift aims to address the limitations of existing benchmarks by offering a vastly larger scale of natural distribution shifts, explicit annotations for what differentiates these shifts, and a quantifiable measure of shift magnitude, enabling more robust evaluation of machine learning models.

What Problem Does This Paper Address?

Machine learning models often struggle when deployed in environments with data distributions different from their training data. This "distribution shift" can lead to significant performance degradation and unreliable behavior. While efforts have been made to create datasets that capture these shifts, existing benchmarks are often limited in:

  • Scale: They typically contain only a small number of distinct distribution shifts.
  • Annotation: They lack detailed explanations of why a shift occurs or what specific contextual changes are involved.
  • Control: It's difficult to precisely control the magnitude of the distribution shift, making it hard to systematically study model behavior under varying levels of challenge.

This gap hinders our ability to thoroughly evaluate and develop models that are truly robust and reliable in real-world, ever-changing scenarios.

Methods

The core innovation of MetaShift lies in its construction methodology, which leverages the rich metadata available in the Visual Genome dataset. The process involves:

  1. Candidate Subset Generation: For each image class (e.g., "cat"), images are grouped based on their associated metadata tags (e.g., "cat with sink," "cat with fence"). This creates numerous potential subsets, each representing a distinct contextual distribution.
  2. Meta-Graph Construction: To model the relationships and similarities between these subsets, a weighted undirected graph (meta-graph) is built. The weight of an edge between two subsets represents their overlap coefficient, indicating how often their contexts co-occur.
  3. Quantifying Distribution Shifts: Spectral embeddings are used to map each subset (node in the meta-graph) to a low-dimensional space. The Euclidean distance between these embeddings serves as a quantifiable measure of the distribution shift between any two subsets. This provides a controlled way to simulate shifts of varying magnitudes.

This approach allows for the creation of a massive number of naturally occurring, well-annotated, and controllable distribution shifts.

Datasets

  • Base Dataset: Visual Genome (used for constructing MetaShift).
  • Newly Introduced Benchmark: MetaShift. This dataset consists of 12,868 sets of natural images across 410 classes. It is built by clustering images from Visual Genome based on their metadata, creating distinct contextual subsets. MetaShift offers significantly more distribution shifts (over 12,000) compared to existing benchmarks, which typically have only a handful. It also provides explicit annotations for each subset and a quantifiable distance score for shifts.
  • Additional Dataset for Generalizability: MS-COCO was also used to demonstrate the applicability of the MetaShift construction methodology to other datasets with metadata.

Key Insights and Ideas

  • Scale Matters: The sheer number of shifts in MetaShift (12,868) allows for a much more comprehensive and nuanced evaluation of model robustness than previously possible.
  • Context is Key: By leveraging metadata, MetaShift captures shifts driven by natural contextual variations (e.g., "cats in a bathroom" vs. "cats outdoors"), which are more realistic than synthetic pixel-level transformations.
  • Quantifiable Shifts Enable Controlled Experiments: The ability to measure and control the magnitude of distribution shifts is crucial for systematically understanding how models behave under increasing stress.
  • Training Conflicts Reveal Deeper Dynamics: MetaShift's annotations enable the visualization of "conflict zones" during training, where certain data subsets might provide conflicting signals, potentially hindering generalization or, conversely, forcing the model to learn more robust features.
  • ERM's Surprising Resilience: Simple Empirical Risk Minimization (ERM) performs surprisingly well, even outperforming specialized robust training methods, especially when distribution shifts are moderate.

Key Results

  • Domain Generalization:
    • ERM performs best when shifts are moderate (small 'd' values).
  • For larger shifts, specialized domain generalization methods show improvement over ERM, but no single method consistently dominates. This highlights the ongoing challenge in this area.
  • Experiments on a 10-class animal classification task confirmed these findings.
  • Subpopulation Shifts:
    • ERM excels in average accuracy when minority groups are well-represented (large 'p').
  • However, for worst-group accuracy, specialized algorithms generally outperform ERM, demonstrating their effectiveness in mitigating performance disparities.
  • Similar to domain generalization, no single algorithm consistently wins across all shift magnitudes.
  • Training Conflicts:
    • Visualizations revealed "conflict zones" where training on certain subsets negatively impacted performance on others.
  • Interestingly, these conflicts, particularly between "indoor" subsets, seemed to push models towards learning more generalizable features for out-of-domain evaluation.
  • Subsets that were "too easy" showed low training contributions and led to models that performed near-perfectly on them, suggesting they provided little learning signal.

Limitations and Future Work

  • Inherited Biases: MetaShift inherits potential biases present in the base dataset (Visual Genome). The authors acknowledge this and suggest using MetaShift to study how data subsets can contribute to model biases.
  • Metadata Imperfection: While noise in metadata can make tasks more realistic, it's a potential limitation. The authors argue that this noise actually enhances realism for subpopulation shift scenarios.
  • Future Work:
    • Exploring other off-the-shelf node embedding methods like Node2Vec for meta-graph analysis.
  • Extending the methodology to other domains beyond image data.
  • Further investigation into the "conflict zones" and their role in learning generalizable features.
  • Developing new methods that can systematically outperform ERM on large distribution shifts.

Summary updated

Abstract

Understanding the performance of machine learning models across diverse data distributions is critically important for reliable applications. Motivated by this, there is a growing focus on curating benchmark datasets that capture distribution shifts. While valuable, the existing benchmarks are limited in that many of them only contain a small number of shifts and they lack systematic annotation about what is different across different shifts. We present MetaShift--a collection of 12,868 sets of natural images across 410 classes--to address this challenge. We leverage the natural heterogeneity of Visual Genome and its annotations to construct MetaShift. The key construction idea is to cluster images using its metadata, which provides context for each image (e.g. "cats with cars" or "cats in bathroom") that represent distinct data distributions. MetaShift has two important benefits: first, it contains orders of magnitude more natural data shifts than previously available. Second, it provides explicit explanations of what is unique about each of its data sets and a distance score that measures the amount of distribution shift between any two of its data sets. We demonstrate the utility of MetaShift in benchmarking several recent proposals for training models to be robust to data shifts. We find that the simple empirical risk minimization performs the best when shifts are moderate and no method had a systematic advantage for large shifts. We also show how MetaShift can help to visualize conflicts between data subsets during model training.
BibTeX
@inproceedings{liang2022metashift,
  author = {Weixin Liang and James Zou},
  title = {{MetaShift: A Dataset of Datasets for Evaluating Contextual Distribution Shifts and Training Conflicts}},
  booktitle = {International Conference on Learning Representations},
  year = {2022},
  url = {https://openreview.net/forum?id=MTex8qKavoS},
}
Figure