Position: Measure Dataset Diversity, Don't Just Claim It

Dora Zhao, Jerone T. A. Andrews, Orestis Papakyriakopoulos, Alice Xiang

Machine learning researchers frequently claim their datasets are "diverse," but these claims are often based on intuition rather than scientific rigor. This paper audits 135 major image and text datasets through the lens of social science measurement theory, revealing a widespread failure to define, measure, or validate diversity. It offers a framework to turn "diversity" from a buzzword into a verifiable metric.

The Problem: Diversity as a "Vibe"

In the ML community, datasets are often treated as neutral, objective reflections of the world. However, datasets are "political artifacts" infused with the values and biases of their creators.

The core issue is the loose use of value-laden constructs like "diversity," "bias," and "quality." While curators often claim their datasets are diverse to signal fairness or robustness, they rarely provide a formal definition of what "diversity" means in their specific context. This ambiguity leads to "scale-thinking"—the fallacious belief that simply making a dataset larger automatically makes it more diverse—and prevents researchers from actually verifying if a dataset meets its ethical or technical goals.

A Framework for Rigorous Measurement

To address these gaps, the authors propose applying measurement theory—a framework from the social sciences used to turn abstract concepts into numerical representations. This approach evaluates datasets across four critical dimensions: Conceptualization, Operationalization, Reliability, and Validity.

A framework for applying measurement theory to dataset curation, highlighting key considerations and recommendations across four stages.

By following this framework, researchers can move away from vague claims and toward documented, reproducible metrics. For example, instead of assuming a web-scraped dataset is diverse because it is large, researchers must define which axes of diversity (e.g., geographic, linguistic, or demographic) they are targeting and how they intend to measure them.

Auditing the State of ML Datasets

The authors performed a systematic literature review of 135 image and text datasets presented at top-tier venues like CVPR, NeurIPS, and ACL. The audit included massive, industry-standard datasets such as SA-1B, COCO, and OpenImages, categorized by their collection methodology (webscraping, crowdsourcing, direct collection, etc.).

Key Insights and Proxy Failures

  • The "Scale" Trap: Diversity is not an automatic byproduct of scale. Larger datasets can sometimes amplify narrow, "canonical" perspectives if not carefully curated.
  • The Justification Gap: Only 52.9% of the surveyed papers explicitly justified why they needed diverse data. Without a clear objective, the metrics used to measure diversity become arbitrary.
  • Model Performance is a Bad Proxy: Nearly half (48.5%) of the papers used downstream model performance (e.g., higher benchmark accuracy) to claim a dataset was diverse. Measurement theory suggests this is a validity error; model performance can improve due to "shortcuts" or overfitting, which does not prove the underlying data is actually diverse.
  • Erasure of Labor: Only 36.8% of datasets included details about annotator quality or training. Since annotators are the "measurement instruments" of a dataset, ignoring their backgrounds makes it impossible to assess the reliability of diversity claims.

Results: Documentation Debt and Geographic Bias

The audit revealed significant systemic gaps in how the ML field handles data:

  • Documentation Debt: Only 40 out of 135 papers were standalone dataset papers. Most datasets are introduced as secondary appendages to a new model, leaving limited space for detailed documentation on collection and diversity.
  • Quality Control Silence: Only 56.3% of datasets provided specifics about their quality control processes.
  • Geographic Bias: Even datasets claiming "geographic diversity" often exclude massive regions (e.g., most of Africa), yet still use the "diverse" label without qualification.

Moving Forward: Toward Data-Centric Rigor

Rigorous measurement requires time and space that current publication formats—which prioritize models over data—often do not support. The authors advocate for "data-centric" research venues, such as the NeurIPS Datasets and Benchmarks track, to normalize these standards.

Furthermore, as social constructs like "race" or "gender" change over time, a dataset that is considered "diverse" today may be seen as reductive in the future. Future work must focus on developing algorithms and documentation standards that can account for these temporal shifts and ensure that "diversity" remains a meaningful, verifiable technical goal.

Summary updated

BibTeX
@inproceedings{zhao2024position,
  author = {Dora Zhao and Jerone T. A. Andrews and Orestis Papakyriakopoulos and Alice Xiang},
  title = {{Position: Measure Dataset Diversity, Don't Just Claim It}},
  booktitle = {International Conference on Machine Learning},
  year = {2024},
  url = {https://openreview.net/forum?id=jsKr6RVDDs},
}
Figure