Understanding the Dataset Practitioners Behind Large Language Model Development

Crystal Qian, Emily Reif, Minsuk Kahng

Abstract

As large language models (LLMs) become more advanced and impactful, it is increasingly important to scrutinize the data that they rely upon and produce. What is it to be a dataset practitioner doing this work? We approach this in two parts: first, we define the role of "dataset practitioners" by performing a retrospective analysis on the responsibilities of teams contributing to LLM development at a technology company, Google. Then, we conduct semi-structured interviews with a cross-section of these practitioners (N=10). We find that although data quality is a top priority, there is little consensus around what data quality is and how to evaluate it. Consequently, practitioners either rely on their own intuition or write custom code to evaluate their data. We discuss potential reasons for this phenomenon and opportunities for alignment.

Introduction

In the transition from the "big data" era to the era of Large Language Models (LLMs), the industry's focus has shifted from sheer volume to high-quality, curated datasets. While model architectures and parameter counts often dominate public discourse, the "data work" behind these models remains largely opaque. LLM data—primarily unstructured text—presents unique challenges that traditional statistical analysis cannot fully address. This study investigates the "dataset practitioners" responsible for curating and evaluating this data, seeking to understand their workflows, the tools they use, and the bottlenecks they face.

Methodology

The researchers employed a two-phase qualitative approach at Google:

  1. Retrospective Analysis: An analysis of internal organizational structures and projects was conducted to define the "dataset practitioner" role, identifying tasks ranging from benchmark curation to deployment decision-making.
  2. Semi-Structured Interviews: 30-minute interviews were conducted with 10 practitioners (N=10) across three primary domains: Tooling (infrastructure builders), Modeling (model trainers), and Evaluation (safety and performance testers).
  3. Thematic Analysis: Interview transcripts were coded to identify recurring themes, challenges, and tool usage patterns.

A Taxonomy of Data Work

The study produced a comprehensive taxonomy mapping the processes, tools, and challenges faced by dataset practitioners across different roles.

Taxonomy of the Dataset Practitioner, mapping processes, tools, and challenges across Tooling, Modeling, and Evaluation roles.

Key Findings

The Paradox of Data Quality

A striking paradox emerged: while 100% of participants cited "Assessing Data Quality" as a major challenge and a top priority, there is no consensus on what "quality" actually means. Definitions vary significantly between teams; for instance, what one team considers high-quality tone for a chatbot might be viewed as noisy or unhelpful by another.

The "Eyeball" Standard and Tooling Gap

Despite the existence of sophisticated AI explainability (XAI) and visualization tools, every participant reported that their primary method for validating data quality was "eyeballing"—visually scanning examples in a spreadsheet. Practitioners rely heavily on their own intuition, leading to a known risk of confirmation bias where they only check data when model outputs appear "wrong."

Furthermore, a significant tooling gap exists. Seven out of nine participants mentioned writing custom Python code in notebooks (e.g., Colab) to explore data, citing a need for flexibility that current UI-based tools lack. Spreadsheets remain dominant not only for their ease of use but also as "social" tools, facilitating collaboration with non-technical stakeholders like legal or policy teams who must sign off on safety datasets.

Desired Features

When asked what they actually wanted in a tool, practitioners converged on three core needs:

  1. Better summarization of specific data slices.
  2. Automated safety and toxicity thresholding.
  3. Improved methods for tracking numeric distributions, such as token length.

Discussion and Future Directions

The reliance on manual intuition and custom scripts suggests that current data evaluation standards lack rigor. The study identifies a "chicken and egg" problem: practitioners avoid specialized tools that don't fit their custom needs, while tools fail to evolve because practitioners are busy building one-off solutions.

Future work should focus on developing conceptual frameworks to help teams reach a consensus on context-specific data quality, standardizing metrics for features like diversity, and designing flexible tools that combine the collaborative ease of spreadsheets with the analytical power of custom Python environments.

Limitations

This study is limited by its small sample size (N=10) and its focus on a single large technology company. The workflows of these practitioners may not fully represent those in smaller startups or academic settings.

Summary updated

Abstract

As large language models (LLMs) become more advanced and impactful, it is increasingly important to scrutinize the data that they rely upon and produce. What is it to be a dataset practitioner doing this work? We approach this in two parts: first, we define the role of "dataset practitioners" by performing a retrospective analysis on the responsibilities of teams contributing to LLM development at a technology company, Google. Then, we conduct semi-structured interviews with a cross-section of these practitioners (N=10). We find that although data quality is a top priority, there is little consensus around what data quality is and how to evaluate it. Consequently, practitioners either rely on their own intuition or write custom code to evaluate their data. We discuss potential reasons for this phenomenon and opportunities for alignment.
BibTeX
@inproceedings{qian2024understanding,
  author = {Crystal Qian and Emily Reif and Minsuk Kahng},
  title = {{Understanding the Dataset Practitioners Behind Large Language Model Development}},
  booktitle = {CHI Extended Abstracts},
  year = {2024},
  doi = {10.1145/3613905.3651007},
}

Loading related content...

Figure