Executable Datasheets: Checking Dataset Claims Without Seeing the Data

Dataset documentation today is a set of promises. A datasheet says the corpus was deduplicated, that personal information was removed, that it covers 2010–2020, that labels are balanced, that it is English-only. Nobody outside the team that built the data can check any of this. That matters most for the language-model systems described at industry venues, which are trained and evaluated on proprietary logs that will never be released. We propose to turn each documentation claim into a small executable test. A language model writes the test from the claim's text. The test runs inside the data owner's environment, and only a verdict and a coarse statistic come out. The owner keeps the data, and an outside reader gets evidence instead of self-report.

The core idea in two sentences: human-written datasheets stay human-written, but every checkable sentence in them is compiled into code that runs against the actual data and reports whether the sentence is true. This converts documentation from an unverifiable narrative into something an academic reviewer can audit about an industrial dataset they will never see.

Core Claim

The transparency literature has built good templates and has named verification as its main hole. It has not built the verification itself. Datasheets for Datasets supplies roughly 50 lifecycle questions and argues that writing the datasheet should not be automated, because the value lies in forcing the creator to reflect. Open Datasheets makes those answers machine-readable in a JSON schema so tools can filter on them. It explicitly leaves open whether the recorded answers, such as consent flags or PII clearance, are true.

Evidence that the answers are often false comes from AutoDDG. That paper profiles tables with statistics plus an LLM to generate descriptions. It reports that human metadata on public portals is missing for over 13% of datasets and contains factual errors, such as saying a dataset covers one year when it covers ten.

Our claim is that claim-conditioned executable checks catch false documentation much more reliably than two alternatives. The first is regenerating a description from the data and comparing it to the datasheet. The second is having an LLM read the datasheet next to a sample of rows. We also claim this can be done while releasing only a few bits per claim.

This design also resolves the tension between human reflection and automation. The human still writes the datasheet, as Gebru et al. insist. Automation only enforces that what the human wrote is true.

Research Program

The system has three parts. First, a claim extractor splits a datasheet into atomic, checkable statements. It uses the Open Datasheets fields where they exist and free text otherwise. It tags each statement by type: temporal range, value range, label distribution, language, duplication rate, presence of PII patterns, schema, or size. Statements that cannot be checked, such as motivations or intended uses, are marked as such rather than guessed at.

Second, a check compiler prompts an LLM to write a short, sandboxed program for each checkable claim. The program returns a verdict (holds, violated, or undeterminable) and one bounded statistic, such as the observed date range or the fraction of rows matching an email pattern. It is constrained to a whitelisted library of data operations, so the owner can review it before it runs.

Third, a report format records the verdicts and the exact check code. It also records the model and host that compiled the checks, plus verdict agreement across repeated compilations. We include the repetition because Towards Reproducible LLM Evaluation shows the same model served through two APIs gives statistically different scores (t = 2.51, p = 0.013). A verification report whose verdicts flip with the host would be worthless.

The academia–industry setting shapes the threat model. The data owner runs the checks and publishes the report. The external reader sees the claims, the check code, and the outputs, but not the data. On the transparency of large AI models proposes controlled access and synthetic data as compromises between proprietary models and open science, but only as policy. Our system is a concrete instance of controlled access, and we measure what it leaks as well as what it catches.

Baselines

The naive reference point is accepting the datasheet as written. Under this baseline, detection recall on false claims is zero by construction and there are no false alarms. It anchors what any verifier adds.

The strongest existing method is profile-then-compare, built from AutoDDG. We run its Content Profiler and Semantic Profiler on the full dataset to produce a data-derived description. We then ask the same LLM to mark each datasheet claim as consistent or inconsistent with that description. It is the strongest competitor because it already sees the full data and already caught real metadata errors.

To keep the comparison fair, both systems get the same compiling LLM (GPT-4o-mini, matching AutoDDG), the same data access, and the same prompt-tuning budget, tuned on a held-out split. Beating it means higher recall on false claims at equal or lower false-alarm rate on true claims, with the gap concentrated on claims a generic profile does not describe, such as deduplication and PII.

No reported result found on Lacuna for claim-verification accuracy of any method. AutoDDG reports only retrieval gains, up to 30% in NDCG@k (a ranking-quality score for search results) on ECIR-DDG and NTCIR-DDG, and cost figures of $0.003 per dataset. Those figures measure searchability, not truthfulness. We will report our re-implementation's numbers as the baseline.

The second baseline is sample-and-judge. An LLM receives the datasheet and a random sample of rows, with the sample size chosen to fill the same context budget. It gives a verdict per claim.

The key ablation isolates execution. We use the identical compiled check but have the LLM predict its output from the sample instead of running it. If this ablation matches the full system, executing code over the full data is not what matters.

A ceiling reference comes from checks hand-written by an expert for each claim. The ceiling tells us how much of the remaining error comes from compilation mistakes.

Why not just…?

Why not just profile the data with AutoDDG and diff against the datasheet? This is the real threat, and it is the main baseline rather than a straw man. Our expectation is that it works on claims matching the profile's generic statistics, such as row counts, value ranges, and column types. It should fail on claims about processes: "deduplicated", "PII removed", "no test-set overlap with SQuAD V2". A profile has no reason to compute near-duplicate rates or email-pattern frequencies unless something asks for them. The per-claim-type breakdown in our experiments settles this. If profile-then-compare matches us on process claims too, the contribution collapses to a report format, and we would say so.

Why not just let an LLM read the datasheet and a sample of rows? Many false claims are rare-event claims. "No PII" is violated by a small fraction of rows. A duplication rate is a property of the whole corpus. A sample that fits in context will usually miss both. The execution ablation measures exactly this gap as a function of injected violation rate. We predict sample-and-judge degrades sharply below roughly one violating row per context window, and the curve will show whether that holds.

Why not just release a synthetic copy of the proprietary data? Efficacy of Synthetic Data as a Benchmark found synthetic data preserves model rankings for intent detection (SNIPS rank correlation 0.94) but falls to 0.20 on political-entity NER. It also found self-generation bias in Mixtral-8x7B, meaning the model scored inflated results on data it had generated. A synthetic copy is also silent on the properties we care about: it cannot show whether the real data contained PII or duplicates. We will nonetheless run synthetic-copy verification as a secondary comparison on the text datasets.

Why not just have experts hand-write data validation tests? This works and serves as our ceiling. It does not scale to the thousands of datasets in a data lake, where AutoDDG's cost argument applies. It also requires the auditor to know the data's schema, which an outside academic lacks.

Proposed Experiments

We build a claim-verification benchmark with known ground truth, drawn from three sources. The first is the 2,417 data.gov tables of ECIR-DDG and the 615 tables of NTCIR-DDG, used with their original metadata. We hand-label a stratified sample of their claims as true or false, which captures naturally occurring errors of the kind AutoDDG observed.

The second is controlled corruption of those same tables and of the text datasets SST-2, CoLA, SQuAD V2, SNIPS and ATIS. Each has a written datasheet in the Datasheets for Datasets format, and we inject mismatches at swept rates. The mismatches are truncated or extended date ranges, planted synthetic emails and phone numbers, duplicated rows, shifted label balance, and mixed-language text.

The third source, from Gap 2 in our notes, tests robustness of the compiler to messy documentation. We rewrite datasheet claims with PromptBench's character-, word- and sentence-level perturbations (PromptBench library). That paper reports a 39% average drop under word-level attacks on standard tasks, and we measure whether check compilation suffers the same fragility.

The primary metrics are recall on false claims, false-alarm rate on true claims, and the rate of "undeterminable" verdicts, reported per claim type and per injected violation rate. Secondary metrics come in three groups. The first is verdict stability across five recompilations and across two hosting routes for the same model, reported as prediction intervals in the style of the reproducibility paper. The second is leakage, measured as bits released per claim, together with a membership test: can an adversary with the report tell whether a specific planted record was in the data? The third is cost per dataset against AutoDDG's $0.003.

Three results would decide the proposal. The system must beat profile-then-compare on process claims while not losing on descriptive claims. The execution ablation must fall below the full system as violations become rare; otherwise execution is not the active ingredient. Verdicts must stay stable across hosts, since a report that changes with the server would reproduce the uncertainty problem rather than fix it. If all three hold, we will publish verification reports for the benchmark datasets and a report template that industry-track papers on proprietary data could attach, so that their data claims become checkable without their data becoming public.