Finding What a Dataset Paper Forgot to Say, by Analogy to Papers That Said It

When a paper describes its data pipeline, the details that end up missing are rarely random. They tend to be the ones the authors found too routine to write down. One example is the threshold used to separate tissue from background in a slide image. Another is whether train and test were split by patient or by image patch. When a group in digital pathology tried to rebuild three winning Camelyon challenge methods from their papers alone, omissions of exactly this kind sank the results. One method's slide-level accuracy fell from 92% to 54.7%, barely above chance, as reported in Reproducibility of deep learning in digital pathology whole slide image analysis. Automated documentation tools cannot catch this. Auto-BenchmarkCard builds benchmark documentation from papers and code, then checks each statement against the sources. Its authors name the unsolved half of the problem the "comprehensiveness gap": a fact-checker can confirm what was written but cannot notice what was never written.

This proposal treats an under-documented pipeline as an incomplete target domain in an analogy. Classical analogy software assumes both sides of an analogy already exist and only aligns them. Analogy-as-generation work instead builds the missing side. We use the same move to build the missing parts of a pipeline description. First, we turn each pipeline description into a graph of steps, artifacts, and parameters. Next, we align the target graph with several well-documented pipelines that have a similar structure. Then we carry over the relations that the documented pipelines have and the target lacks. Each relation that carries over becomes a specific, located question for the authors, such as "your patch-labeling step consumes slide-level masks, but no overlap threshold is reported." It is never a guessed answer. The bet is that structure is what tells you which missing detail matters. The test of that bet is whether this method beats a frequency checklist mined from the same corpus on omissions that only matter because of how a particular pipeline is built.

Core Claim

Structural alignment between a target pipeline and documented analogues can detect omissions that depend on the pipeline's own design, and it does so better than flat checklists or unstructured LLM prompting. The machinery comes from structure-mapping theory, Gentner's account of analogy. When two relational structures are aligned, any relation in the base that has no counterpart in the target is a "candidate inference," something the analogy predicts should also hold in the target. The Structure Mapping Engine (SME), a standard implementation of that theory, computes these. The semantic-network evolutionary algorithms in A semantic network-based evolutionary algorithm for computational creativity. and Automated Generation of Cross-Domain Analogies via Evolutionary Computation used SME scores as the fitness that drove the creation of new conceptual networks. They checked validity against ConceptNet and WordNet. We reuse that pairing of structural fitness and validity constraint for a different purpose. Those papers generated whole new domains for creativity. We generate only the missing part of a target that is fixed and partly known. A completed slot is accepted only if several independent documented pipelines agree on it.

The claim is falsifiable in a sharp way. If the advantage over a corpus-frequency checklist does not appear on design-dependent omissions in particular, the structural ingredient is not earning its cost.

Research Program

The first component turns pipeline text into graphs. We parse papers with Docling, as Auto-BenchmarkCard does, and dataset cards from the Hugging Face corpus studied in Navigating Dataset Documentations in AI. An LLM then extracts a typed graph from each document. Nodes are data artifacts (raw slides, masks, patches), operations (normalize, segment, label, deduplicate, split), and parameters. Edges are relations such as consumes, produces, parameterized-by, and split-by. The relation vocabulary is kept small and fixed so that alignment is meaningful. We estimate extraction error separately by hand-checking a sample. Every later result could be an artifact of bad graphs, so this is reported first.

The second component retrieves analogues and completes the target. For a target graph we retrieve $k$ documented base graphs using structural features, meaning operation types and how they are connected, rather than topic words. This choice is deliberate, because it tests whether pipeline shape transfers across subject areas. Each base is aligned to the target with SME, which yields candidate inferences. We then search over completed target graphs. Their fitness is the summed structural alignment with all $k$ bases, minus a penalty for nodes or relations that violate type constraints. In the spirit of the commonsense mutations used by the semantic-network evolutionary algorithm, we use an operation-ontology check rather than ConceptNet. One example of a violation is a threshold attached to an operation that takes no threshold. The search may combine slots from different bases, which is where it is truly generative: the completed target can contain a configuration that no single base has. Each slot that survives is turned into a question attached to a specific pipeline step. As a final filter, a fact-checker, FactReasoner from the Auto-BenchmarkCard stack, confirms that the slot is not already stated in the target's text, which removes false alarms caused by extraction misses.

We do not fill in values. The output is a list of located, prioritized questions. This answers the objection from Datasheets for datasets that documentation should come from careful human reflection. The system asks sharper questions, and the human still answers them. The evidence from the Hugging Face card study is why sharper questions are needed: the "Considerations for Using the Data" section makes up only 2.1% of card text, and only 7.9% of zero-download datasets complete all sections. The generic questions are already available, and people skip them.

Baselines

The naive reference point is the Datasheets questionnaire itself. Its roughly fifty fixed questions are matched against the target text, and every question with no answer is flagged. It needs no learning and represents what a careful reviewer with a template would do. Beating it is required, and it should be easy at the level of specific parameters, because none of its questions name a stain-normalization template.

The strongest non-LLM competitor is a corpus-frequency checklist. Using the same documented corpus our method retrieves from, we mine every parameter or relation type mentioned by more than a set fraction of pipelines in the target's field. We flag those the target lacks, ranked by how often they appear. It uses exactly the same data, it has a single threshold tuned on the same validation split we use, and it is the alternative most likely to match us. Beating it means higher recall at a fixed number of flags on the design-dependent stratum defined below, by a margin larger than the bootstrap confidence interval. Parity on the common stratum is acceptable and expected.

The strongest LLM competitor is retrieval-augmented prompting. The same LLM receives the target text plus the same $k$ retrieved base documents as raw text and is asked to list unreported details with their pipeline step. We give it the same token budget as our method's total LLM use, including extraction, and tune its prompt on the validation split with a comparable number of trials. Auto-BenchmarkCard's composition stage serves as a second variant: we read the blank fields it leaves as its flags. We found no reported result on Lacuna for omission detection by Auto-BenchmarkCard or any other system. The paper reports only entailment scores for statements it did produce. The nearest reported evidence that forcing explicit structure helps LLMs is from NOVA-Test. In that study, requiring a variable-level mapping table raised plausibly novel hypotheses from 3 of 12 under literature-search prompting to 10 of 12 (p = 0.006). That was a different task, so it motivates our design and does not serve as a baseline number.

Two ablations isolate the key ingredient. In the first, a flat-diff version keeps the same retrieved bases and the same extracted graphs but replaces alignment with a set difference of parameter names, which removes structure while holding the information fixed. In the second, single-base completion ($k=1$) tests whether agreement across several analogues is what controls false alarms.

Why not just…?

Why not just prompt a strong LLM with the retrieved papers and ask what is missing? This is a real threat, and the experiment is built to settle it. The concern is that an LLM reading ten pathology papers will notice that the target never mentions stain normalization. Our expectation that it falls short comes from two observations. Auto-BenchmarkCard's LLM composition leaves fields blank when the sources are silent instead of inferring that something is missing. NOVA-Test reports that LLMs anchor locally and are swayed by fluent text unless forced into explicit mappings. The equal-budget comparison in this proposal decides the question. If retrieval-augmented prompting matches us on the design-dependent stratum, the contribution shrinks to an interpretable, step-located output format, and we will report it that way.

Why not just mine a frequency checklist from well-documented pipelines? This is the alternative we consider most likely to work equally well, and we changed the evaluation because of it. On omissions that nearly every pipeline in a field must report, such as split strategy or image resolution, a frequency list will catch them as well as alignment does. We therefore split the gold omissions into common ones and design-dependent ones. A design-dependent omission is relevant only because of the target's particular structure, for example a patch-overlap threshold that exists only in pipelines that label patches from slide masks. The whole claim rests on the second stratum. If frequency matches us there, the structural method is not worth its complexity.

Why not just retrieve the single most similar documented paper and copy its sections? Nearest-neighbor retrieval is the classical stand-in for analogy. The morphology results in Solving Morphological Analogies Through Generation and the argument in Analogical Proportions: Why They Are Useful in AI show that retrieval fails exactly when the needed target configuration is absent from the store. In our setting that happens when no single base pipeline has the target's combination of steps. The $k=1$ ablation is this baseline in structural form, and the flat-diff ablation is it in unstructured form. Our method should beat both most clearly on hybrid pipelines.

Why not just require authors to fill in datasheets? Mandates are the honest first answer, and our output is meant to feed them rather than replace them. The Hugging Face audit shows that voluntary practice in the wild leaves the risk and preprocessing content thin. The Camelyon audit shows that even prize-winning, high-scrutiny papers omitted the details that decided whether they could be reproduced. A template does not know which detail a given pipeline needs.

Proposed Experiments

The first benchmark uses controlled deletion. We take fully documented pipelines, namely dataset cards that complete all recommended sections from the 7,433-card Hugging Face corpus, together with the Camelyon and WILDS method papers. We remove specific parameter statements and test whether each system recovers them. To keep the detection from riding on textual seams, deletions are made at the clause level and the text is then smoothed by an LLM. To limit memorization by the LLM baselines, we favor cards and papers released after the model's training cutoff and report results separately for older and newer documents. Two annotators label each deleted item as common or design-dependent before any system is run. The metrics are recall at a fixed number of flags $k$, precision judged by annotators, and localization accuracy, meaning whether the flag is attached to the correct pipeline step.

The second benchmark consists of real omissions. The omitted details identified in the Camelyon reproducibility audit, covering stain templates, segmentation thresholds, patch-labeling criteria, and patient-level split boundaries for the Wang, Lee, and Liu methods, form a small but authentic gold set. Here we report recall only, since the audit lists what was missing. Retrieval is restricted to documents that exclude those three papers and their direct follow-ups.

The decisive check asks whether the flags matter. We re-implement the Wang pipeline on Camelyon16 from its paper. The audit reports slide-level AUC falling from 0.925 to 0.891 and patch accuracy from 98.4% to 80.5% on reimplementation. We set aside the parameters the paper leaves unspecified. Some of these are flagged only by our method, some only by the strongest baseline, and some by neither. Each set is varied over a small grid of plausible values, under matched compute per set. If structural flags pick out the omissions that cause failures, then varying our flagged parameters should account for more of the AUC variance than varying the baseline's flags or the unflagged ones. If it does not, the method finds omissions that are real but unimportant, and we would report that as a negative result about the value of completeness detection.

We also apply the method to the Camelyon17-WILDS setting, where accuracy under empirical risk minimization drops from about 93% in-distribution to about 70% across hospitals in WILDS. The question there is whether flagged acquisition and preprocessing details are the ones that describe hospital differences. This would connect the work to the broader claim that robustness evaluation needs metadata at the level of each domain, or site, that is currently not reported.