TL;DR
This paper introduces a trustworthy AI pipeline for ingesting messy, heterogeneous scientific datasets into relational databases. By following the rule "Agents Propose, Code Verifies, Humans Decide," the system uses LLMs to reason about data structures while employing deterministic code to catch plausible-looking mistakes, ensuring that 100% of data is either correctly loaded, quarantined with a reason, or audited.
What Problem Does This Paper Address?
Public scientific datasets (like those in the GEO archive) are notoriously "messy." They often arrive as a jumble of spreadsheets, wide matrices, and cohort files with inconsistent encodings and overlapping keys. Manually cleaning and connecting these tables is a major bottleneck for researchers.
While LLM agents can automate this process by writing and executing loading code, they suffer from a dangerous flaw: silent failure. An agent might generate code that runs without error but produces a scientifically incorrect database—for example, by accidentally merging different patients who happen to share the same ID number across different study cohorts. Without a rigorous audit trail, these errors propagate into downstream analysis unnoticed.
Methods
The authors propose a modular architecture that separates reasoning from execution across three "planes":
- Reasoning Plane: Six specialized LLM agents (like the "Architect" and "Critic") perform narrow tasks. Crucially, they never see the full raw data or write code. Instead, they receive "evidence cards"—compact summaries of file shapes, headers, and statistical profiles—and return typed proposals (Pydantic objects).
- Tool Plane: Deterministic code handles the heavy lifting. A Verifier tests agent proposals against 39 different check types (e.g., checking for primary key uniqueness or join coverage). If a check fails, the system enters a repair loop.
- Control Plane: A state machine manages the workflow through eight stages, including two mandatory human checkpoints (CP1 for schema approval, CP2 for data load approval).
The system utilizes two specific "propose-verify" loops. Loop 1 focuses on structural integrity (keys and joins), while Loop 2 refines the meaning of the data (e.g., recoding "0/1" flags to Booleans).
Datasets
The pipeline was evaluated on four public biomedical datasets that combine "omics" measurements with clinical metadata:
- GSE22203: Mouse expression series (wide matrix).
- COVIDome: Four long measurement tables for 105 samples.
- GSE73072: A complex respiratory-virus study with seven cohort files where subject IDs restart at 1 in each file.
- breast.TCGA: A held-out dataset used to test the system's ability to handle unseen data shapes.
Key Insights and Ideas
- Evidence-Based Reasoning: By giving agents compact "evidence" rather than raw data, the system prevents the LLM from hallucinating based on data values and keeps token costs extremely low.
- The "Fail Loud" Principle: If a row cannot be placed or a value fails a type check, the system doesn't guess. It quarantines the row with a specific reason (e.g.,
duplicate_grainorunparseable_value), ensuring no data is lost or silently corrupted. - Deterministic Gatekeeping: The 39 verifier checks act as a "contract" between the AI's reasoning and the database's execution. This ensures that even if an agent proposes a plausible-but-wrong schema, the code will reject it before a single row is written.
- Versatility: The same design successfully generated different schema shapes (star schemas vs. multiple fact tables) based on the inherent structure of the input files without needing dataset-specific code.
Key Results
- Error Detection: In 20 runs across the datasets, 9 of the first proposals failed a verifier check. Every single failure was caught. The system repaired 8 of these automatically through its loops, and the 9th was resolved by a simple one-line human edit.
- Data Integrity: The system loaded 45.1 million values with a zero reconciliation gap.
- Robustness: In a test with nine planted data errors (like duplicated rows or mismatched types), the system successfully quarantined or audited every single one.
- Efficiency: Discovering a dataset's structure typically took only 4–7 LLM calls, costing under $0.50 per dataset in API fees.
Limitations and Future Work
- Human-in-the-Loop: The system still requires a human reviewer at two checkpoints, which may limit pure end-to-end automation at massive scales.
- Scale of Metadata: While the system handles large data values efficiently, datasets with thousands of named columns (metadata) can enlarge the LLM prompt, potentially increasing costs or hitting context limits.
- Future Directions: The authors plan to apply the pipeline to other scientific domains beyond biomedicine to prove its generalizability to any field dealing with heterogeneous tabular data.
Summary updated