An Agentic Approach to Metadata Reasoning

Jiani Zhang, Sercan O. Arik, Cosmin Arad, Fatma Ozcan, Alon HalevyarXiv 2026 · arXiv

TL;DR

As LLM agents are increasingly tasked with complex data analytics, discovering the right data sources within massive enterprise data lakes has become a critical bottleneck. This paper introduces Metadata Reasoner, an agentic framework that intelligently navigates large-scale metadata catalogs to select a minimal yet sufficient set of tables for any given task. By combining contrastive semantic search with on-demand verification tools, it achieves an 83.16% F1-score on real-world data selection benchmarks and significantly boosts downstream Text-to-SQL accuracy.

What Problem Does This Paper Address?

Modern enterprise data lakes are massive, messy, and disorganized. They are filled with thousands of tables, horizontal partitions, staging copies, and corrupted duplicates. When building LLM-driven autonomous agents to answer business questions or orchestrate data pipelines, identifying which specific tables are needed is incredibly difficult.

Standard retrieval systems (like vector search or BM25) rank tables individually based on topic relevance, but they fail to evaluate combinations of tables—such as verifying if a valid join path exists or if the data granularity matches the query. Simply stuffing all metadata into an LLM’s long context window doesn't work either; it causes "attention dilution," where the model overlooks crucial schema constraints due to prompt bloat and high token costs.

Methods

The authors propose the Metadata Reasoner, an agentic loop that balances context window constraints with deep reasoning through a multi-stage pipeline:

  1. Discrimination-Oriented Metadata Construction: To prevent similar tables from blending together in vector space, a two-stage LLM process generates table descriptions that explicitly highlight differences (e.g., distinct temporal or geographic scopes) rather than generic summaries.
  2. Attached Metadata (The "Warm Start"): When candidate tables are retrieved, they are fed to the agent alongside pre-computed physical schemas and LLM-synthesized statistical profiles (value ranges, null ratios, and cardinality) to provide immediate context without raw data dumps.
  3. On-the-Fly Specialized Tools: To resolve complex multi-table constraints without saturating the context window, the agent dynamically invokes lightweight tools:
  4. column_profiler() for granular column statistics.
  5. data_finder() to run deterministic, boolean SQL queries verifying if specific entities (like a country code) exist in a table.
  6. joinability_check() to consult a pre-computed join graph and validate relational paths.
Figure 1: The architecture of the Metadata Reasoner. Given an analytic task and a metadata catalog, the agent autonomously executes a series of planning, search, and rea- soning steps. It leverages both attached metadata from re- trieved candidate tables and on-the-fly metadata accessed via specialized tools to output a sufficient and minimal set of tables, along with a natural-language justification.

Datasets

The framework was evaluated on two primary environments:

  • KramaBench: A real-world benchmark simulating messy data lakes across six diverse domains (Astronomy, Legal, Environment, Biomedical, Wildfire, and Archeology). The Astronomy domain is particularly high-cardinality, containing nearly 1,500 candidate tables.
  • Synthetically-Scaled BIRD: Based on the popular BIRD text-to-SQL dataset, the authors engineered a noisy data lake by injecting horizontal partitions (requiring unions), duplicate tables (staging/testing vs. production variants), and low-quality tables with broken foreign keys or missing data.

Key Insights and Ideas

  • Context Management Trumps Context Size: Even as LLM context windows expand, intelligent partitioning of metadata into attached summaries and on-the-fly tool queries is essential to prevent models from getting distracted by irrelevant schema details.
  • Contrastive Descriptions Improve Separability: Embedding tables based on how they differ from their semantic peers (group-aware prompting) drastically improves vector search precision compared to profiling tables in isolation.
  • Semantic Justifications Guide Downstream Tasks: Beyond selecting tables, the Reasoner outputs a natural-language justification documenting join keys and filter predicates. This acts as a crucial semantic bridge that helps downstream Text-to-SQL generators write correct queries.

Key Results

  • Superior Selection Accuracy: On KramaBench, the Metadata Reasoner achieved an average F1-score of 83.16%, outperforming standard vector search (50.77%) and a strong Agentic SQL baseline (79.87%). In the massive Astronomy domain, it more than doubled vector search performance (72.31% vs. 32.80%).
  • Robustness to Data Messiness: In the noisy synthetic BIRD environment, the framework maintained an 85.5% F1-score, whereas vector search plummeted to 30.0% due to a 35.6% noise contamination rate. The Reasoner demonstrated a 99% success rate in avoiding low-quality or staging data.
Distribution of selected table types showing noise robustness
  • Downstream Text-to-SQL Boost: Feeding the Metadata Reasoner’s precise table selections into a downstream Text-to-SQL generator improved final SQL execution accuracy to 71.28%—a 14.90 percentage point improvement over standard top-10 vector retrieval.

Limitations and Future Work

The authors identified a few primary failure modes to address in future iterations:

  • Data-Dependent Step Ordering: The current system operates in a single reasoning pass without executing intermediate queries. It struggles with multi-stage planning where the choice of the next table depends on the runtime data values returned by a previous step (accounting for 42.9% of failures).
  • Granularity and Partition Mismatches: The agent occasionally selects summary tables instead of high-resolution source tables, or misses specific physical partition files in large union sets.
  • Next Steps: Future work will focus on integrating dynamic execution feedback, explicit granularity validation, partition-enumeration heuristics, and expanding the system to handle unstructured data and public web search.

Summary updated

Abstract

As LLM-driven autonomous agents evolve to perform complex, multi-step tasks that require integrating multiple datasets, the problem of discovering relevant data sources becomes a key bottleneck. Beyond the challenge posed by the sheer volume of available data sources, data-source selection is difficult because the semantics of data are extremely nuanced and require considering many aspects of the data. To address this, we introduce the Metadata Reasoner, an agentic approach to metadata reasoning, designed to identify a small set of data sources that are both sufficient and minimal for a given analytical task. The Metadata Reasoner leverages a table-search engine to retrieve candidate tables, and then autonomously consults various aspects of the available metadata to determine whether the candidates fit the requirements of the task. We demonstrate the effectiveness of the Metadata Reasoner through a series of empirical studies. Evaluated on the real-world KramaBench datasets for data selection, our approach achieves an average F1-score of 83.16%, outperforming the strongest baseline by 3.3 percentage points and weaker baselines by 38 percentage points. Furthermore, evaluations on a newly created synthetic benchmark based on the BIRD data source reveal that the Metadata Reasoner is highly robust against redundant and low-quality tables that may be in the data lake. In this noisy environment, it maintains an average of 85.5% F1-score for selecting the right datasets and demonstrates a 99% success rate in avoiding low-quality data.
BibTeX
@misc{zhang2026agentic,
  author = {Jiani Zhang and Sercan O. Arik and Cosmin Arad and Fatma Ozcan and Alon Halevy},
  title = {{An Agentic Approach to Metadata Reasoning}},
  year = {2026},
  eprint = {2604.20144},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2604.20144},
}

Loading related content...

Figure