TL;DR
As LLM agents are increasingly tasked with complex data analytics, discovering the right data sources within massive enterprise data lakes has become a critical bottleneck. This paper introduces Metadata Reasoner, an agentic framework that intelligently navigates large-scale metadata catalogs to select a minimal yet sufficient set of tables for any given task. By combining contrastive semantic search with on-demand verification tools, it achieves an 83.16% F1-score on real-world data selection benchmarks and significantly boosts downstream Text-to-SQL accuracy.
What Problem Does This Paper Address?
Modern enterprise data lakes are massive, messy, and disorganized. They are filled with thousands of tables, horizontal partitions, staging copies, and corrupted duplicates. When building LLM-driven autonomous agents to answer business questions or orchestrate data pipelines, identifying which specific tables are needed is incredibly difficult.
Standard retrieval systems (like vector search or BM25) rank tables individually based on topic relevance, but they fail to evaluate combinations of tables—such as verifying if a valid join path exists or if the data granularity matches the query. Simply stuffing all metadata into an LLM’s long context window doesn't work either; it causes "attention dilution," where the model overlooks crucial schema constraints due to prompt bloat and high token costs.
Methods
The authors propose the Metadata Reasoner, an agentic loop that balances context window constraints with deep reasoning through a multi-stage pipeline:
- Discrimination-Oriented Metadata Construction: To prevent similar tables from blending together in vector space, a two-stage LLM process generates table descriptions that explicitly highlight differences (e.g., distinct temporal or geographic scopes) rather than generic summaries.
- Attached Metadata (The "Warm Start"): When candidate tables are retrieved, they are fed to the agent alongside pre-computed physical schemas and LLM-synthesized statistical profiles (value ranges, null ratios, and cardinality) to provide immediate context without raw data dumps.
- On-the-Fly Specialized Tools: To resolve complex multi-table constraints without saturating the context window, the agent dynamically invokes lightweight tools:
column_profiler()for granular column statistics.data_finder()to run deterministic, boolean SQL queries verifying if specific entities (like a country code) exist in a table.joinability_check()to consult a pre-computed join graph and validate relational paths.
Datasets
The framework was evaluated on two primary environments:
- KramaBench: A real-world benchmark simulating messy data lakes across six diverse domains (Astronomy, Legal, Environment, Biomedical, Wildfire, and Archeology). The Astronomy domain is particularly high-cardinality, containing nearly 1,500 candidate tables.
- Synthetically-Scaled BIRD: Based on the popular BIRD text-to-SQL dataset, the authors engineered a noisy data lake by injecting horizontal partitions (requiring unions), duplicate tables (staging/testing vs. production variants), and low-quality tables with broken foreign keys or missing data.
Key Insights and Ideas
- Context Management Trumps Context Size: Even as LLM context windows expand, intelligent partitioning of metadata into attached summaries and on-the-fly tool queries is essential to prevent models from getting distracted by irrelevant schema details.
- Contrastive Descriptions Improve Separability: Embedding tables based on how they differ from their semantic peers (group-aware prompting) drastically improves vector search precision compared to profiling tables in isolation.
- Semantic Justifications Guide Downstream Tasks: Beyond selecting tables, the Reasoner outputs a natural-language justification documenting join keys and filter predicates. This acts as a crucial semantic bridge that helps downstream Text-to-SQL generators write correct queries.
Key Results
- Superior Selection Accuracy: On KramaBench, the Metadata Reasoner achieved an average F1-score of 83.16%, outperforming standard vector search (50.77%) and a strong Agentic SQL baseline (79.87%). In the massive Astronomy domain, it more than doubled vector search performance (72.31% vs. 32.80%).
- Robustness to Data Messiness: In the noisy synthetic BIRD environment, the framework maintained an 85.5% F1-score, whereas vector search plummeted to 30.0% due to a 35.6% noise contamination rate. The Reasoner demonstrated a 99% success rate in avoiding low-quality or staging data.
- Downstream Text-to-SQL Boost: Feeding the Metadata Reasoner’s precise table selections into a downstream Text-to-SQL generator improved final SQL execution accuracy to 71.28%—a 14.90 percentage point improvement over standard top-10 vector retrieval.
Limitations and Future Work
The authors identified a few primary failure modes to address in future iterations:
- Data-Dependent Step Ordering: The current system operates in a single reasoning pass without executing intermediate queries. It struggles with multi-stage planning where the choice of the next table depends on the runtime data values returned by a previous step (accounting for 42.9% of failures).
- Granularity and Partition Mismatches: The agent occasionally selects summary tables instead of high-resolution source tables, or misses specific physical partition files in large union sets.
- Next Steps: Future work will focus on integrating dynamic execution feedback, explicit granularity validation, partition-enumeration heuristics, and expanding the system to handle unstructured data and public web search.
Summary updated