TL;DR
EvoOntology bridges the "agent-data gap" by introducing an interactive, self-evolving ontology layer served via the Model Context Protocol (MCP). By autonomously building and continuously refining domain knowledge from agent interaction histories, it allows LLM data agents to navigate complex, heterogeneous data sources with significantly higher accuracy and token efficiency.
What Problem Does This Paper Address?
When Large Language Model (LLM) data agents interact with heterogeneous data sources—such as SQL databases, CSV files, and unstructured documents—they face a fundamental agent-data gap. Because the agent does not know the data structure or content a priori, it is forced to blindly issue exploratory queries, guess where specific concepts live, and wade through irrelevant content.
Existing solutions fall short: directly exploring raw data scales poorly and quickly exhausts context windows, while manually creating "semantic layers" to inject into prompts is costly, inflexible, and often distracts the LLM with irrelevant metadata. The authors address the need for a scalable, interactive intermediate layer that explicitly maps domain concepts to physical data and dynamically adapts to specific agent behaviors.
Methods
EvoOntology introduces an agent-first builder-and-evolver framework structured into three distinct layers:
- Content Layer: A typed semantic graph containing Terms (domain concepts), Mappings (links to physical data fields), Constraints (business rules), and Evidence (data observations).
- Schema Layer: Defines the structural rules and permitted relationship types for the content layer.
- Tool Layer: Exposes the ontology to the data agent via Model Context Protocol (MCP) tools, allowing the agent to actively query the ontology at runtime (using
fbrowseto search terms andfresolveto retrieve detailed mappings).
The lifecycle of EvoOntology consists of two main phases:
- Evidence-Grounded Initialization: A builder agent extracts candidate concepts from a training workload and runs executable probes against raw data sources to automatically construct a verified initial ontology.
- Trajectory-Grounded Evolution: An evolution agent analyzes historical interaction trajectories to diagnose failures. It attributes errors to specific layers, proposes localized patches (edits), and passes them through a Backbone-Conditional Paired Validation Gate to ensure updates genuinely improve performance before deployment.
Datasets
The framework was evaluated using four LLM backbones across three diverse data-agent benchmarks:
- Deep Data Research (DDR-Bench): Evaluated on the 10-K financial scenario, requiring open-ended data research and synthesis across highly heterogeneous sources.
- InsightBench: A business-analytics benchmark requiring agents to surface insights and summaries from CSV datasets.
- BIRD: A large-scale text-to-SQL benchmark evaluated under the Oracle Knowledge setting to measure query correctness and execution efficiency.
Key Insights and Ideas
- On-Demand Tooling Beats Prompt Injection: Dumping an entire semantic layer into an LLM prompt causes instruction distraction and context bloat. Serving the ontology as an interactive MCP tool allows the agent to selectively query only what it needs for the current execution step.
- Ontology Evolution is LLM-Specific: Different LLM backbones (e.g., GPT vs. Claude) evolve distinct ontologies with relatively low overlap (pairwise Jaccard similarity < 0.62). Cross-backbone transfer experiments show that an ontology evolved for one LLM performs worse when deployed on another, proving that semantic layers must adapt to specific model behaviors.
- Self-Evolution Enhances Token Efficiency: Rather than increasing computational overhead, a well-evolved ontology shortens the agent's trajectory by reducing repeated schema discovery. This optimization reduces total token consumption per task by approximately 20% compared to a baseline agent without an ontology layer.
Key Results
- DDR-Bench: EvoOntology achieved a massive average gain of +17.8 points in Trajectory-Wise accuracy over the baseline ReAct agent. In contrast, traditional static semantic layers often degraded performance due to context clutter.
- BIRD (Text-to-SQL): Delivered consistent improvements across all backbones, yielding an average execution accuracy (EX) boost of 7.4 points and a Valid Efficiency Score (VES) increase of 8.6 points.
- InsightBench: Improved both insight generation and summary quality, showing a mean overall gain of 1.9 points (and up to +6.1 points on DeepSeek-V4-Flash).
- Ablation Studies: Confirmed that the validation gate is the most critical element of the evolution loop; removing it caused an 11.2-point drop in accuracy due to unfiltered regressions.
Limitations and Future Work
A dedicated limitations section is not explicitly discussed in the paper. However, the authors' analysis implies a few boundaries:
- Workload Saturation: The evolution curve flattens out after a few rounds, suggesting the framework stabilizes once it covers the recurrent concepts of a specific workload, but may require re-triggering if the user query distribution shifts drastically.
- Dependency on Probing: The initialization phase relies heavily on successful execution of database probes; highly messy, unindexed, or restricted raw data sources might hinder the builder agent's initial grounding capability.
Summary updated