Same Context, Different Agents: A Behavioral Study of Semantic Layers, Orchestration and Compaction in Data Agents

Zhivar Sourati, Marianne Menglin Liu, Zheng Wang, Rongguang Wang, Sai Ashish Somayajula, Tao Sheng, Kyu J. Han, Sujith Ravi, Dan RotharXiv 2026 · arXiv

TL;DR

LLM-based data agents fail in predictable ways, falling into three behavioral archetypes: Investigators, Premature Answerers, and Early Stoppers. This study reveals that popular "context-engineering" fixes—like semantic layers or orchestration—only improve performance when they specifically target an agent's behavioral flaws; for high-performing agents, these same techniques can actually degrade accuracy and inflate costs.

What Problem Does This Paper Address?

Building autonomous agents that can analyze heterogeneous data (SQL databases, messy CSVs, web pages, and PDFs) is a major goal for the LLM community. However, these agents are notoriously unreliable. They often "hallucinate" answers without looking at the data, give up on complex tasks, or get overwhelmed by long context windows.

To fix this, researchers use Context-Engineering Techniques (CETs). The three most common are:

  1. Semantic Layers: Providing a curated "map" or description of the data.
  2. Orchestration: Breaking a task into sub-tasks handled by specialized agents (e.g., a SQL specialist and a web specialist).
  3. Compaction: Summarizing or truncating the interaction history to keep the prompt manageable.

While these are often treated as silver bullets, this paper argues that we don't actually know how they interact with different LLMs or if they are always beneficial.

Methods

The authors conducted a massive behavioral study, testing seven frontier LLMs (including versions of GPT, Claude, Gemini, and GLM) across 1,514 tasks. They evaluated every possible combination of the three CETs (8 configurations total) to see how they affected accuracy, cost, and agent behavior.

Crucially, they categorized agents into three groups based on their baseline behavior:

  • Investigators: Models that naturally use tools, persist through errors, and ground their answers in data (e.g., Claude Opus 5.5, GPT-6 Astra).
  • Premature Answerers: Models that tend to answer questions immediately from internal knowledge without using provided data tools (e.g., GPT-5.5).
  • Early Stoppers: Models that act but frequently give up, claiming "no data found," or get stuck in repetitive loops (e.g., Gemini 2.5 Flash).

Datasets

The study utilized five diverse benchmarks totaling 1,514 tasks:

  • KramaBench: Data lakes with irregular layouts.
  • DataClaw: Noisy enterprise tables and policy texts in multiple languages.
  • FDABench (Report & Choice): Chaining evidence across databases, documents, and the web.
  • HybridDeepResearch (HDR): Tasks requiring the agent to discover schemas and chain clues across the web and databases.

Key Insights and Ideas

  • CETs are not "defaults": A technique that rescues a weak model can sabotage a strong one. For example, giving a Semantic Layer to an Investigator often causes them to stop looking at the actual data because they trust the summary too much, leading to a 5-point drop in accuracy.
  • Orchestration forces engagement: It is the most effective "cure" for Premature Answerers. By introducing a planner and a verifier, the model is forced to produce evidence before an answer is accepted.
  • The "Match" is everything: Orchestration helps Premature Answerers (+23 points) but hurts Investigators. Semantic layers help Early Stoppers (+4.6 points) at a fraction of the cost of orchestration.
  • Compaction is risky: While it keeps prompts small, it can cause agents to lose track of what they've already found, leading to "long loops" where the agent repeatedly fetches the same data.
  • Diminishing Returns: Combining CETs rarely results in the sum of their individual gains. They often fix the same underlying habits, so stacking them just adds cost without extra accuracy.
Figure 1: Same context, different agents (illustra- tion). Given the same question and data, an investiga- tor checks every source and answers from the data, a premature answerer answers without looking, and an early stopper gives up partway.

Key Results

  • Cost vs. Accuracy: Orchestration is the most expensive fix, often multiplying costs by 4x. For Premature Answerers, this is a worthy trade-off for the massive accuracy gain, but for other groups, it's an expensive way to lose points.
  • Behavioral Shifts: The authors used "radar charts" to visualize how CETs change agent DNA. Orchestration significantly raises "Engagement" (Acts, Coverage, Persists), while Semantic Layers raise "Reliability" (Grounded, No give-up) but often at the expense of Engagement.
Figure 3: Radar charts showing the distinct behavioral profiles of Investigators, Premature Answerers, and Early Stoppers.
  • The "Wrong Slice" Problem: Even when agents look at the data, the most common failure mode (40.5%) is selecting the wrong "slice" or field of data. Interestingly, no CET currently fixes this problem.

Limitations and Future Work

  • Model Scope: The study is limited to seven specific models. As LLMs evolve, their baseline behaviors may shift.
  • Fixed Configurations: The CETs were tested as static "on/off" switches. The authors suggest that future work should focus on adaptive agents that can detect their own failure modes and switch on the appropriate CET dynamically.
  • The Data Selection Gap: Since CETs don't help agents pick the right data slice once they are looking, new research is needed into better "data-reading" prompts and reasoning chains.

Summary updated

Abstract

Answering questions over heterogeneous data means combining evidence from files, databases, documents and the web. Data agents automate this, but their success depends on context. Three context-engineering techniques (CETs) help by shaping it: semantic layers describe the data, orchestration splits work among agents, and compaction condenses interaction history. Existing studies validate each CET alone, leaving open how agent behavior shapes its effect. We present a comprehensive analysis of the three CETs, alone and combined, with seven large language models (LLMs) on 1,514 long-horizon tasks from five benchmarks, covering behavior, accuracy and cost. We find that data agents without CETs form three behavior groups: investigators, premature answerers that answer without inspecting data, and early stoppers that give up. Each CET delivers its known benefit, from more investigation to smaller prompts, but only the behavior group it matches turns that benefit into accuracy, such as premature answerers under orchestration. For investigators, the same CETs can hurt: the semantic layer stops them from looking, orchestration makes them overconfident, costing 5 percentage points, and compaction makes them repeat steps. Combining CETs yields smaller gains and losses than their sum. We derive guidance for choosing CETs by agent behavior, task demands and accuracy-cost trade-offs.
BibTeX
@misc{sourati2026same,
  author = {Zhivar Sourati and Marianne Menglin Liu and Zheng Wang and Rongguang Wang and Sai Ashish Somayajula and Tao Sheng and Kyu J. Han and Sujith Ravi and Dan Roth},
  title = {{Same Context, Different Agents: A Behavioral Study of Semantic Layers, Orchestration and Compaction in Data Agents}},
  year = {2026},
  eprint = {2610.20959},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.20959},
}
Figure