TL;DR
This paper introduces SANA, a novel ablation framework for diagnosing failures in Exploratory Question Answering (EQA) agents operating over massive data lakes. By idealizing key components like search, planning, and data analysis, SANA pinpoints whether an agent's shortcomings lie in its core capabilities or its decision-making "action policy," offering a more granular understanding of EQA agent performance.
What Problem Does This Paper Address?
Answering complex questions over vast, distributed data lakes (Exploratory Question Answering or EQA) is a challenging task for AI agents. These agents need to not only search for relevant data but also plan their steps, analyze the retrieved information, and make intelligent decisions about what to do next. The problem is that simply measuring end-to-end accuracy doesn't tell us why an agent failed. Was it a bad search query, a flawed plan, an incorrect data analysis, or a poor decision by the agent's "action policy"? Existing benchmarks don't effectively disentangle these failure modes, making it hard to systematically improve EQA agents.
Methods
The core contribution is the SANA (Search Agent Navigation Ablation) framework. SANA tackles this problem by:
- Creating Runtime Profiles: For solved EQA tasks, SANA generates detailed profiles containing the sequence of correct data sources, sanitized subquestions (to avoid giving away answers), and execution records (specifying the dataset, query, intent, and answer for data analysis steps).
- Constructing Idealized Components: Using these profiles, SANA creates "ideal" versions of the search, planning, and data analysis tools. These idealized tools act as oracles, providing perfect outputs for their respective functions.
- Performing Ablation Studies: SANA then systematically replaces the agent's actual components with these idealized versions. By observing how performance changes when a specific component is idealized, researchers can isolate its contribution and identify bottlenecks. For example, if idealizing the search component significantly boosts accuracy, it indicates that search was a major limitation. If accuracy remains low even with idealized search, planning, and data analysis, the failure likely lies within the agent's action policy.
Datasets
The authors evaluated SANA on two existing EQA benchmarks:
- LakeQA: A benchmark specifically designed for exploratory QA over large data lakes.
- KramaBench: Adapted for this study by exposing its data as a collection of sources and converting its monolithic solution scripts into sequential, dependent hops to align with the EQA task structure.
Key Insights and Ideas
- Data Analysis is Consistently Hard: Across both benchmarks and agent sizes, data analysis emerged as a significant bottleneck. This suggests that even when agents find the right data, they struggle to correctly extract information or perform the necessary computations.
- Search is Crucial for Large Data Lakes: In the LakeQA benchmark, which features a massive data lake, search was a major limitation. This highlights the difficulty of navigating and retrieving relevant information from extremely large and diverse datasets.
- Planning is Less of a Bottleneck (but Following Plans is): While idealizing planning showed some gains, the impact was less pronounced than for search or data analysis. The authors found that agents can often create reasonable plans, but they struggle to reliably follow them throughout the EQA trajectory. This points to a need for better runtime policies that help agents stay on track.
- Action Policy Matters: Even with idealized search, planning, and data analysis, a performance gap often remains. This "residual gap" is attributed to the agent's action policy – its core decision-making logic. This highlights that simply improving individual tools isn't enough; the agent's ability to orchestrate them effectively is critical.
- Agent Size Matters: The study used two models, gpt-5.4-nano (weaker) and gpt-5-mini (stronger). The weaker agent benefited more from idealizations, particularly in search, indicating it struggles more with fundamental navigation and retrieval. The stronger agent, while still benefiting, showed that its failures were more concentrated in execution and finalization.
Key Results
- Significant Accuracy Gains: Idealizing components led to substantial improvements in semantic match (SM) scores. For instance, on LakeQA, idealizing all components boosted gpt-5.4-nano's SM from 20.7% to 37.0% and gpt-5-mini from 56.3% to 76.3%.
- Data Analysis Bottleneck: Idealizing data analysis provided significant gains, especially for the stronger gpt-5-mini agent (+18.5% on LakeQA), suggesting it often has the right intent but fails in implementation.
- Search Impact Varies: Ideal search improved performance significantly on LakeQA (+14.1% for gpt-5.4-nano), but had a smaller impact on the converted KramaBench, where data analysis was the primary bottleneck.
- Planning Follow-Through is Key: Audit results showed that while agents could generate similar plans to ideal ones, they often failed to follow them, with only 19.0% of gpt-5.4-nano's baseline trajectories being "mostly followed."
Limitations and Future Work
- Diagnostic Approximations: The "ideal" tools are approximations. Ideal search is restricted to known gold sources, and ideal data analysis relies on semantic matching, meaning they might not capture all valid scenarios.
- Task-Suite Specificity: The conclusions are based on the specific LakeQA and converted KramaBench datasets. The findings might not generalize universally to all EQA benchmarks.
- LLM Judges: The evaluation relies on LLM judges for semantic matching, plan assessment, and failure auditing, which can introduce biases or errors.
Future work should focus on developing more robust runtime policies that can:
- Track subgoal progress effectively.
- Commit to validated sources.
- Recover from trajectory drifts.
- Verify intermediate evidence.
- Perform thorough final answer checks before stopping.
Summary updated