TL;DR
The Pneuma Project introduces PNEUMA-SEEKER, a system that helps users articulate and fulfill complex data information needs by reifying them as relational schemas. This approach automates data discovery and preparation by iteratively aligning a dynamic data model with user intent, bridging the gap between vague human queries and actionable data insights.
What Problem Does This Paper Address?
Data discovery and preparation are persistent bottlenecks in the data management lifecycle, especially when users have vague, evolving, or hard-to-operationalize information needs. While Large Language Models (LLMs) are increasingly capable, the challenge shifts from executing tasks to helping users precisely articulate their goals. This paper tackles the problem of how to effectively translate these often implicit and dynamic user intents into concrete data models and queries that can guide automated data retrieval and preparation.
Methods
PNEUMA-SEEKER employs a novel, modular architecture built on three key principles:
- Context Specialization: The system decomposes the data workflow into distinct subtasks (retrieval, integration, orchestration), with each component operating on a specialized, narrow context. This reduces LLM hallucination and manages context window limitations.
- Dynamic Planning with Conductor: Instead of static pipelines, a central LLM-powered agent, CONDUCTOR, dynamically assembles execution plans based on real-time progress and user feedback. This allows for flexible adaptation to evolving information needs.
- Shared State for Convergence: The user's active information need is represented as a relational data model (T, Q), where T is a set of tables and Q is a sequence of SQL queries. This model serves as a shared state between the user and the system, allowing for iterative refinement and convergence towards the latent information need.
The system integrates advances in Retrieval-Augmented Generation (RAG), agentic frameworks, and structured data preparation. Key components include:
- CONDUCTOR: Orchestrates the workflow, reasons internally, calls tools, modifies the state (T, Q), and communicates with the user.
- IR SYSTEM: Abstracts heterogeneous retrieval formats into document objects, supporting table discovery (PNEUMA-RETRIEVER), domain knowledge storage (Document Database), and external information lookup (Web Search).
- MATERIALIZER: Populates the target data model (T) with data, performing integration and transformation using tools like a SQL Executor and Python interpreter.
Datasets
The evaluation uses two datasets from KramaBench [14]: Archaeology (12 questions, 5 tables) and Environment (20 questions, 36 tables). Web Search was disabled during evaluation to prevent leaking benchmark information. The paper does not introduce new benchmarks but uses existing ones for evaluation.
Key Insights and Ideas
- Reifying Information Needs as Schemas: The core idea is that an information need can be represented as a relational schema (T, Q). This provides a concrete, actionable target for the system and a clear object for user feedback, moving beyond purely natural language interactions.
- Dynamic Orchestration is Crucial: Static, rule-based pipelines are brittle and difficult to adapt to new use cases or evolving user intents. A dynamic, state-driven orchestrator like CONDUCTOR is essential for handling the iterative and often unpredictable nature of data discovery.
- Context Specialization Manages LLM Limitations: Breaking down complex tasks into smaller, context-specific subtasks is vital for LLM performance, especially when dealing with structured data and large, heterogeneous contexts. This also helps manage computational costs and prevent hallucinations.
- Emergent Documentation and Organizational Memory: PNEUMA-SEEKER's design inherently captures organizational knowledge by prompting users to articulate their goals and assumptions. This can build an emergent documentation layer, preserving "tribal knowledge" and strengthening institutional memory.
Key Results
- Convergence: PNEUMA-SEEKER consistently achieved the highest percentage of convergence (user's active information need matching the latent one) compared to baselines like FTS, PNEUMA-RETRIEVER, and LLAMAINDEX.
- Efficiency: Median turns to convergence were comparable to LLAMAINDEX, indicating efficient interaction.
- Accuracy: PNEUMA-SEEKER significantly outperformed baselines like LLAMAINDEX and DS-GURU in answering questions accurately, achieving up to 55% accuracy on the Environment dataset, while LLAMAINDEX achieved 0%. This highlights the importance of actual computation beyond just interpreting retrieved context.
- Cost: The paper provides estimated token usage and costs for different LLMs, showing that PNEUMA-SEEKER can be cost-effective even with smaller models.
- Qualitative Findings: The system successfully helps users articulate latent intents, guides discovery, and produces fit-for-purpose documents.
Limitations and Future Work
- LLM SIM Limitations: The evaluation relies on LLM SIM for user simulation, which is not a perfect substitute for real human studies.
- Latency: PNEUMA-SEEKER has a response latency of around 70.26 seconds, compared to near-instantaneous responses from FTS and PNEUMA-RETRIEVER.
- Dynamic Pipeline Limitations: While dynamic pipelines are more adaptable, they still fail if a necessary capability (e.g., semantic joins) is missing.
- Future Work: The authors plan to explore more advanced interaction models, such as incremental feedback from CONDUCTOR to MATERIALIZER. They also emphasize the importance of further user studies to understand how users interpret and act on the evolving (T, Q) state. The long-term vision includes building "internal data markets" by leveraging the captured tribal knowledge.
Summary updated