Abstract
Knowledge-Based Conversational Question Answering (KBCQA) requires navigating complex linguistic context and strict logical constraints. We present SEAL (Self-Evolving Agentic Learning), a framework that decomposes complex queries into semantic cores and templates. SEAL incorporates a self-evolving memory and reflection module, allowing the system to improve its reasoning capabilities dynamically throughout a dialogue. Evaluated on the SPICE benchmark, SEAL achieves state-of-the-art performance, demonstrating significant improvements in structural accuracy and complex reasoning robustness without the need for explicit retraining.
Introduction: The Challenges of Conversational QA
Conversational Question Answering over Knowledge Graphs (KBCQA) is a high-stakes puzzle that requires an AI to bridge the gap between natural language and structured data. To provide accurate answers, a system must overcome three primary hurdles:
- Contextual Complexity: Resolving coreferences (e.g., "Who is his father?") and ellipsis (missing information from previous turns) is essential for maintaining dialogue coherence.
- Logical Rigor: Translating natural language into precise, executable code (like SPARQL) for Knowledge Graphs (KG) containing millions of entities requires high fidelity.
- Structural Fragility: Large Language Models (LLMs) often suffer from hallucinations, generating "plausible-looking" queries that are syntactically broken or reference non-existent relations.
Existing methods typically attempt to generate entire queries in a single step, which is highly error-prone, or utilize rigid step-by-step agents that are computationally expensive and slow. SEAL addresses these issues by shifting the paradigm toward a two-stage agentic process.
Methodology: The SEAL Framework
SEAL introduces a modular approach to query generation and system evolution, focusing on semantic decomposition and dynamic learning.
Stage 1: S-expression Core Extraction & Calibration
Instead of generating a complete query immediately, the LLM extracts a "minimal S-expression core"—the essential semantic components of the question. An agent then "calibrates" this core by correcting syntax errors and employing a "light linking" strategy. This strategy aligns surface-level names with actual Knowledge Graph IDs using semantic similarity, ensuring the query is grounded in the database's reality.
Stage 2: Template-Based Composition
The system predicts the question type (e.g., "count," "verify," or "comparison") and selects a corresponding logical template. The calibrated core is then plugged into this template to create a fully executable query. This separation of semantics (the core) from structure (the template) prevents the performance collapse often seen when LLMs attempt to generate long, complex sequences.
The Self-Evolving Mechanism
The "secret sauce" of SEAL is its ability to learn from its own interactions through three components:
- Local Memory: Tracks entities within the current conversation to maintain context.
- Global Memory: Acts as a dynamic library of successful query patterns.
- Reflection Module: After executing a query, the agent evaluates the result. If successful, it distills the logic into a new template and saves it to Global Memory. This allows the system to evolve and handle similar complex queries more efficiently in future turns without manual retraining.
Evaluation and Results
SEAL was evaluated on the SPICE benchmark, a conversational semantic parsing dataset based on Wikidata. The dataset contains complex multi-turn interactions involving logical, quantitative, and comparative reasoning.
Key Performance Metrics
- Structural Accuracy: SEAL improved the parsing success rate to 88.1%, a significant leap from the 66.0% achieved by the strong KB-BINDER baseline.
- Complex Reasoning: In specific "Logical Reasoning" tasks, SEAL achieved an m-F1 of 73.08, outperforming existing unsupervised baselines.
- Robustness to Length: While competing models saw performance drop as query complexity increased, SEAL maintained high stability, even for queries containing more than 32 elements.
- Zero-Shot Capability: The self-evolving version of SEAL achieved an overall score of 34.32, significantly outperforming zero-shot KB-BINDER (19.76), proving that the system can learn effectively with minimal initial data.
Discussion and Insights
The research highlights several critical insights for the future of KBCQA. First, decomposition is key; by isolating the semantic core, the model avoids the pitfalls of long-form generation. Second, the authors found that Top-1 Linking is sufficient; because LLMs are semantically precise, picking the single best match often outperforms traditional "top-k" approaches that introduce noise.
However, limitations remain. The current S-expression-to-SPARQL transformation is tied to specific patterns and requires further generalization. Additionally, the calibration agent may occasionally "fix" queries that were intended to return null results, potentially impacting precision. Finally, the computational overhead of multiple LLM calls suggests a need for smaller, specialized models for subtasks like coreference resolution.
Conclusion
SEAL represents a significant step forward in creating dynamic, self-improving AI agents for Knowledge Graph interaction. By combining semantic calibration with a self-evolving memory, SEAL transforms from a static parser into a learning agent capable of mastering complex conversational reasoning.
Summary updated