TL;DR
This paper introduces LECSP, a novel method for Continual Semantic Parsing (CSP) that tackles catastrophic forgetting and improves generalization without requiring real data replay. By analyzing SQL syntax variances between tasks, LECSP guides Large Language Models (LLMs) to reconstruct memories and fill knowledge gaps, leading to state-of-the-art performance on challenging benchmarks.
What Problem Does This Paper Address?
Semantic parsing aims to translate natural language questions into structured queries (like SQL) for databases. In real-world applications, databases are dynamic, meaning new tasks and data distributions emerge over time. Continual Semantic Parsing (CSP) addresses the challenge of training a single model to handle a sequence of these tasks, ensuring it performs well on both current and past tasks without forgetting. Existing methods often rely on replaying historical data (raising privacy concerns) or assume ideal settings (like knowing which model to use for a given task), which are impractical. The core problem is how to enable models to learn new tasks while retaining knowledge from previous ones, especially when data is scarce and privacy is a concern.
Methods
LECSP tackles CSP with a two-stage approach:
- Memory Reconstruction with LLMs:
- Domain Information Elimination: To focus on syntax, domain-specific entities (like table/column names) are masked from question-SQL pairs.
- Component Bias Analysis: SQL skeletons are clustered, and the "component bias" (differences in SQL syntax features between the current and previous tasks) is identified. This bias is crucial as it doesn't require historical data.
- Inter-Task Memory Completion: The identified component bias guides LLMs to generate synthetic "pseudo-samples" (question-SQL pairs) that fill the memory gaps between tasks. This is a "soft" generation process, using SQL skeletons to guide LLM output.
- Memory Calibration: To combat LLM hallucinations, a calibration strategy involving iterative self-correction and SQL skeleton-based sampling is employed to improve the accuracy and fidelity of generated pseudo-samples.
- Intra-Task Memory Reinforcement: For each annotated sample, new synthetic instances are generated by replacing entities, guided by the current task's SQL syntax features, to reinforce memory within the task.
- Task-Aware Dual-Teacher Distillation Learning Framework:
- LLMs as Teacher 1: The LLM (specifically, Mixtral-8x7B-Instruct-v0.1) acts as a teacher to transfer knowledge from the reconstructed memories (both historical and current task-specific) to a smaller "Student" model.
- Previous Student as Teacher 2: The student model from the previous task serves as a second teacher. This helps to constrain the current student's learning and maintain consistency of shared knowledge, mitigating potential biases introduced by the LLM teacher. KL divergence is used to align the output distributions of the current and previous student models.
The final loss function combines the loss on original annotated data with distillation losses from both teachers.
Datasets
The paper evaluates LECSP on two standard CSP benchmarks:
- Spider-stream-semi: Derived from the Spider dataset, it features 10 tasks with complex SQL syntax (JOIN, GROUP BY) and limited annotated samples per task (fewer than 500).
- Combined-stream: Combines the Spider and WikiSQL datasets, comprising 7 tasks. This benchmark focuses on the dual impact of table domain and SQL structure changes, using simpler SQL syntax for single-table queries.
The authors also introduce "cold start" scenarios by randomly shuffling task order, which is more reflective of real-world deployment than the typical "warm start" where the first task has the most data.
Key Insights and Ideas
- SQL Syntax Variance as a Memory Guide: The core idea that analyzing differences in SQL syntax without historical data can effectively guide LLMs to reconstruct relevant memories is a significant contribution. This bypasses the need for data replay.
- LLM-Guided Pseudo-Sample Generation: Leveraging LLMs to generate synthetic data for memory reconstruction, guided by syntax, is a clever way to overcome data scarcity and LLM hallucination issues through calibration.
- Dual-Teacher Distillation for Robustness: Using both an LLM and the previous student model as teachers provides a robust knowledge transfer mechanism, balancing the power of LLMs with the stability of prior task learning.
- Cold Start Performance: The strong performance in cold-start scenarios highlights the method's adaptability and robustness to initial task order and data availability, a critical aspect for real-world deployment.
Key Results
- LECSP significantly outperforms existing methods, including those using data replay or ideal settings, on both Spider-stream-semi and Combined-stream datasets.
- Improvements of up to 8.8% in Forward Transfer (FWT) are observed compared to SOTA methods like SFNET.
- LECSP achieves performance close to, and sometimes exceeding, the ORACLE setting (which represents the upper bound) in FWT, demonstrating superior generalization.
- In cold-start scenarios, LECSP shows significantly less performance degradation compared to baselines like PEFT and ProgPrompt, which are heavily reliant on initial task data.
- The memory calibration strategy is crucial, with its removal leading to a sharp performance drop of up to 31.4%.
Limitations and Future Work
- LLM Hallucinations: While memory calibration mitigates LLM hallucinations, it's an ongoing challenge. The paper acknowledges that LLMs can still produce noisy outputs.
- Computational Cost: While LECSP aims to be more efficient than training LLMs directly for every task, it still relies on LLMs for memory reconstruction, which can be computationally intensive. The paper mentions using open-source LLMs for offline deployment to address cost and privacy.
- Future Work: The paper suggests exploring more advanced LLM-based memory reconstruction techniques and further investigating the transferability of knowledge across highly dissimilar task domains. The authors also highlight the potential for migrating LLM knowledge to smaller models for practical deployment.
Summary updated