Explaining Large Language Model-Based Neural Semantic Parsers (Student Abstract)

Daking Rai, Yilun Zhou, Bailin Wang, Ziyu Yao

Abstract

While large language models (LLMs) have demonstrated strong capability in structured prediction tasks such as semantic parsing, few amounts of research have explored the underlying mechanisms of their success. Our work studies different methods for explaining an LLM-based semantic parser and qualitatively discusses the explained model behaviors, hoping to inspire future research toward better understanding them.

Introduction

Semantic parsing—the process of translating natural language questions into structured queries like SQL—has traditionally relied on specialized structured decoders. Recently, general-purpose Large Language Models (LLMs) like T5, which generate output tokens sequentially, have begun to outperform these specialized architectures. However, the sequential nature of LLMs makes their decision-making process opaque. It remains unclear whether these models truly grasp the underlying database schema or simply rely on surface-level pattern matching. This study evaluates various local explanation methods to identify which techniques provide the most faithful insights into LLM-based semantic parsers.

Methodology

The study evaluates five local explanation techniques applied to a UnifiedSKG model (a T5-based parser) using the Spider dataset, a benchmark for complex, cross-domain text-to-SQL tasks.

  1. LIME: Approximates the model locally with an interpretable surrogate.
  2. Shapley Value: A game-theoretic approach measuring a feature's average marginal contribution.
  3. Kernel SHAP: An efficient estimation of Shapley values via linear classification.
  4. LERG (LERG_L and LERG_S): Sequence-generation adaptations of LIME and Shapley values, respectively.
  5. Attention: Utilizing the last layer of cross-attention weights as a proxy for importance.

Faithfulness is measured using two primary metrics: Necessity (the performance drop when important features are removed) and Sufficiency (the model's confidence when only important features are retained).

Results and Faithfulness

Quantitative analysis reveals that Shapley-based methods, particularly LERG_S, are significantly more faithful than simpler alternatives like LIME or raw Attention.

Necessity and Sufficiency metrics across different explanation methods. LERG_S demonstrates superior faithfulness by maximizing necessity and minimizing sufficiency values.

As shown in Figure 1, LERG_S consistently outperforms other methods across various necessity and sufficiency ratios. While Attention-based explanations are less accurate than LERG_S, they prove to be a more faithful "shortcut" than LIME, offering a computationally cheaper alternative for rapid insights.

Qualitative Insights and Model Behavior

Beyond quantitative metrics, the study explores the "plausibility" of explanations—how well they align with human reasoning. LERG_S generated plausible explanations for 85% of correct predictions and 70% of incorrect ones.

Visualization of explanations for various scenarios: (a) correct prediction, (b) incorrect prediction, (c) in-domain example, and (d) hard-level compositional generalization.

Qualitative analysis (Figure 2) shows that even when the model fails, its reasoning is often "near the mark." For instance, in cases of incorrect column selection, the model is frequently observed attending to the correct table, suggesting that errors are often localized rather than complete hallucinations. Furthermore, the model maintains high plausibility in both in-domain (75%) and out-of-domain (80%) scenarios, indicating a consistent reasoning process even when generalizing to unfamiliar database schemas.

Limitations and Future Work

Despite the success of Shapley-based methods, their high computational cost remains a barrier, as they require numerous model passes per explanation. Future research should focus on improving the efficiency of these methods or refining attention-based alternatives. Additionally, current methods look at input tokens in isolation; future work should explore feature interactions to explain how complex relationships drive specific SQL clauses like JOIN or INTERSECT. Finally, the authors suggest using these explanations as a debugging tool to identify specific logic failures in compositional generalization.

Summary updated

Abstract

While large language models (LLMs) have demonstrated strong capability in structured prediction tasks such as semantic parsing, few amounts of research have explored the underlying mechanisms of their success. Our work studies different methods for explaining an LLM-based semantic parser and qualitatively discusses the explained model behaviors, hoping to inspire future research toward better understanding them.
BibTeX
@inproceedings{rai2023explaining,
  author = {Daking Rai and Yilun Zhou and Bailin Wang and Ziyu Yao},
  title = {{Explaining Large Language Model-Based Neural Semantic Parsers (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i13.27014},
}
Figure