Explaining Large Language Model-Based Neural Semantic Parsers (Student Abstract)

Daking Rai, Yilun Zhou, Bailin Wang, Ziyu Yao

Introduction

Large Language Models (LLMs) have demonstrated remarkable capabilities in semantic parsing—the process of converting natural language queries into structured logical forms like SQL. While traditional systems relied on specialized decoders, modern LLMs like T5 achieve state-of-the-art performance by treating this as a sequence-to-sequence translation task. However, the internal logic of these models remains opaque. To deploy LLMs in critical environments, we must understand why they generate specific tokens—whether they are correctly mapping question keywords to database schema elements or relying on spurious correlations. This research evaluates local explanation methods to identify those that most faithfully represent the model's internal reasoning.

Methodology

The study compares several local explanation methods that attribute importance scores to input features (question tokens and schema elements) for a given prediction:

  • LIME: Approximates the model locally with an interpretable linear regressor.
  • Shapley Values & Kernel SHAP: Game-theoretic methods calculating the marginal contribution of each feature.
  • LERG (Local Explanation for Response Generation): A framework tailored for sequence tasks, with variants LERG_L (LIME-based) and LERG_S (Shapley-based).
  • Attention: Uses cross-attention weights as a proxy for feature importance.

Faithfulness is measured using two primary metrics: Necessity (the increase in model perplexity when the most important features are removed) and Sufficiency (the model's perplexity when only the most important features are retained).

Quantitative Evaluation

The evaluation was conducted on the Spider dataset using a T5-base model (UnifiedSKG). The results consistently showed that Shapley-based methods, particularly LERG_S, outperform others in faithfulness.

Quantitative comparison of explanation methods using Necessity and Sufficiency metrics.

As shown in the quantitative analysis, LERG_S and Kernel SHAP demonstrate the highest necessity values and lowest sufficiency values across various ratios. This confirms that these methods more accurately identify the input tokens that drive the model's output. Interestingly, while raw Attention is less precise than Shapley-based methods, it remains more faithful than LIME-based approaches and offers a computationally efficient alternative.

Qualitative Analysis and Plausibility

Beyond quantitative metrics, the study assessed the "plausibility" of explanations—how well they align with human intuition. A qualitative review found that 85% of correct predictions were supported by plausible explanations.

Visualizations of model explanations for various prediction scenarios.

The visualizations in the study illustrate how the model attributes SQL tokens to input components. In correct predictions, the model accurately links SQL keywords to the relevant question tokens. Crucially, even in 70% of incorrect predictions, the model's logic remains plausible; it may look at the correct part of the question but fail to map it to the precise schema element. The model also demonstrates strong compositional generalization, correctly handling complex, multi-part queries by attributing different segments of the SQL to their corresponding natural language counterparts.

Conclusion

This research highlights that Shapley-value-based methods, specifically LERG_S, provide the most faithful explanations for LLM-based semantic parsers. While computationally intensive, these methods offer critical insights into model behavior, revealing that LLMs often follow human-like logic even when they fail. Future work will focus on reducing the computational cost of these explanations and developing better user interfaces to help non-experts debug and trust these powerful models.

Summary updated

Abstract

While large language models (LLMs) have demonstrated strong capability in structured prediction tasks such as semantic parsing, few amounts of research have explored the underlying mechanisms of their success. Our work studies different methods for explaining an LLM-based semantic parser and qualitatively discusses the explained model behaviors, hoping to inspire future research toward better understanding them.
BibTeX
@inproceedings{rai2023explaining,
  author = {Daking Rai and Yilun Zhou and Bailin Wang and Ziyu Yao},
  title = {{Explaining Large Language Model-Based Neural Semantic Parsers (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i13.27014},
}
Figure