The Role of Output Vocabulary in T2T LMs for SPARQL Semantic Parsing

Debayan Banerjee, Pranav Ajit Nair, Ricardo Usbeck, Chris Biemann

Abstract

In this work, we analyse the role of output vocabulary for text-to-text (T2T) models on the task of SPARQL semantic parsing. We perform experiments within the context of knowledge graph question answering (KGQA), where the task is to convert questions in natural language to the SPARQL query language. We observe that the query vocabulary is distinct from human vocabulary. Language Models (LMs) are pre-dominantly trained for human language tasks, and hence, if the query vocabulary is replaced with a vocabulary more attuned to the LM tokenizer, the performance of models may improve. We carry out carefully selected vocabulary substitutions on the queries and find absolute gains in the range of 17% on the GrailQA dataset.

Introduction

Semantic parsing—the process of translating natural language questions into formal machine-readable queries—is a cornerstone of modern Knowledge Graph Question Answering (KGQA). Currently, the dominant paradigm involves using Text-to-Text (T2T) Language Models, such as T5, to perform this translation. However, these models are pre-trained on massive corpora of human language, creating a fundamental friction when they are tasked with generating the technical, symbol-heavy syntax of SPARQL.

This research investigates whether the mismatch between human-centric pre-training and formal query syntax acts as a performance bottleneck. By "re-skinning" the output vocabulary to align more closely with the model's pre-existing knowledge, we demonstrate that significant performance gains can be achieved without altering the underlying model architecture.

The Tokenizer Tax

Language models do not process text as raw strings; they use tokenizers to break text into manageable units. SPARQL is replete with special characters (e.g., {, }, ^) and technical keywords that are rare in natural language. Consequently, a standard tokenizer often fragments a single SPARQL keyword into multiple, semantically thin "sub-tokens."

In the GrailQA dataset, the 48 core SPARQL keywords are decomposed into 124 sub-tokens by the T5 tokenizer. This increases the "decoding load"—the number of individual tokens the model must predict to complete a query. Longer sequences inherently increase the probability of error at each step of the generation process.

Methodology

To test the impact of vocabulary on performance, we employed two versions of the T5 model (Small and Base) across two training paradigms: Fine-tuning (updating all weights) and Prefix Tuning (updating only a small set of task-specific parameters while keeping the model frozen).

We introduced several Vocabulary Substitution strategies to map standard SPARQL syntax to alternative representations:

  • Original: The baseline SPARQL query (using minor masking for special characters).
  • Dictionary: Keywords are replaced with common English words (e.g., SELECT becomes DOG). This leverages the model's familiarity with high-frequency tokens from its pre-training phase.
  • Char1: Keywords are replaced with single characters (e.g., SELECT becomes A). This minimizes the total output sequence length.
  • Char2, Char4, Char8: Keywords are replaced with random strings of fixed lengths to isolate the effect of sequence length on model accuracy.

Results and Analysis

The experimental results on the GrailQA dataset (comprising ~31,000 training and ~8,800 test examples) reveal a significant performance gap between native SPARQL and substituted vocabularies.

Performance Gains

Both T5-Small and T5-Base showed dramatic improvements when using simplified vocabularies:

  • T5-Small (Fine-tuned): Accuracy rose from 74.08% (Original) to 92.13% (Char1), an absolute gain of roughly 18%.
  • T5-Base (Fine-tuned): Accuracy improved from 74.45% to 92.61%.
  • Prefix Tuning: This method proved highly sensitive to vocabulary complexity. While T5-Small reached 84.89% with the Char1 mapping, its performance plummeted as the substituted strings grew longer (e.g., in the Char8 setting).

Key Insights

  1. Familiarity Matters: The Dictionary setting consistently outperformed the Original SPARQL syntax. This suggests that LMs are more proficient at generating words they encountered millions of times during pre-training, even if the semantic mapping is arbitrary.
  2. Decoding Load is Critical: As the length of the substituted strings increased (from Char2 to Char8), performance steadily declined. This confirms that the sheer number of tokens the model must generate is a primary driver of error.
  3. Fine-tuning Robustness: Full fine-tuning is significantly more resilient to vocabulary changes than prefix tuning. The frozen weights in prefix tuning struggle to adapt to the increased complexity of long, random token sequences.

Conclusion

Our findings suggest that the way we represent formal languages to T2T models is just as important as the model architecture itself. By reducing the "tokenizer tax" and aligning the output space with the model's pre-training, we can achieve substantial gains in semantic parsing accuracy. Future work will explore the internal attention mechanisms of these models to better understand how they process simplified versus complex technical vocabularies.

Summary updated

Abstract

In this work, we analyse the role of output vocabulary for text-to-text (T2T) models on the task of SPARQL semantic parsing. We perform experiments within the the context of knowledge graph question answering (KGQA), where the task is to convert questions in natural language to the SPARQL query language. We observe that the query vocabulary is distinct from human vocabulary. Language Models (LMs) are pre-dominantly trained for human language tasks, and hence, if the query vocabulary is replaced with a vocabulary more attuned to the LM tokenizer, the performance of models may improve. We carry out carefully selected vocabulary substitutions on the queries and find absolute gains in the range of 17% on the GrailQA dataset.
BibTeX
@inproceedings{banerjee2023role,
  author = {Debayan Banerjee and Pranav Ajit Nair and Ricardo Usbeck and Chris Biemann},
  title = {{The Role of Output Vocabulary in T2T LMs for SPARQL Semantic Parsing}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.774},
}
Figure