The Role of Output Vocabulary in T2T LMs for SPARQL Semantic Parsing

Debayan Banerjee, Pranav Ajit Nair, Ricardo Usbeck, Chris Biemann

Introduction

Semantic parsing—the task of translating natural language into formal queries like SPARQL—is a cornerstone of Knowledge Graph Question Answering (KGQA). While Text-to-Text (T2T) models like T5 have shown promise, they face a significant "tokenizer tax." These models are pre-trained on human language, and their tokenizers are optimized for English. When encountering SPARQL's specialized syntax (e.g., curly braces, colons, and keywords), the tokenizer often fragments them into numerous sub-word units, increasing the complexity of the decoding task.

Methodology: Vocabulary Substitution

To mitigate this bottleneck, this research explores changing the model's output vocabulary. Instead of forcing the model to generate raw SPARQL, the query is mapped to an alternative vocabulary before training.

The authors proposed several substitution strategies to simplify the output space:

  1. Masked Query: Replaces structural characters (like { and }) with textual identifiers (e.g., OB for Open Bracket).
  2. Dictionary Substitution: Maps SPARQL keywords to common English words (e.g., SELECT becomes DOG). This leverages the model's pre-existing knowledge of these tokens.
  3. CharN Substitution: Replaces keywords with random strings of fixed lengths (1, 2, 4, or 8).
Examples of vocabulary substitution strategies for a SPARQL query.

Experimental Setup

The study utilized the GrailQA dataset, a large-scale benchmark based on Freebase containing approximately 64,331 questions. The authors focused on a high-resource split of roughly 31,000 training examples. Two T5 variants (Small and Base) were evaluated using two training regimes: Fine-tuning (FT), which updates all weights, and Prefix Tuning (PT), which keeps the model frozen and learns a small set of task-specific parameters.

Results and Analysis

The results demonstrate that aligning the output vocabulary with the model's pre-trained tokenizer leads to dramatic performance improvements across all settings.

Performance comparison across different output vocabularies on GrailQA.

Key Findings:

  • The Tokenizer Bottleneck: Models performed significantly better when outputting "nonsense" tokens (like random letters) than actual SPARQL. For T5-Base (FT), accuracy rose from 74.45% (original) to 92.61% with the char1 vocabulary—an absolute gain of over 18%.
  • Simplicity and Length: There is a clear inverse relationship between token length and performance. Shorter tokens (e.g., char1) consistently outperformed longer ones (e.g., char8). Shorter sequences reduce decoding steps and minimize potential errors.
  • Efficiency: The char1 setting reduced the average target sequence length (ALFL) from 125 to 57 tokens, effectively halving the decoder's workload.
  • Frequency Advantage: The dictionary approach, using common words, performed nearly as well as char1. This suggests that tokens seen frequently during pre-training are more easily repurposed for new tasks, regardless of their original semantic meaning.

Conclusion

This work highlights that the difficulty of semantic parsing for T2T models is largely a consequence of vocabulary mismatch. By simplifying the output space to align with the model's tokenizer, researchers can achieve substantial gains in both accuracy and efficiency. These findings suggest that "speaking the model's language" is a powerful strategy for improving code-generation tasks in LLMs.

Summary updated

Abstract

In this work, we analyse the role of output vocabulary for text-to-text (T2T) models on the task of SPARQL semantic parsing. We perform experiments within the the context of knowledge graph question answering (KGQA), where the task is to convert questions in natural language to the SPARQL query language. We observe that the query vocabulary is distinct from human vocabulary. Language Models (LMs) are pre-dominantly trained for human language tasks, and hence, if the query vocabulary is replaced with a vocabulary more attuned to the LM tokenizer, the performance of models may improve. We carry out carefully selected vocabulary substitutions on the queries and find absolute gains in the range of 17% on the GrailQA dataset.
BibTeX
@inproceedings{banerjee2023role,
  author = {Debayan Banerjee and Pranav Ajit Nair and Ricardo Usbeck and Chris Biemann},
  title = {{The Role of Output Vocabulary in T2T LMs for SPARQL Semantic Parsing}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.774},
}
Figure