Abstract
In this work, we present an end-to-end Knowledge Graph Question Answering (KGQA) system named GETT-QA. GETT-QA uses T5, a popular text-to-text pre-trained language model. The model takes a question in natural language as input and produces a simpler form of the intended SPARQL query. In the simpler form, the model does not directly produce entity and relation IDs. Instead, it produces corresponding entity and relation labels. The labels are grounded to KG entity and relation IDs in a subsequent step. To further improve the results, we instruct the model to produce a truncated version of the KG embedding for each entity. The truncated KG embedding enables a finer search for disambiguation purposes. We find that T5 is able to learn the truncated KG embeddings without any change of loss function, improving KGQA performance. As a result, we report strong results for LC-QuAD 2.0 and SimpleQuestions-Wikidata datasets on end-to-end KGQA over Wikidata.
Introduction
Knowledge Graph Question Answering (KGQA) is the process of translating natural language questions into formal queries (such as SPARQL) to retrieve precise answers from structured databases like Wikidata. A primary challenge in this field is "grounding": the mapping of natural language mentions to specific Knowledge Graph (KG) identifiers. For instance, while a human understands "Barack Obama," a system must identify the specific ID Q76. This task is complicated by entity collisions—where multiple distinct entities share the same label—and the arbitrary nature of KG identifiers, which language models struggle to predict directly.
Methodology
GETT-QA (Graph Embedding based T2T Transformer) addresses these challenges by utilizing the T5 architecture in a novel text-to-text framework. The system operates through a multi-stage pipeline designed to bridge the gap between linguistic labels and graph structures.
Skeleton SPARQL Generation
Rather than forcing the transformer to output cryptic KG identifiers, GETT-QA trains the model to generate a "skeleton" SPARQL query. This skeleton uses natural language labels for entities and relations, wrapped in special tags (e.g., <ent>Barack Obama</ent>). This allows the model to leverage its pre-trained linguistic knowledge more effectively.
Truncated KG Embeddings
The core innovation of GETT-QA is the inclusion of truncated graph embeddings. Alongside the entity label, the model is trained to output the first 10 dimensions of the entity’s 200-dimensional TransE embedding as a string of floats. By treating these numerical vectors as text, the model learns a "soft" identifier that represents the entity's structural position within the graph.
The Grounding Pipeline
Once the skeleton query and embeddings are generated, the system performs grounding:
- Label Search: Elasticsearch (BM25) identifies the top 100 candidate entities matching the generated label.
- Embedding Re-ranking: The system calculates the dot product between the T5-generated truncated embedding and the actual embeddings of the 100 candidates to re-rank them.
- Relation Linking: Relations are matched using BERT embedding similarity, benefiting from the relatively small search space of relations compared to entities.
- Query Execution: The system combines the top 6 entity and top 3 relation candidates into potential SPARQL queries, executing them against the KG until a valid result is returned.
Experimental Evaluation
The model was evaluated on two benchmark datasets over Wikidata: LC-QuAD 2.0, which features complex, multi-hop questions, and SimpleQuestions-Wikidata, which focuses on single-triple subject-predicate-object queries.
Key Results
GETT-QA demonstrated significant performance improvements over existing baselines:
- LC-QuAD 2.0: The system achieved a Precision@1 of 0.403, outperforming UNIQORN (0.331) and QAnswer (0.308).
- SimpleQuestions-Wikidata: The system reached an F1 score of 0.761, a substantial increase over previous benchmarks like Text2Graph (0.618).
- Efficiency: The end-to-end process, including re-ranking, averages 1.2 seconds per question.
Discussion and Insights
One of the most surprising findings is that the T5 model can learn to output numerical vectors as text strings without any modifications to its loss function. This capability is critical for disambiguation; on the LC-QuAD 2.0 dataset, the addition of truncated embeddings resulted in an 8% absolute gain in precision.
Furthermore, the researchers discovered that shorter embeddings (10 dimensions) were more effective than longer ones (30 dimensions). In higher-dimensional spaces, the angular discriminability decreases, making it more difficult for the transformer's output head to accurately predict the specific "direction" of an entity in the embedding space.
Conclusion and Future Work
GETT-QA demonstrates that combining text-to-text transformers with structural graph embeddings significantly enhances the accuracy of KGQA systems. While grounding remains a bottleneck—with the correct entity appearing in the top-k candidates only 41% of the time for complex queries—the use of "soft" identifiers provides a promising path forward. Future research will investigate modified loss functions for embedding generation and the use of text-based entity descriptions to further refine the disambiguation process.
Summary updated