GETT-QA: Graph Embedding Based T2T Transformer for Knowledge Graph Question Answering

Debayan Banerjee, Pranav Ajit Nair, Ricardo Usbeck, Chris Biemann

TL;DR

GETT-QA is an end-to-end Knowledge Graph Question Answering (KGQA) system that uses a T5 transformer to generate SPARQL queries containing entity labels and truncated numerical graph embeddings. By training the model to "print" these embeddings as text, the researchers significantly improved the system's ability to disambiguate entities in massive datasets like Wikidata.

What Problem Does This Paper Address?

Knowledge Graph Question Answering (KGQA) is the task of taking a natural language question (e.g., "Who is the father of Barack Obama?") and translating it into a formal query language like SPARQL to retrieve an answer from a structured database.

The "Achilles' heel" of this process is grounding. In a Knowledge Graph (KG) like Wikidata, entities have arbitrary IDs (e.g., Q76 for Barack Obama). Language models are excellent at processing text labels, but they struggle to predict these arbitrary IDs, especially for entities they haven't seen frequently during training. Furthermore, many entities share the same label—there is a Barack Obama who was a President (Q76) and a Barack Obama that is a painting (Q47513588). Distinguishing between these "collisions" is a major hurdle for end-to-end systems.

Methods

The authors introduce GETT-QA (Graph Embedding based T2T Transformer), which utilizes the T5 (Text-to-Text Transfer Transformer) architecture. Their approach involves several novel steps:

  1. Skeleton SPARQL Generation: Instead of asking T5 to produce a final query with IDs, they train it to generate a "skeleton" query. This skeleton contains the natural language labels for entities and relations (e.g., <ent>Barack Obama</ent>) surrounded by special tags.
  2. Truncated KG Embeddings: This is the core innovation. Alongside the entity label, the model is trained to output a truncated version of the entity's TransE embedding—specifically, the first 10 dimensions of a 200-dimensional vector, formatted as a string of floats.
  3. The Grounding Pipeline:
    • Label Search: The system uses Elasticsearch (BM25) to find the top 100 candidate entities matching the generated label.
  4. Embedding Re-ranking: It then calculates the dot product between the truncated embedding generated by T5 and the actual embeddings of those 100 candidates.
  5. Relation Linking: Relations are matched using BERT embedding similarity, as the number of possible relations is much smaller than the number of entities.
  6. Execution: The system combines the top 6 entity candidates and top 3 relation candidates to form potential SPARQL queries, executing them against the KG until a valid response is found.

Datasets

The model was evaluated on two major benchmarks over Wikidata:

  • LC-QuAD 2.0: A dataset of ~30,000 complex questions involving multi-hop reasoning, qualifiers, and aggregates (count, min/max).
  • SimpleQuestions-Wikidata: A dataset of ~34,000 questions that can be answered with a single triple (subject-predicate-object).

Key Insights and Ideas

  • Numbers as Text: Surprisingly, the T5 model can learn to output numerical vectors as strings without any modification to its loss function or architecture. It effectively learns a "soft" identifier that captures the structural position of an entity in the graph.
  • The Power of Disambiguation: On the LC-QuAD 2.0 dataset, adding truncated embeddings led to an 8% absolute gain in precision. This confirms that the model's bottleneck wasn't understanding the question, but rather picking the right "Barack Obama" from the graph.
  • The Curse of Dimensionality: The researchers found that shorter embeddings (10 dimensions) actually performed better for label matching than longer ones (30 dimensions). They hypothesize that at higher dimensions, the "angular space" becomes less discriminable, making it harder for the model to learn the specific "direction" of an entity.
  • Template-Free Flexibility: Unlike many previous systems that rely on rigid query templates, GETT-QA generates the SPARQL structure dynamically, allowing it to handle a wider variety of complex question types.

Key Results

  • LC-QuAD 2.0: GETT-QA achieved a Precision@1 of 0.403, outperforming several established systems like UNIQORN (0.331) and QAnswer (0.308).
  • SimpleQuestions-Wikidata: The system reached an F1 score of 0.761, a significant jump over the previous best-reported results (Text2Graph at 0.618).
  • Efficiency: Despite the complex re-ranking and combination steps, the system maintains an average response time of 1.2 seconds per question, making it viable for real-world interaction.

Limitations and Future Work

The authors identify grounding as the remaining primary source of error. Even with embeddings, the correct entity and relation were found in the top-k candidates only 41% of the time for LC-QuAD 2.0.

Another specific failure mode involves COUNT queries. In SPARQL, if a query for a non-existent relationship is wrapped in a COUNT, the KG returns 0 rather than an error. This causes the system to stop at the first (potentially incorrect) query it tries, rather than exploring other candidate combinations.

Future work will explore using modified loss functions specifically designed for generating embeddings and investigating other "soft" identifiers, such as text-based entity descriptions.

Summary updated

Abstract

In this work, we present an end-to-end Knowledge Graph Question Answering (KGQA) system named GETT-QA. GETT-QA uses T5, a popular text-to-text pre-trained language model. The model takes a question in natural language as input and produces a simpler form of the intended SPARQL query. In the simpler form, the model does not directly produce entity and relation IDs. Instead, it produces corresponding entity and relation labels. The labels are grounded to KG entity and relation IDs in a subsequent step. To further improve the results, we instruct the model to produce a truncated version of the KG embedding for each entity. The truncated KG embedding enables a finer search for disambiguation purposes. We find that T5 is able to learn the truncated KG embeddings without any change of loss function, improving KGQA performance. As a result, we report strong results for LC-QuAD 2.0 and SimpleQuestions-Wikidata datasets on end-to-end KGQA over Wikidata.
BibTeX
@inproceedings{banerjee2023gett,
  author = {Debayan Banerjee and Pranav Ajit Nair and Ricardo Usbeck and Chris Biemann},
  title = {{GETT-QA: Graph Embedding Based T2T Transformer for Knowledge Graph Question Answering}},
  booktitle = {ESWC},
  year = {2023},
  url = {https://openreview.net/forum?id=wutrZb6D9dm},
}

Loading related content...

Figure