Multilingual Verbalisation of Knowledge Graphs

Yifei Song, William Soto Martinez, Anna Nikiforovskaya, Evan Chapple, Claire GardentEMNLP 2025

TL;DR

This paper tackles the challenge of generating natural language descriptions from knowledge graphs (KGs) across multiple languages, especially those with limited resources. They introduce new multilingual datasets and compare three KG-to-text generation approaches, finding that few-shot prompting with large language models (LLMs) offers the most robust and effective solution, particularly for low-resource languages.

What Problem Does This Paper Address?

Generating text from knowledge graphs (KG-to-Text) is crucial for making structured data more accessible. However, most existing work focuses on English, leaving a significant gap in how to scale this to languages with varying resource availability. The core challenges are: 1) the scarcity of parallel KG-text data for training and evaluation in many languages, and 2) the difficulty of adapting models to these diverse linguistic landscapes. This paper aims to bridge this gap by exploring multilingual KG-to-Text generation, with a specific focus on low-resource languages.

Methods

The authors compare three main approaches for multilingual KG-to-Text generation:

  • NLG+MT (Hybrid Pipeline): This method first uses a KG-to-English model (ControlPrefixes) to generate English text from a KG, followed by Machine Translation (MT) to the target language. This approach doesn't require parallel multilingual data for the KG-to-Text step itself, relying instead on existing MT systems.
  • FTMT (Fine-Tuned Machine Translation): This is a direct, end-to-end approach where pre-trained MT models are fine-tuned on multilingual "silver" KG-text data. This method leverages MT models but adapts them specifically for the KG-to-Text task.
  • FEW-SHOT LLM Prompting: This approach utilizes large language models (LLMs) without any additional training. Instead, it relies on carefully crafted prompts that include a few examples (few-shot learning) to guide the LLM in generating text from KGs. The authors explore various strategies for selecting these few-shot examples and designing the prompts, including enriching them with target language entity and property labels.

Datasets

  • WebNLG: The authors utilize existing WebNLG test sets for English, Russian, Breton, Welsh, Irish, and Maltese, as well as a new Spanish test set. This dataset is used for in-domain evaluation.
  • 5L-GOLD-OOD: A newly constructed out-of-domain test set for five high-resource languages (English, French, Spanish, Russian, Chinese). This dataset was created by extracting graphs from Wikidata and aligning them with manually verified and machine-translated verbalizations, designed to be distinct from existing datasets.
  • 9L-SILVER-WebNLG: A large, automatically generated (silver) multilingual training dataset created by machine-translating the English WebNLG training data into nine target languages. This dataset is used for fine-tuning MT models and as a retrieval basis for few-shot prompting.

Key Insights and Ideas

  • LLMs are King for Multilingual KG-to-Text: The most significant finding is that few-shot prompting with LLMs consistently outperforms traditional NLG+MT and FTMT approaches, especially for low-resource languages. This highlights the power of LLMs' cross-lingual transfer capabilities.
  • Prompt Engineering Matters: The authors demonstrate that the way prompts are designed and few-shot examples are selected has a substantial impact on LLM performance. Their best strategy involved enriching prompts with target language labels and selecting few-shot examples based on property overlap with the input graph.
  • Low-Resource Languages Benefit Most from LLMs: While all methods see a performance drop on low-resource languages, the FEW-SHOT LLM approach significantly narrows this gap, suggesting it can leverage its multilingual pre-training to compensate for the lack of direct training data.
  • Domain Shift is a Challenge for Traditional Methods: NLG+MT and FTMT models show a noticeable performance degradation when moving from seen (WebNLG) to unseen (5L-GOLD-OOD) data, indicating they struggle with more specialized or complex KGs. LLMs, particularly FEW-SHOT, are more robust to this domain shift.
  • Language Representation in LLM Pre-training is Key: LLM performance is strongly influenced by how well a language is represented in their pre-training data. Qwen models excel in Chinese and Russian, while Llama models perform better in English and Spanish, aligning with their respective training corpora.

Key Results

  • The FEW-SHOT LLM prompting approach achieved the best overall performance across all nine languages, outperforming NLG+MT and FTMT by a significant margin.
  • For English, the FEW-SHOT approach achieved BLEU scores up to 97.58 (Table 14), while NLG+MT reached 96.86 and FTMT 95.16.
  • On the 5L-GOLD-OOD dataset, FEW-SHOT LLMs showed a marked advantage, particularly for languages well-represented in their pre-training data like English and Chinese. For instance, Qwen2.5-72B achieved a BLEU of 52.46 on English 5L-GOLD-OOD, compared to NLG+MT's 34.97.
  • For low-resource languages (Breton, Irish, Maltese, Welsh), FEW-SHOT prompting significantly narrowed the performance gap compared to NLG+MT and FTMT, demonstrating its effectiveness in data-scarce scenarios. For example, on Breton 5L-GOLD-OOD, FEW-SHOT achieved a BLEU of 35.0, while NLG+MT was 13.4 and FTMT 14.3 (Figure 2).

Limitations and Future Work

  • Data Noise: The use of machine translation for creating silver training data and for generating test set translations introduces some noise, which can affect model performance.
  • Limited Domain Coverage: While the new 5L-GOLD-OOD dataset covers a range of Wikidata categories, it does not encompass all possible entity and property combinations.
  • Scalability and Generalization: The authors suggest that training on larger datasets and evaluating on more languages and knowledge bases would be necessary to ensure broader generalization capabilities.
  • Potential Data Contamination: For Russian, Llama's strong performance on the in-domain WebNLG test set might be due to memorization of training data, as it underperforms on the out-of-domain set compared to Qwen. This highlights the need for careful evaluation to distinguish true generalization from memorization.
  • Future Work: The paper suggests exploring methods to improve KG-to-Text generation for domains beyond Wikidata and for a wider range of languages. They also emphasize the importance of training and assessing models to avoid introducing harmful biases in real-world applications.

Summary updated

BibTeX
@inproceedings{song2025multilingual,
  author = {Yifei Song and William Soto Martinez and Anna Nikiforovskaya and Evan Chapple and Claire Gardent},
  title = {{Multilingual Verbalisation of Knowledge Graphs}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure