Measuring the Effect of Disfluency in Multilingual Knowledge Probing Benchmarks

Kirill Semenov, Rico SennrichEMNLP 2025

TL;DR

This paper investigates how the grammatical correctness and fluency of prompts affect large language models' (LLMs) ability to retrieve factual knowledge across multiple languages. The authors demonstrate that using naturally translated sentences significantly improves LLM performance compared to traditional templated prompts, suggesting that current multilingual benchmarks may underestimate LLMs' true capabilities.

What Problem Does This Paper Address?

Evaluating the factual knowledge of LLMs, especially across many languages, is crucial. Existing multilingual benchmarks, like MLAMA, often rely on templated sentences where named entities are inserted. This process can lead to ungrammatical or awkward phrasing in the target language, particularly in morphologically rich languages. Such disfluencies can confuse LLMs, making it difficult to determine if poor performance stems from a lack of cross-lingual knowledge transfer or simply from poorly formed prompts. This paper aims to bridge this gap by directly comparing the performance of LLMs on templated prompts versus naturally translated, fluent prompts.

Methods

The core of the study involves creating and evaluating three types of prompt verbalizations for factual knowledge retrieval:

  1. Template Filling: Using pre-defined templates and filling them with named entities (NEs) from Wikidata. This is the standard approach in benchmarks like MLAMA.
  2. Google Translate (GT) Verbalization: English templates are filled with NEs, and then the entire sentence is translated into the target language using Google Translate.
  3. ChatGPT Verbalization: Similar to GT, but using ChatGPT (specifically GPT-4o-mini) for sentence-level translation, with explicit instructions to maintain grammatical agreement and use provided NE translations.

The researchers then probe a Llama-2-7b model with these different verbalizations and measure its ability to correctly identify the object NE. The evaluation metric used is R@n (Recall at n), which measures the percentage of times the correct object NE is ranked within the top n predictions. They also analyze the mean rank of the correct object.

Datasets

The primary dataset used is MLAMA (Kassner et al., 2021), a multilingual benchmark for factual knowledge probing. The authors sampled a subset of languages and relations from MLAMA.

  • Slavic Languages: Russian (ru), Czech (cs), Ukrainian (uk), and Croatian (hr).
  • Non-Slavic Languages: Spanish (es), Chinese (zh), Vietnamese (vi), Indonesian (id), and Danish (da).

They created new versions of the MLAMA dataset by translating the original templates into fluent sentences using Google Translate and ChatGPT.

Key Insights and Ideas

  • Fluency Matters: The most significant insight is that grammatically correct and fluent prompts dramatically improve LLM performance on factual knowledge retrieval tasks. This suggests that current multilingual benchmarks might be underestimating LLMs' true capabilities.
  • GT Outperforms ChatGPT (Surprisingly): In many cases, Google Translate's sentence-level translations yielded better results than those generated by ChatGPT, even with few-shot prompting. The authors attribute this to GT's ability to "explicate" by adding clarifying information or using more natural phrasing, which helps resolve ambiguities.
  • Grammaticality is Key: The study highlights that not only the fluency of the prompt's beginning but also the correct inflection of the object NE (e.g., using the correct case or gender agreement) significantly contributes to better retrieval.
  • "Reversal Curse" Mitigation: Fluent translations can help mitigate phenomena like the "reversal curse," where LLMs struggle with inverse relations (e.g., "X is the capital of Y" vs. "The capital of X is Y").
  • Language-Specific Nuances: The benefits of sentence-level translation vary across languages, with some languages showing larger gains than others, potentially due to their morphological complexity or resource availability.

Key Results

  • Significant Performance Increase: Sentence-level translations (both GT and ChatGPT) consistently led to higher R@1 scores compared to templated prompts across most languages. For instance, in Czech and Russian, GT translation improved performance by up to 10%.
  • Improved Ranking: GT verbalizations also resulted in a lower mean rank for the correct object, indicating more precise retrieval.
  • Gender Agreement: For facts involving female subjects, templated prompts often used incorrect masculine verb forms. Sentence-level translations, especially from GT and ChatGPT, significantly improved the accuracy of feminine verb usage, leading to better factual extraction.
  • Alias Inclusion: Allowing object aliases as correct answers increased scores but did not fundamentally change the observed trends between verbalization types.

Limitations and Future Work

  • Limited Sample Size: The study focused on a subset of languages and relations from MLAMA due to the manual effort required for creating and verifying fluent translations.
  • Google Translate Control: The authors had limited control over Google Translate's output, which led to artifacts like explicitation, potentially contributing to its strong performance.
  • Distractor Choice: The method for sampling distractors was pseudo-random, and the optimal strategy for distractor selection remains an open research question.
  • Fact Imbalance: The MLAMA dataset exhibits an imbalance in the number of facts across languages, which could influence results.
  • Scalable Grammaticality Measurement: Developing a scalable metric for automatically assessing prompt grammaticality is an open challenge. The authors explored quality estimation (QE) scores and grammar error correction systems but found them insufficient on their own.
  • Prompting Strategies: Further research is needed to develop effective prompting strategies for all language typologies, especially verb-final languages.

The authors encourage the community to prioritize the grammaticality and fluency of prompts in multilingual benchmarks to obtain more reliable and interpretable LLM evaluations.

Summary updated

BibTeX
@inproceedings{semenov2025measuring,
  author = {Kirill Semenov and Rico Sennrich},
  title = {{Measuring the Effect of Disfluency in Multilingual Knowledge Probing Benchmarks}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure