Do LLMs exhibit the same commonsense capabilities across languages?

Iván Martínez-Murillo, Elena Lloret, Paloma Moreda, Albert GattarXiv 2025 · arXiv · OpenReview

TL;DR

This paper investigates whether Large Language Models (LLMs) possess consistent commonsense reasoning abilities across different languages. They introduce MULTICOM, a new benchmark dataset for multilingual commonsense generation, and find that while LLMs perform best in English, their capabilities significantly drop for less-resourced languages, highlighting a critical gap in current AI.

What Problem Does This Paper Address?

Commonsense reasoning is a fundamental human ability that allows us to make inferences based on implicit knowledge about the world. While LLMs have shown impressive capabilities in generating human-like text, it's unclear if these abilities translate equally across languages, especially given that most LLM training data is English-centric. This imbalance could lead to AI systems that are proficient in English but fall short in understanding and generating commonsense in other languages. The paper aims to bridge this gap by systematically evaluating the multilingual commonsense generation performance of various LLMs.

Methods

The researchers employed a multi-pronged evaluation strategy:

  • Automatic Metrics: They used established metrics like BERTScore, Universal Sentence Encoder (USE) with Cosine Similarity, and Dependency Parsing with Levenshtein Distance. These metrics compare generated sentences against both a "target" (correct) sentence and a "counterfactual" (grammatically correct but nonsensical) sentence. A larger score difference between the target and counterfactual suggests better commonsense.
  • LLM-as-a-Judge: To complement automatic metrics, they used two LLMs, Prometheus-V2.0 and JudgeLM, to rate the commonsensical quality of generated sentences. This approach leverages the advanced reasoning capabilities of LLMs to provide more nuanced evaluations.
  • Human Evaluation: A subset of the generated sentences was evaluated by human annotators to validate the findings from the automatic and LLM-based evaluations.

They also explored the impact of providing a "context sentence" alongside the keyword triplet, hypothesizing it might aid generation, especially in lower-resource languages.

Datasets

The core of the study is the introduction of the MULTICOM dataset. This benchmark is an extension of the Spanish COCOTEROS corpus. MULTICOM includes:

  • Keywords: Triplets of words that must be included in the generated sentence.
  • Context: Sentences providing background information.
  • Target Sentence: A correct, commonsensical sentence.
  • Counterfactual Sentence: A grammatically correct but nonsensical sentence using the same keywords.
  • Unrelated Sentence: A sentence that doesn't use the keywords.

MULTICOM extends COCOTEROS to four languages: English, Spanish, Dutch, and Valencian. This multilingual aspect is crucial for assessing cross-lingual performance.

Key Insights and Ideas

  • English Dominance: LLMs consistently perform best in English for commonsense generation, with a significant drop-off in performance for other languages, especially less-resourced ones like Valencian.
  • Contextual Support is Nuanced: While the hypothesis was that context would universally help, the results were mixed. Context sometimes improved performance, particularly for underrepresented languages, but in other cases, it had no effect or even a detrimental impact, suggesting model-specific dependencies.
  • LLM-as-a-Judge Reliability: The study found that LLM-based evaluators (Prometheus and JudgeLM) largely align with human judgments, reinforcing their utility for evaluating commonsense reasoning.
  • Language Resource Imbalance Matters: The performance disparity strongly suggests that the lack of diverse, high-quality training data for languages other than English is a major bottleneck for achieving equitable AI capabilities.

Key Results

  • Performance Gap: Across all evaluated LLMs (LLaMA, Qwen, Gemma, EuroLLM, Salamandra), English consistently yielded the highest scores for commonsense generation, as measured by the difference between similarity to target and counterfactual sentences.
  • Valencian Struggles: Valencian, a low-resource language, consistently received the lowest scores, indicating significant challenges for LLMs in generating commonsensical text in this language.
  • Contextual Impact Varies: For instance, the LLaMA models showed mixed results with context, while EuroLLM and Salamandra sometimes benefited more from it, especially in Dutch and Valencian.
  • Model Size: Larger models generally performed slightly better, but the difference wasn't always substantial.
  • Human vs. LLM Agreement: Human evaluations confirmed the LLM-as-a-judge findings, ranking English highest and Valencian lowest, with Spanish generally second.

Limitations and Future Work

Limitations:

  • Automatic Metric Bias: Language-specific encoders used in automatic metrics might be biased towards English, potentially underestimating performance in other languages.
  • Prompting Language: Prompts for LLM-as-a-judge were in English, which could introduce subtle biases if the judging LLM has varying exposure to target languages.
  • Open-Source Focus: The study was limited to open-source models due to cost and access restrictions of proprietary models (like GPT-4, Grok, Gemini).
  • Valencian Data Scarcity: The evaluation for Valencian was constrained by model availability (using Catalan models) and participant availability, requiring caution in interpretation.

Future Work:

  • Fine-tuning LLM Judges: Further fine-tuning LLM-as-a-judge models to better align with nuanced multilingual commonsense reasoning.
  • Proprietary Model Comparison: Evaluating proprietary models like GPT-4, Grok, and Gemini.
  • Improving Low-Resource Capabilities: Developing methods to enhance commonsense reasoning and generation in low-resource languages, potentially through more training data or external knowledge injection.

Summary updated

Abstract

This paper explores the multilingual commonsense generation abilities of Large Language Models (LLMs). To facilitate this investigation, we introduce MULTICOM, a novel benchmark that extends the COCOTEROS dataset to four languages: English, Spanish, Dutch, and Valencian. The task involves generating a commonsensical sentence that includes a given triplet of words. We evaluate a range of open-source LLMs, including LLaMA, Qwen, Gemma, EuroLLM, and Salamandra, on this benchmark. Our evaluation combines automatic metrics, LLM-as-a-judge approaches (using Prometheus and JudgeLM), and human annotations. Results consistently show superior performance in English, with significantly lower performance in less-resourced languages. While contextual support yields mixed results, it tends to benefit underrepresented languages. These findings underscore the current limitations of LLMs in multilingual commonsense generation. The dataset is publicly available at https://huggingface.co/datasets/gplsi/MULTICOM.
BibTeX
@misc{martinezmurillo2025do,
  author = {Iván Martínez-Murillo and Elena Lloret and Paloma Moreda and Albert Gatt},
  title = {{Do LLMs exhibit the same commonsense capabilities across languages?}},
  year = {2025},
  eprint = {2509.06401},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2509.06401},
}
Figure