Are Multilingual LLMs Culturally-Diverse Reasoners? An Investigation into Multicultural Proverbs and Sayings

Chen Liu, Fajri Koto, Timothy Baldwin, Iryna Gurevych

Large Language Models (LLMs) are often celebrated as ultimate polyglots. However, fluency in a language does not necessarily equate to an understanding of the underlying culture. Researchers from the Technical University of Darmstadt and MBZUAI recently investigated whether multilingual LLMs (mLLMs) are truly "culturally-diverse reasoners" or if they merely memorize text without grasping "cultural common ground." Their findings, presented at NAACL 2024, reveal a significant "culture gap" that persists even when language barriers are removed.

The MAPS Benchmark: Testing Cultural Common Ground

Reasoning in social contexts requires cultural common ground—shared knowledge, values, and metaphors specific to a community. Proverbs serve as a primary vehicle for this. For instance, "The apple doesn't fall far from the tree" is a pragmatic observation about family traits, not a literal statement about fruit.

To evaluate this, the researchers introduced MAPS (MulticulturAl Proverbs and Sayings), a benchmark comprising 2,313 proverbs across six languages: English, German, Russian, Bengali, Mandarin Chinese, and Indonesian. Unlike datasets that focus on simple matching, MAPS requires models to perform inference within situational contexts.

The MAPS benchmark evaluates mLLMs by presenting proverbs within conversational contexts and requiring the model to choose the correct interpretation from multiple choices.

Experimental Setup: Memorization vs. Reasoning

The study evaluated several state-of-the-art mLLMs, including BLOOMZ, mT0, XGLM, XLM-R, and LLaMA-2, using a two-pronged approach:

  1. Memorization Task: Testing if a model can complete a proverb when the final word is masked (e.g., "Half a loaf is better than ____").
  2. Reasoning Task: Assessing if a model can select the correct interpretation of a proverb used in a short dialogue.

Additionally, the team conducted a "Cross-Culture Gap Analysis" by translating Chinese proverbs into English. This isolated "culture" from "language," allowing them to measure performance drops when models encountered foreign cultural concepts in a familiar language.

Key Findings: The Limits of Memorization

The research yielded several critical insights into how mLLMs handle cultural nuances:

  • Memorization ≠ Understanding: Models proficient at completing proverbs (memorization) did not necessarily excel at interpreting them in context (reasoning). This suggests mLLMs treat proverbs as "frozen" strings rather than semantic units.
  • The "Negative" Reasoning Trap: When asked what a person does not mean by a proverb, model performance dropped significantly. This indicates a "positive bias" where models struggle with negation in cultural contexts.
  • Figurative Difficulty: Models generally found figurative proverbs harder to interpret than literal ones, with the exception of Chinese, likely due to higher data density for such expressions in training sets.

Scaling and the Culture Gap

The study found that reasoning ability is largely an emergent property of model scale. While smaller models often performed near chance levels, larger models like LLaMA-2 (70B) showed marked improvements.

Scaling behavior of LLaMA-2 models, showing that reasoning accuracy on the MAPS benchmark improves significantly as the number of parameters increases.

Despite these gains, a "culture gap" remains. Even with high-quality human translations, models performed worse on proverbs from "distant" cultures. For example, the performance loss due to cultural distance was calculated at 19.40 points for LLaMA-2 in the Chinese-to-English case study. This underscores a lack of underlying cultural schemas—such as the symbolic weight of specific animals or metaphors—necessary for effective global communication.

Conclusion

The MAPS benchmark highlights that while mLLMs are becoming more capable, they still struggle to navigate the pragmatic nuances of diverse cultures. Future work aims to expand MAPS to lower-resource languages and explore methods for injecting cultural priors into LLM training to create truly global AI assistants.

Check out the dataset: https://github.com/UKPLab/maps

Summary updated

Abstract

Large language models (LLMs) are highly adept at question answering and reasoning tasks, but when reasoning in a situational context, human expectations vary depending on the relevant cultural common ground. As languages are associated with diverse cultures, LLMs should also be culturally-diverse reasoners. In this paper, we study the ability of a wide range of state-of-the-art multilingual LLMs (mLLMs) to reason with proverbs and sayings in a conversational context. Our experiments reveal that: (1) mLLMs “know” limited proverbs and memorizing proverbs does not mean understanding them within a conversational context; (2) mLLMs struggle to reason with figurative proverbs and sayings, and when asked to select the wrong answer (instead of asking it to select the correct answer); and (3) there is a “culture gap” in mLLMs when reasoning about proverbs and sayings translated from other languages. We construct and release our evaluation dataset MAPS (MulticulturAl Proverbs and Sayings) for proverb understanding with conversational context for six different languages.
BibTeX
@inproceedings{liu2024are,
  author = {Chen Liu and Fajri Koto and Timothy Baldwin and Iryna Gurevych},
  title = {{Are Multilingual LLMs Culturally-Diverse Reasoners? An Investigation into Multicultural Proverbs and Sayings}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.18653/v1/2024.naacl-long.112},
}
Figure