Large Language Models (LLMs) are often celebrated as ultimate polyglots. However, fluency in a language does not necessarily equate to an understanding of the underlying culture. Researchers from the Technical University of Darmstadt and MBZUAI recently investigated whether multilingual LLMs (mLLMs) are truly "culturally-diverse reasoners" or if they merely memorize text without grasping "cultural common ground." Their findings, presented at NAACL 2024, reveal a significant "culture gap" that persists even when language barriers are removed.
The MAPS Benchmark: Testing Cultural Common Ground
Reasoning in social contexts requires cultural common ground—shared knowledge, values, and metaphors specific to a community. Proverbs serve as a primary vehicle for this. For instance, "The apple doesn't fall far from the tree" is a pragmatic observation about family traits, not a literal statement about fruit.
To evaluate this, the researchers introduced MAPS (MulticulturAl Proverbs and Sayings), a benchmark comprising 2,313 proverbs across six languages: English, German, Russian, Bengali, Mandarin Chinese, and Indonesian. Unlike datasets that focus on simple matching, MAPS requires models to perform inference within situational contexts.
Experimental Setup: Memorization vs. Reasoning
The study evaluated several state-of-the-art mLLMs, including BLOOMZ, mT0, XGLM, XLM-R, and LLaMA-2, using a two-pronged approach:
- Memorization Task: Testing if a model can complete a proverb when the final word is masked (e.g., "Half a loaf is better than ____").
- Reasoning Task: Assessing if a model can select the correct interpretation of a proverb used in a short dialogue.
Additionally, the team conducted a "Cross-Culture Gap Analysis" by translating Chinese proverbs into English. This isolated "culture" from "language," allowing them to measure performance drops when models encountered foreign cultural concepts in a familiar language.
Key Findings: The Limits of Memorization
The research yielded several critical insights into how mLLMs handle cultural nuances:
- Memorization ≠ Understanding: Models proficient at completing proverbs (memorization) did not necessarily excel at interpreting them in context (reasoning). This suggests mLLMs treat proverbs as "frozen" strings rather than semantic units.
- The "Negative" Reasoning Trap: When asked what a person does not mean by a proverb, model performance dropped significantly. This indicates a "positive bias" where models struggle with negation in cultural contexts.
- Figurative Difficulty: Models generally found figurative proverbs harder to interpret than literal ones, with the exception of Chinese, likely due to higher data density for such expressions in training sets.
Scaling and the Culture Gap
The study found that reasoning ability is largely an emergent property of model scale. While smaller models often performed near chance levels, larger models like LLaMA-2 (70B) showed marked improvements.
Despite these gains, a "culture gap" remains. Even with high-quality human translations, models performed worse on proverbs from "distant" cultures. For example, the performance loss due to cultural distance was calculated at 19.40 points for LLaMA-2 in the Chinese-to-English case study. This underscores a lack of underlying cultural schemas—such as the symbolic weight of specific animals or metaphors—necessary for effective global communication.
Conclusion
The MAPS benchmark highlights that while mLLMs are becoming more capable, they still struggle to navigate the pragmatic nuances of diverse cultures. Future work aims to expand MAPS to lower-resource languages and explore methods for injecting cultural priors into LLM training to create truly global AI assistants.
Check out the dataset: https://github.com/UKPLab/maps
Summary updated