TL;DR
This paper investigates whether Large Language Models (LLMs) possess consistent commonsense reasoning abilities across different languages. They introduce MULTICOM, a new benchmark dataset for multilingual commonsense generation, and find that while LLMs perform best in English, their capabilities significantly drop for less-resourced languages, highlighting a critical gap in current AI.
What Problem Does This Paper Address?
Commonsense reasoning is a fundamental human ability that allows us to make inferences based on implicit knowledge about the world. While LLMs have shown impressive capabilities in generating human-like text, it's unclear if these abilities translate equally across languages, especially given that most LLM training data is English-centric. This imbalance could lead to AI systems that are proficient in English but fall short in understanding and generating commonsense in other languages. The paper aims to bridge this gap by systematically evaluating the multilingual commonsense generation performance of various LLMs.
Methods
The researchers employed a multi-pronged evaluation strategy:
- Automatic Metrics: They used established metrics like BERTScore, Universal Sentence Encoder (USE) with Cosine Similarity, and Dependency Parsing with Levenshtein Distance. These metrics compare generated sentences against both a "target" (correct) sentence and a "counterfactual" (grammatically correct but nonsensical) sentence. A larger score difference between the target and counterfactual suggests better commonsense.
- LLM-as-a-Judge: To complement automatic metrics, they used two LLMs, Prometheus-V2.0 and JudgeLM, to rate the commonsensical quality of generated sentences. This approach leverages the advanced reasoning capabilities of LLMs to provide more nuanced evaluations.
- Human Evaluation: A subset of the generated sentences was evaluated by human annotators to validate the findings from the automatic and LLM-based evaluations.
They also explored the impact of providing a "context sentence" alongside the keyword triplet, hypothesizing it might aid generation, especially in lower-resource languages.
Datasets
The core of the study is the introduction of the MULTICOM dataset. This benchmark is an extension of the Spanish COCOTEROS corpus. MULTICOM includes:
- Keywords: Triplets of words that must be included in the generated sentence.
- Context: Sentences providing background information.
- Target Sentence: A correct, commonsensical sentence.
- Counterfactual Sentence: A grammatically correct but nonsensical sentence using the same keywords.
- Unrelated Sentence: A sentence that doesn't use the keywords.
MULTICOM extends COCOTEROS to four languages: English, Spanish, Dutch, and Valencian. This multilingual aspect is crucial for assessing cross-lingual performance.
Key Insights and Ideas
- English Dominance: LLMs consistently perform best in English for commonsense generation, with a significant drop-off in performance for other languages, especially less-resourced ones like Valencian.
- Contextual Support is Nuanced: While the hypothesis was that context would universally help, the results were mixed. Context sometimes improved performance, particularly for underrepresented languages, but in other cases, it had no effect or even a detrimental impact, suggesting model-specific dependencies.
- LLM-as-a-Judge Reliability: The study found that LLM-based evaluators (Prometheus and JudgeLM) largely align with human judgments, reinforcing their utility for evaluating commonsense reasoning.
- Language Resource Imbalance Matters: The performance disparity strongly suggests that the lack of diverse, high-quality training data for languages other than English is a major bottleneck for achieving equitable AI capabilities.
Key Results
- Performance Gap: Across all evaluated LLMs (LLaMA, Qwen, Gemma, EuroLLM, Salamandra), English consistently yielded the highest scores for commonsense generation, as measured by the difference between similarity to target and counterfactual sentences.
- Valencian Struggles: Valencian, a low-resource language, consistently received the lowest scores, indicating significant challenges for LLMs in generating commonsensical text in this language.
- Contextual Impact Varies: For instance, the LLaMA models showed mixed results with context, while EuroLLM and Salamandra sometimes benefited more from it, especially in Dutch and Valencian.
- Model Size: Larger models generally performed slightly better, but the difference wasn't always substantial.
- Human vs. LLM Agreement: Human evaluations confirmed the LLM-as-a-judge findings, ranking English highest and Valencian lowest, with Spanish generally second.
Limitations and Future Work
Limitations:
- Automatic Metric Bias: Language-specific encoders used in automatic metrics might be biased towards English, potentially underestimating performance in other languages.
- Prompting Language: Prompts for LLM-as-a-judge were in English, which could introduce subtle biases if the judging LLM has varying exposure to target languages.
- Open-Source Focus: The study was limited to open-source models due to cost and access restrictions of proprietary models (like GPT-4, Grok, Gemini).
- Valencian Data Scarcity: The evaluation for Valencian was constrained by model availability (using Catalan models) and participant availability, requiring caution in interpretation.
Future Work:
- Fine-tuning LLM Judges: Further fine-tuning LLM-as-a-judge models to better align with nuanced multilingual commonsense reasoning.
- Proprietary Model Comparison: Evaluating proprietary models like GPT-4, Grok, and Gemini.
- Improving Low-Resource Capabilities: Developing methods to enhance commonsense reasoning and generation in low-resource languages, potentially through more training data or external knowledge injection.
Summary updated