Abstract
We present MELA (Multilingual Evaluation of Linguistic Acceptability), the largest benchmark to date for linguistic acceptability, featuring 46K samples across 10 diverse languages. Our study evaluates Large Language Models (LLMs) and supervised baselines, revealing that GPT-4o possesses superior multilingual syntactic capabilities. We also demonstrate significant cross-lingual transfer, where training on a language like Icelandic improves performance on an unrelated language like Chinese. Probing experiments further show that learning acceptability judgments enhances a model's internal representation of syntax, such as part-of-speech and dependency structures.
Introduction
Linguistic acceptability—the ability to distinguish grammatically "well-formed" sentences from "ill-formed" ones—is a core component of linguistic competence. While English has long had the CoLA benchmark, multilingual equivalents have been limited by small sizes or translation artifacts. MELA fills this gap with a large-scale, expert-curated dataset designed to test how well models grasp the fundamental rules of diverse language families.
Methods and Datasets
MELA consists of 46,000 samples across 10 languages: English, Chinese, Italian, Russian, German, French, Spanish, Japanese, Arabic, and Icelandic. Unlike previous datasets, MELA uses a data-driven approach, sourcing examples from linguistics textbooks and journals to ensure high-quality, natural language patterns.
The researchers evaluated models using three primary methods:
- Benchmarking: Comparing supervised models (XLM-R) against state-of-the-art LLMs (GPT-4o, Baichuan2) in zero-shot and few-shot settings.
- Cross-lingual Transfer: Measuring how fine-tuning on one language impacts performance on others.
- Edge Probing: Assessing whether models acquire genuine syntactic knowledge (e.g., POS tagging) through acceptability training.
Key Insights: Universal Syntax and Transfer
A major finding is that syntactic knowledge is surprisingly transferable across unrelated languages. This suggests that models may be learning "universal" linguistic properties.
As illustrated in the Chinese (zh) performance plot, models show significant gains even when trained on non-Chinese data. Specifically, fine-tuning on just 500 Icelandic examples led to a 23.16 MCC score on Chinese. While "cognate advantage" exists—where training on related languages like German helps more with Icelandic—the broad transfer across families highlights the robustness of these syntactic representations.
Key Results
GPT-4o emerged as the top performer, achieving an average of 51.14 MCC, significantly outperforming supervised XLM-R (37.72 MCC). Open-source models generally lagged behind, often requiring few-shot examples to exceed random chance.
The scaling plots across all 10 languages demonstrate that both in-language and cross-lingual performance improve steadily with more data. Furthermore, providing just two in-language examples in a few-shot setting was more effective than providing English examples, emphasizing the importance of language-specific context for LLMs.
Conclusion and Future Work
MELA establishes a new standard for evaluating multilingual syntax. While currently focused on 10 languages, future work will expand to include more diverse language families and non-Latin scripts. The benchmark serves as a vital diagnostic tool for understanding the "core linguistic competence" of increasingly powerful AI models.
Summary updated