MELA: Multilingual Evaluation of Linguistic Acceptability

Ziyin Zhang, Yikang Liu, Weifang Huang, Junyu Mao, Rui Wang, Hai Hu

Abstract

We present MELA (Multilingual Evaluation of Linguistic Acceptability), the largest benchmark to date for linguistic acceptability, featuring 46K samples across 10 diverse languages. Our study evaluates Large Language Models (LLMs) and supervised baselines, revealing that GPT-4o possesses superior multilingual syntactic capabilities. We also demonstrate significant cross-lingual transfer, where training on a language like Icelandic improves performance on an unrelated language like Chinese. Probing experiments further show that learning acceptability judgments enhances a model's internal representation of syntax, such as part-of-speech and dependency structures.

Introduction

Linguistic acceptability—the ability to distinguish grammatically "well-formed" sentences from "ill-formed" ones—is a core component of linguistic competence. While English has long had the CoLA benchmark, multilingual equivalents have been limited by small sizes or translation artifacts. MELA fills this gap with a large-scale, expert-curated dataset designed to test how well models grasp the fundamental rules of diverse language families.

Methods and Datasets

MELA consists of 46,000 samples across 10 languages: English, Chinese, Italian, Russian, German, French, Spanish, Japanese, Arabic, and Icelandic. Unlike previous datasets, MELA uses a data-driven approach, sourcing examples from linguistics textbooks and journals to ensure high-quality, natural language patterns.

The researchers evaluated models using three primary methods:

  1. Benchmarking: Comparing supervised models (XLM-R) against state-of-the-art LLMs (GPT-4o, Baichuan2) in zero-shot and few-shot settings.
  2. Cross-lingual Transfer: Measuring how fine-tuning on one language impacts performance on others.
  3. Edge Probing: Assessing whether models acquire genuine syntactic knowledge (e.g., POS tagging) through acceptability training.

Key Insights: Universal Syntax and Transfer

A major finding is that syntactic knowledge is surprisingly transferable across unrelated languages. This suggests that models may be learning "universal" linguistic properties.

Linguistic acceptability performance for Chinese, illustrating transfer from other languages.

As illustrated in the Chinese (zh) performance plot, models show significant gains even when trained on non-Chinese data. Specifically, fine-tuning on just 500 Icelandic examples led to a 23.16 MCC score on Chinese. While "cognate advantage" exists—where training on related languages like German helps more with Icelandic—the broad transfer across families highlights the robustness of these syntactic representations.

Key Results

GPT-4o emerged as the top performer, achieving an average of 51.14 MCC, significantly outperforming supervised XLM-R (37.72 MCC). Open-source models generally lagged behind, often requiring few-shot examples to exceed random chance.

Performance across 10 languages showing scaling and cross-lingual transfer.

The scaling plots across all 10 languages demonstrate that both in-language and cross-lingual performance improve steadily with more data. Furthermore, providing just two in-language examples in a few-shot setting was more effective than providing English examples, emphasizing the importance of language-specific context for LLMs.

Conclusion and Future Work

MELA establishes a new standard for evaluating multilingual syntax. While currently focused on 10 languages, future work will expand to include more diverse language families and non-Latin scripts. The benchmark serves as a vital diagnostic tool for understanding the "core linguistic competence" of increasingly powerful AI models.

Summary updated

Abstract

In this work, we present the largest benchmark to date on linguistic acceptability: Multilingual Evaluation of Linguistic Acceptability -- MELA, with 46K samples covering 10 languages from a diverse set of language families. We establish LLM baselines on this benchmark, and investigate cross-lingual transfer in acceptability judgements with XLM-R. In pursuit of multilingual interpretability, we conduct probing experiments with fine-tuned XLM-R to explore the process of syntax capability acquisition. Our results show that GPT-4o exhibits a strong multilingual ability, outperforming fine-tuned XLM-R, while open-source multilingual models lag behind by a noticeable gap. Cross-lingual transfer experiments show that transfer in acceptability judgment is non-trivial: 500 Icelandic fine-tuning examples lead to 23 MCC performance in a completely unrelated language -- Chinese. Results of our probing experiments indicate that training on MELA improves the performance of XLM-R on syntax-related tasks. Our data is available at https://github.com/sjtu-compling/MELA.
BibTeX
@inproceedings{zhang2024mela,
  author = {Ziyin Zhang and Yikang Liu and Weifang Huang and Junyu Mao and Rui Wang and Hai Hu},
  title = {{MELA: Multilingual Evaluation of Linguistic Acceptability}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
  url = {https://openreview.net/forum?id=3VrgRGm4Ax},
}
Figure